Skip to content

support for inheriting package recepies and improved generated packages#35

Merged
pbuncic merged 2 commits into
mainfrom
pkg-inheritance-and-gen-pkgs
Nov 12, 2025
Merged

support for inheriting package recepies and improved generated packages#35
pbuncic merged 2 commits into
mainfrom
pkg-inheritance-and-gen-pkgs

Conversation

@smuzaffar

Copy link
Copy Markdown
Collaborator

Thanks to @akritkbehera , this PR has the following improvements

Variable expansion in resolve_tag:

The following is now supported so we can have numerical version of PKG_VERSION without having to repeat ourself it in tag

package: demo
version: 2.1.2
tag: v%(version)s

Fixed repo directory

There was wrong path being constructed if configuration file is passed which was being interpreted as folder. Previously running bits build demo --config=bits.rc led to a path repositories/bits.rc/general.bits/defaults-release.sh. Now this resolves to repositories/general.bits/defaults-release.sh

Improved Generated packages

Search for all generated packages in all repositories and not just the first one

YAML Merge & Inheritance Behavior for Recipes

This PR introduces structured merge and inheritance behavior for YAML-based recipes.

The goal is to enable field-level control with the following policies:

  • merge: combine fields from parent and child.
  • remove: exclude fields from parent.
  • inherit: forcibly inherit specific fields from parent even if overridden.

Base: general.bits

package: demo
version: vGeneral
sources: 
 - www.xyz-%(version)s.tar.gz
source:
 - xyz.tar.gz
variables:
  bar: from_cms.bits
  foo: from_general.bits
---
<Recipe From General>
#package demo.sh
from: general.bits
version: vCMS
variables:
  bar: from_cms.bits
merge_policy:
 - merge: variables
 - remove: env
# - inherit: source
---
# Resulting Recipe (after merge/inheritance resolution)
package: demo
version: vCMS
variables:
  foo: from_general.bits
sources: 
 - www.xyz-%(version)s.tar.gz
env:
  general: general
---
<Build Recipe mentioned General>

@smuzaffar

Copy link
Copy Markdown
Collaborator Author

what ??


The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings

@pbuncic

pbuncic commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

Well, we have to pay or make it public...

@pbuncic
pbuncic requested a review from ktf November 6, 2025 14:48
@pbuncic

pbuncic commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

@ktf, can you review this pull request? It would be nice to have some tests in this pull request to assure that nothing is broken wrt current aliBuild model. We were recently working on exactly the same problem (merging recipes) and @Ni-Langguth has a pull request in making aiming to address this problem. The idea was to use chain defaults using --defaults a::b::c syntax and combine them into one effective definition. You are going beyond that by adding merge_policy, we need to see how to combine that. Also, I added possibility to include yaml nodes from another file, another dimension to look into.

@Ni-Langguth

Copy link
Copy Markdown

Hey, I've written tests for merging defaults, they are over on my branch here, where we developed the n defaults functions: #29
They cover the cases that I could think of, maybe I am missing some.

I am also looking into tests for including yaml files.

@Ni-Langguth

Copy link
Copy Markdown

I wrote tests on my branch, as I mentioned before. If you could do the same for the changes you have made, that would be great.

This branch breaks a bunch of CI tests (not the ones I wrote, tests in tests/test_packagelist.py), @smuzaffar @akritkbehera can you run the CI on your local machines to find the broken tests or should I put what I found here in the conversation?

Could we also get a follow up on the github CI please @pbuncic ?

I have rebased my branch onto this one, there were no conflicts in doing so. Once the tests don't break here anymore, I will rebase again and make sure again that nothing breaks. I hope we can merge then.

@smuzaffar

Copy link
Copy Markdown
Collaborator Author

@akritkbehera , can you please check why CI tests are failing here.

[root@lcgapp-alma9-physical bits]# tox -e py39 -rvv
======================================================================
ERROR: test_force_rebuild_command_line (test_packagelist.ForceRebuildTestCase)
The --force-rebuild option must take precedence, if given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 208, in test_force_rebuild_command_line
    specs, _, _, _, _ = getPackageListWithDefaults(
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_force_rebuild_recipe (test_packagelist.ForceRebuildTestCase)
If the recipe specifies force_rebuild, it must be applied.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 202, in test_force_rebuild_recipe
    specs, _, _, _, _ = getPackageListWithDefaults(["force-rebuild"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_dirty_system_check (test_packagelist.ReplacementTestCase)
Check that prefer_system_check runs in isolation and doesn't create files in cwd.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 190, in test_dirty_system_check
    getPackageListWithDefaults(["dirty_prefer_system_check"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_disable (test_packagelist.ReplacementTestCase)
Check that not specifying any replacement disables the package.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 133, in test_disable
    getPackageListWithDefaults(["disable"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_missing_replacement_spec (test_packagelist.ReplacementTestCase)
Check a warning is displayed when the replacement spec is not found.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 182, in test_missing_replacement_spec
    getPackageListWithDefaults(["missing-spec"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_replacement_given (test_packagelist.ReplacementTestCase)
Check that specifying a replacement spec means it is used.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 145, in test_replacement_given
    getPackageListWithDefaults(["with-replacement"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
ERROR: test_replacement_recipe_given (test_packagelist.ReplacementTestCase)
Check that specifying a replacement recipe means it is used.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 162, in test_replacement_recipe_given
    getPackageListWithDefaults(["with-replacement-recipe"])
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_packagelist.py", line 97, in getPackageListWithDefaults
    return_values = getPackageList(
  File "/build/nlanggut/bitsPlayground1/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/utilities.py", line 697, in getPackageList
    err, spec, recipe = parseRecipe(getRecipeReader(filename, configDir, generatedPackages[pkgdir]), generatedPackages)
KeyError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/CONFIG_DIR'

======================================================================
FAIL: test_actionParsing (test_args.ArgsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_args.py", line 94, in test_actionParsing
    self.assertEqual(args[k], v)
AssertionError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/test/alidist' != 'test/alidist'
- /build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/test/alidist
+ test/alidist


======================================================================
FAIL: test_recipes (test_parseRecipe.TestRecipes)

@akritkbehera

Copy link
Copy Markdown
Collaborator

Yes @smuzaffar
Looking into it.

@smuzaffar

Copy link
Copy Markdown
Collaborator Author

@akritkbehera , I think I have managed to fix the CI tests issue. For CI tests https://github.com/bitsorg/bits/blob/main/tests/test_utilities.py mocks the exists function and use relative paths. Using relative paths within utilities.py should avoid CI test failure

@smuzaffar

Copy link
Copy Markdown
Collaborator Author

@Ni-Langguth , CI tests are failing here due to private repo. Can you run these locally to check if issue is fixed?

@Ni-Langguth

Copy link
Copy Markdown

@smuzaffar I just tested on my machine. I still see:

FAIL: test_actionParsing (test_args.ArgsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_args.py", line 94, in test_actionParsing
    self.assertEqual(args[k], v)
AssertionError: '/build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/test/alidist' != 'test/alidist'
- /build/nlanggut/bitsPlayground1/bits/.tox/py39/tmp/test/alidist
+ test/alidist

and

======================================================================
FAIL: test_recipes (test_parseRecipe.TestRecipes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/nlanggut/bitsPlayground1/bits/tests/test_parseRecipe.py", line 77, in test_recipes
    self.assertEqual(err,  "Unable to parse test_broken_1.sh. Header missing.")
AssertionError: None != 'Unable to parse test_broken_1.sh. Header missing.'

----------------------------------------------------------------------

These failures are not on the main branch nor on my branch and my branch does not fix them.

Do you see them too when you try to run the tests?

@smuzaffar
smuzaffar force-pushed the pkg-inheritance-and-gen-pkgs branch from 98c967f to be92c6c Compare November 12, 2025 06:41
@smuzaffar
smuzaffar force-pushed the pkg-inheritance-and-gen-pkgs branch from be92c6c to 1ae6136 Compare November 12, 2025 06:52
@smuzaffar

Copy link
Copy Markdown
Collaborator Author

@Ni-Langguth , I have fixed the above mentioned errors.

  • @akritkbehera had improved the error handling for parseRecipe which broke one of the CI jobs. I think previously bits was keep on building even if there are broken recipes that is why @akritkbehera was calling error() to fail and stop on broken recipes. For now I have rolled back to old style return of err message. We will see what we can do to improve it
  • Other error was that we were converting the relative configDir to its absolute path in the args.py. That broke the other CI test as test was assuming to get back the same relative path. We will see if we can live with relative paths.

I still get the following errors but these errors are already there for main branch so I do not thing change here is reason for those failures.

======================================================================
ERROR: test_deps (test_deps.DepsTestCase)
Check doDeps doesn't raise an exception.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/el8_amd64_gcc13/external/python3/3.9.14-e16d2924e9eb9db8fddd14e187cf7209/lib/python3.9/unittest/mock.py", line 1336, in patched
    return func(*newargs, **newkeywargs)
  File "/build/muz/bits-test/bits/tests/test_deps.py", line 81, in test_deps
    doDeps(args, MagicMock())
  File "/build/muz/bits-test/bits/.tox/py39/lib/python3.9/site-packages/bits_helpers/deps.py", line 100, in doDeps
    fp.write(dot)
ValueError: I/O operation on closed file

----------------------------------------------------------------------
Ran 60 tests in 1.476s

FAILED (errors=1, skipped=2)
py39: exit 1 (2.48 seconds) /build/muz/bits-test/bits/.tox/py39/tmp> coverage run --source=/build/muz/bits-test/bits -a -m unittest discover /build/muz/bits-test/bits/tests pid=2669924
  py39: FAIL code 1 (27.91=setup[24.45]+cmd[0.98,2.48] seconds)
  evaluation failed :( (28.15 seconds)

@Ni-Langguth

Ni-Langguth commented Nov 12, 2025

Copy link
Copy Markdown

Since your commits are squashed, I can't see what you changed, could you point out what that was please?
Sorry, I guess I had not refreshed this page, didn't see your comment.

I think throwing an error for broken recipes is a good idea, we could add this in another PR, let's get this merged first?

I rebased my branch onto this one and can now run the tests as successfully as before.

I suggest that we merge this branch into main first (once it is reviewed), then I can rebase my branch onto main once more and merge it as well.

Do you want to write tests for your developments so that we don't break them accidentally in the future?

@Ni-Langguth

Copy link
Copy Markdown

I left a comment about the remaining error on a commit here: b6c13cc

Added Banners to show configuredDirectory
Removed --config=file from being passed on further if config is passed.
@pbuncic
pbuncic merged commit 1504f02 into main Nov 12, 2025
0 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants