Creation of dependencies broken/only usable with alidist

[Ni-Langguth]

This documents the creation of a build dependency graph. Creating a dependency graph is not possible in repos other than alidist at the moment, because bits is hardcoded to read defaults-release.sh in alidist.

To reproduce the error I got, do:

docker run -it --pull always -e SHELL -u root --cpus=64 --security-opt seccomp:unconfined -v /build:/build -v /ec/conf:/ec/conf -v /cvmfs:/cvmfs --network host gitlab-registry.cern.ch/sft/docker/alma9

mkdir /build/nlanggut/bitsPlayground1 && cd /build/nlanggut/bitsPlayground1
export PATH=/build/nlanggut/bitsPlayground1/bits/:$PATH
yum install -y bison flex python-yaml python-requests python-jinja2
git clone https://github.com/bitsorg/bits.git
git clone https://github.com/bitsorg/lcg.bits.git
cd lcg.bits
git submodule update --recursive --init
bits deps boost --outgraph graph.pdf

The error I got:

[root@lcgapp-alma9-physical lcg.bits]# bits deps boost --outgraph graph.pdf
ERROR: Default `alidist/ml.bits/defaults-release.sh' does not exists.
ERROR: 
Traceback (most recent call last):
  File "/build/nlanggut/bitsPlayground1/bits/bitsBuild", line 133, in <module>
    doMain(args, parser)
  File "/build/nlanggut/bitsPlayground1/bits/bitsBuild", line 70, in doMain
    sys.exit(0 if doDeps(args, parser) else 1)
  File "/build/nlanggut/bitsPlayground1/bits/bits_helpers/deps.py", line 18, in doDeps
    (err, overrides, taps) = parseDefaults(args.disable, defaultsReader, debug)
  File "/build/nlanggut/bitsPlayground1/bits/bits_helpers/utilities.py", line 556, in parseDefaults
    defaultsMeta, defaultsBody = defaultsGetter()
  File "/build/nlanggut/bitsPlayground1/bits/bits_helpers/deps.py", line 17, in <lambda>
    defaultsReader = lambda: readDefaults(args.configDir, args.defaults, parser.error, args.architecture)
  File "/build/nlanggut/bitsPlayground1/bits/bits_helpers/utilities.py", line 365, in readDefaults
    if exists(xDefaults):
  File "/usr/lib64/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[pbuncic]

Please try with the master, I fixed it by avoiding another Docker invocation.

[pbuncic]

@Ni-Langguth, please check if this works for you now.

[Ni-Langguth]

@pbuncic I am on sick leave, I will check once I'm back. Have you been able to reproduce the issue as I have described here?

[Ni-Langguth]

@pbuncic I tried again, the same error occurs

[pbuncic]

@Ni-Langguth, please try again now. I was testing before in alice.bits and had the alibuild directory checked out so, by chance it worked for me.

[Ni-Langguth]

@pbuncic can you tell me what you did exactly to get it to work? I still have the same error when doing what I posted above.

[pbuncic]

 > pbuncic@bits:~/Software/Bits$ uname -a
Linux bits 6.17.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct 18 10:10:29 UTC 2025 x86_64 GNU/Linux
 > docker run -it --pull always -e SHELL -u root --cpus=8 --security-opt seccomp:unconfined -v /tmp:/tmp --network host gitlab-registry.cern.ch/sft/docker/alma9
# yum install -y bison flex python-yaml python-requests python-jinja2 graphviz
# cd /tmp/bits/
# export PATH=$PWD:$PATH
 # cd ..
 # git clone https://github.com/bitsorg/lcg.bits.git
 # cd lcg.bits/
  # git clone https://github.com/bitsorg/common.bits.git
  # pbuncic@bits:/tmp/lcg.bits$ cat bits.rc
[bits]
  organisation=LCG

[LCG]
  pkg_prefix=VO_LCG
  sw_dir=../sw
  repo_dir=.
  search_path=common 

  # bits deps boost --outgraph /tmp/x.pdf
Dependencies graph generated: /tmp/x.pdf

[Ni-Langguth]

Are you sure this is everything? Cloning bits, for example, is not included in your instructions. Is there anything else missing? Can you please repeat what I did and see what happens for you, including sshing to the cluster machine? I cannot execute these commands on your device, and they cause an ERROR on the public container I am using.
To me, it seems clear that bits is looking specifically for alidist in this graph function, which does not exist in the lcg.bits repo.

What I did..

..on my machine, to get to the cluster:

ssh root@lcgapp-alma9-physical

..on the cluster machine, to get into the container:

docker run -it --pull always -e SHELL -u root --cpus=8 --security-opt seccomp:unconfined -v /tmp:/tmp --network host gitlab-registry.cern.ch/sft/docker/alma9

..inside the docker container:

yum install -y bison flex python-yaml python-requests python-jinja2 graphviz
cd /tmp
git clone https://github.com/bitsorg/bits.git
cd bits
export PATH=$PWD:$PATH
cd ..
git clone https://github.com/bitsorg/lcg.bits.git
cat bits.rc
bits deps boost --outgraph /tmp/x.pdf

What I got:

[root@lcgapp-alma9-physical lcg.bits]# cat bits.rc
[bits]
  organisation=LCG

[LCG]
  pkg_prefix=VO_LCG
  sw_dir=../sw
  repo_dir=.
  search_path=bits,general,simulation,hepmc,analysis,ml

[root@lcgapp-alma9-physical lcg.bits]# bits deps boost --outgraph /tmp/x.pdf
ERROR: Package boost not found in .

When changing into common.bits:

[root@lcgapp-alma9-physical lcg.bits]# cd common.bits/
[root@lcgapp-alma9-physical common.bits]# bits deps boost --outgraph /tmp/x.pdf
ERROR: Default `alidist/defaults-release.sh' does not exists.
ERROR: 
Traceback (most recent call last):
  File "/tmp/bits/bitsBuild", line 133, in <module>
    doMain(args, parser)
  File "/tmp/bits/bitsBuild", line 70, in doMain
    sys.exit(0 if doDeps(args, parser) else 1)
  File "/tmp/bits/bits_helpers/deps.py", line 21, in doDeps
    (err, overrides, taps) = parseDefaults(args.disable, defaultsReader, debug)
  File "/tmp/bits/bits_helpers/utilities.py", line 556, in parseDefaults
    defaultsMeta, defaultsBody = defaultsGetter()
  File "/tmp/bits/bits_helpers/deps.py", line 20, in <lambda>
    defaultsReader = lambda: readDefaults(args.configDir, args.defaults, parser.error, args.architecture)
  File "/tmp/bits/bits_helpers/utilities.py", line 365, in readDefaults
    if exists(xDefaults):
  File "/usr/lib64/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[pbuncic]

Since we agreed to put all recipes in one repository (common.bits) you need to modify bits.rc to reflect that (which is why I showed what is in the file, I thought that this was obvious to understand). So, you need to run the subsequent bits commands in the lcg.bits directory and not in lcg.bits/common.bits.

[pbuncic]

BTW, the reason why you see the reference to alidist/defaults-release.sh in your output is that it did not find bits.rc file and reverted to backward alibuild compatible mode, which assumes that the recipes are in the alibuild directory.

[Ni-Langguth]

Yes! Works with our GitLab repo.

When I created this issue, we had not integrated common.bits yet, and the problem was the same between common.bits and the other submodules we had previously. I did not expect a solution that works with common.bits, but not with other submodules.

FYI, this breaks CI tests:
3f3fded

[Ni-Langguth]

But we had bits.rc since October 14 with this search_path (https://github.com/bitsorg/lcg.bits/blame/143d583d345f0660ef497206983fcc41faa02613/bits.rc):

search_path=bits,general,simulation,hepmc,analysis,ml

Why was it not found? Does bits revert to searching in alibuild for all recipes it does not find in the search_path?

[pbuncic]

In your last attempt while you still had in your bits.rc

search_path=bits,general,simulation,hepmc,analysis,ml

you followed what I typed which assumed that you checked out common.bits AND modified bits.rc. There I did not run

git submodule update --recursive --init

and submodule directories were empty.

I do not understand why test is failing, perhaps you can look and fix it.