How to build libmambapy 2.0 without having libmamba installed already

[flying-sheep]

Hi, I want to package libmamba, libmambapy, and a dynamically linked mamba/micromamba for Arch Linux

I came across #2971, but there seems to be no solution there.

My goal is to build and package everything together.

I can cmake in the root directory, which builds everything but libmambapy. But if I then then invoke python -m build, it will invoke cmake again, this time trying to find libmamba in the system. This fails because I only built everything and didn't install libmamba yet.

I clumsily tried to make the python build system build everything together, but that failed:

sed -i 's/cmake_args=/cmake_source_dir="..", cmake_args=/' setup.py

export SKBUILD_CONFIGURE_OPTIONS="\
    -D CMAKE_C_COMPILER=gcc \
    -D CMAKE_CXX_COMPILER=g++ \
    -D CMAKE_BUILD_WITH_INSTALL_RPATH=ON \
    -D BUILD_LIBMAMBA=ON \
    -D BUILD_LIBMAMBAPY=ON \
    -D BUILD_MICROMAMBA=OFF \
    -D BUILD_MAMBA_PACKAGE=OFF \
    -D CMAKE_JOB_POOLS:STRING=compile=5;link=2 \
    --preset=mamba-unix-shared-release"

python -m build --wheel --no-isolation

Error:

Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The CXX compiler identification is GNU 14.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Ninja' generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy/_skbuild/linux-x86_64-3.12/cmake-build
  Command:
    /usr/lib/python3.12/site-packages/cmake/data/bin/cmake /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25 -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=/usr/lib/python3.12/site-packages/ninja/data/bin/ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy/_skbuild/linux-x86_64-3.12/cmake-install/src/libmambapy -DPYTHON_VERSION_STRING:STRING=3.12.6 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/lib/python3.12/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.12 -DPYTHON_LIBRARY:PATH=/usr/lib/libpython3.12.so -DPython_EXECUTABLE:PATH=/usr/bin/python -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.12 -DPython_NumPy_INCLUDE_DIRS:PATH=/usr/lib/python3.12/site-packages/numpy/_core/include -DPython3_EXECUTABLE:PATH=/usr/bin/python -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.12 -DPython3_NumPy_INCLUDE_DIRS:PATH=/usr/lib/python3.12/site-packages/numpy/_core/include -DCMAKE_MAKE_PROGRAM:FILEPATH=/usr/lib/python3.12/site-packages/ninja/data/bin/ninja -DMAMBA_INSTALL_PYTHON_EXT_LIBDIR=/home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy/_skbuild/linux-x86_64-3.12/cmake-install/src/libmambapy -DCMAKE_BUILD_TYPE:STRING=Release -D CMAKE_C_COMPILER=gcc -D CMAKE_CXX_COMPILER=g++ -D CMAKE_BUILD_WITH_INSTALL_RPATH=ON -D BUILD_LIBMAMBA=ON -D BUILD_LIBMAMBAPY=ON -D BUILD_MICROMAMBA=OFF -D BUILD_MAMBA_PACKAGE=OFF -D 'CMAKE_JOB_POOLS:STRING=compile=5;link=2' --preset=mamba-unix-shared-release

Not searching for unused variables given on the command line.
Preset CMake variables:

  BUILD_LIBMAMBA_TESTS="ON"
  BUILD_MAMBA="ON"
  BUILD_SHARED="ON"
  CMAKE_AR=""
  CMAKE_CXX_FLAGS_DEBUG=""
  CMAKE_CXX_FLAGS_RELEASE="-march=native -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src=/usr/src/debug/python-libmamba -flto=auto"
  CMAKE_C_FLAGS_DEBUG=""
  CMAKE_C_FLAGS_RELEASE="-march=native -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src=/usr/src/debug/python-libmamba -flto=auto"
  CMAKE_EXE_LINKER_FLAGS="-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto"
  CMAKE_Fortran_COMPILER=""
  CMAKE_Fortran_FLAGS_DEBUG=""
  CMAKE_Fortran_FLAGS_RELEASE=""
  CMAKE_INSTALL_LIBDIR="lib"
  CMAKE_LINKER=""
  CMAKE_PREFIX_PATH=""
  CMAKE_RANLIB=""

-- Building libmamba v2.0.0
-- libmamba binary version: v2.0.0
-- Setting LTO for target libmamba::solv-cpp
-- Adding shared libmamba target
-- Setting LTO for target libmamba::libmamba-dyn
   -> Dynamically linking against libmamba (shared) dependencies
-- Building libmamba C++ tests
-- Found pybind11: /usr/include (found version "2.13.6")
-- Adding executable mamba
-- Setting LTO for target micromamba::mamba
-- Configuring done (0.9s)
-- Generating done (0.0s)
-- Build files have been written to: /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/build/mamba-unix-shared-release
Error: could not load cache
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
    cmkr.make(make_args, install_target=cmake_install_target, env=env)
  File "/usr/lib/python3.12/site-packages/skbuild/cmaker.py", line 697, in make
    self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
  File "/usr/lib/python3.12/site-packages/skbuild/cmaker.py", line 742, in make_impl
    raise SKBuildError(msg)

An error occurred while building with CMake.
  Command:
    /usr/lib/python3.12/site-packages/cmake/data/bin/cmake --build . --target install --config Release --
  Install target:
    install
  Source directory:
    /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy
  Working directory:
    /home/phil/Dev/PKGBUILDs/checkouts/Python/python-libmamba/src/mamba-2024.09.25/libmambapy/_skbuild/linux-x86_64-3.12/cmake-build
Please check the install target is valid and see CMake's output for more information.

[jjerphan]

Hi,

I would recommend checking this part of the build script used for conda-forge's distribution of libmambapy.

[flying-sheep]

Thanks, but that doesn‘t help. As said, I want to build everything together instead of building libmamba, then installing it, then building libmambapy (which can at this point rely on find_package finding libmamba in the system)

[jjerphan]

Have you tried using CMAKE_INSTALL_PREFIX such as what is indicated in this part of the documentation?

[henryiii]

With this setup:

micromamba create -n mamba --file dev/environment-micromamba-static.yml
micromamba activate mamba
micromamba install -c conda-forge scikit-build setuptools python-build

I think it is:

cmake -S. -Bbuild \
    -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
    -DBUILD_LIBMAMBA=ON \
    -DBUILD_LIBMAMBAPY=ON \
    -DBUILD_MICROMAMBA=OFF \
    -DBUILD_MAMBA_PACKAGE=OFF \
    -DBUILD_SHARED=ON
cmake --build build --parallel 8
cmake --install build --prefix install
cd libmambapy
export SKBUILD_CONFIGURE_OPTIONS="\
    -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
    -DBUILD_LIBMAMBA=ON \
    -DBUILD_LIBMAMBAPY=ON \
    -DBUILD_MICROMAMBA=OFF \
    -DBUILD_MAMBA_PACKAGE=OFF \
    -Dlibmamba_ROOT=$PWD/../install"
python -m build -x --wheel --no-isolation

I've got a different lib mamba in my /usr/local from homebrew that I used to set up the environment, and it's colliding, but otherwise I think that would do what is requested here?

[flying-sheep]

Awesome! So one needs to build the parts separately, but can point the second one to use the first.

Thank you so much!

I bet other Linux distributions’ packagers for conda will also be very grateful to you for this!

cc @opoplawski

[opoplawski]

I'm running into the following issue with 2.0.6 when doing the cmake --install build --prefix install step:

-- Installing: /home/orion/fedora/libmamba/libmamba-2.0.6-build/mamba-2.0.6/install/lib64/cmake/libmamba/libmambaTargets-relwithdebinfo.cmake
-- Installing: /usr/etc/profile.d/mamba.sh
CMake Error at redhat-linux-build/micromamba/cmake_install.cmake:54 (file):
  file INSTALL cannot copy file
  "/home/orion/fedora/libmamba/libmamba-2.0.6-build/mamba-2.0.6/redhat-linux-build/micromamba/etc/profile.d/mamba.sh"
  to "/usr/etc/profile.d/mamba.sh": Permission denied.
Call Stack (most recent call first):
  redhat-linux-build/cmake_install.cmake:57 (include)

I guess I have BUILD_MAMBA=ON in the build step, but I was hoping to avoid having another build step afterwards. Am I stuck with that?

[jjerphan]

You need to specify CMAKE_INSTALL_PREFIX to the prefix of your installation.

[henryiii]

This is wrong:

mamba/micromamba/CMakeLists.txt

Line 118 in 27010d1

DESTINATION ${CMAKE_INSTALL_PREFIX}/etc/profile.d/

That hardcodes CMAKE_INSTALL_PREFIX (which is what this would be relative to anyway) which makes --prefix not work. You can work around it by setting CMAKE_INSTALL_PREFIX during the configuration step.

[opoplawski]

I had to get rid of that anyway in the Fedora build to get it to install into /etc/profile.d instead of /usr/etc/profile.d. But I didn't go back to see if that fixes my multiple build issue.

[opoplawski]

No, it would have to become etc/... but then you run into it getting put in /usr/etc which is wrong. Seems like a basic cmake issue - see https://gitlab.kitware.com/cmake/cmake/-/issues/25852

[flying-sheep]

This is wrong:

mamba/micromamba/CMakeLists.txt

Line 118 in 27010d1

DESTINATION ${CMAKE_INSTALL_PREFIX}/etc/profile.d/

how can we fix it? your workaround doesn’t work because we need to install twice: once to get libmambapy to build and once to the directory that’s going to be packaged up (these directories can’t be the same as we don’t have access to the package directory during building)

[flying-sheep]

ah, got it: #4057