Linking issue "Failed to dlopen libcutensornet", and the potential solution.

[liweintu]

After a fresh installation from poetry, when running pytest test_cuquantum_cutensor_backend.py, I encountered this error below.

FAILED test_cuquantum_cutensor_backend.py::test_eval[1] - RuntimeError: Failed to dlopen libcutensornet

After debugging using Gemini, the error seems to be related to shared libraries folders not being able to be located, which was due to installation via pip/poetry, as explained below.

The error you're encountering happens because cuquantum-python-cu12 and its related dependencies (like cutensor, cusolver, etc.) are installed via pip/poetry. When installed this way on Linux, their shared libraries (.so files) are placed in separate folders inside your Python environment's site-packages directory. The system's dynamic linker cannot automatically locate them, resulting in the Failed to dlopen libcutensornet runtime error as it fails to resolve inner dependencies like libcusolver.so.11.

The following fix works, that is, adding these folders to LD_LIBBRARY_PATH (recommended by Gemini too), after which, the error goes away.

# 1. Define the base path for your environment's site-packages
export SITE_PACKAGES="/home/liwei/miniforge3/envs/qibotn_cuda12_py312/lib/python3.12/site-packages"

# 2. Add the NVIDIA and cuQuantum library paths to LD_LIBRARY_PATH
export LD_LIBRARY_PATH="$SITE_PACKAGES/nvidia/cublas/lib:$SITE_PACKAGES/nvidia/cusolver/lib:$SITE_PACKAGES/nvidia/cusparse/lib:$SITE_PACKAGES/nvidia/cuda_nvrtc/lib:$SITE_PACKAGES/nvidia/cuda_runtime/lib:$SITE_PACKAGES/nvidia/nccl/lib:$SITE_PACKAGES/cuquantum/lib:$SITE_PACKAGES/cutensor/lib:$LD_LIBRARY_PATH"

# 3. Run your tests again
pytest test_cuquantum_cutensor_backend.py

Maybe we should update LD_LIBBRARY_PATH too according to the installation folders?

[scarrazza]

Yeah, adding a comment about it in the documentation looks good to me. Thanks @liweintu .

[alecandido]

SITE_PACKAGES is not consumed by Python. The Python variable for that is PYTHONPATH, and it only has an effect on import Python packages.
Then, Python packages can also be shared libraries (not necessarily .py scripts, or folders containing __init__.py), but still taken from the same path.

Since in this case it was just shortcut in the LD_LIBRARY_PATH definition, you could even save the export.

In any case, it is a bit weird, because CuQuantum is not the only package to ship compiled extensions, and others are just working out of the box.
E.g. even NumPy is just doing the same. The following is part of the content of the NumPy package:

❯ : ls numpy/ numpy.libs/
numpy/:
__init__.cython-30.pxd  __init__.py   [...]  core       doc dtypes.pyi     exceptions.pyi  fft  [...]
__init__.pxd            __init__.pyi  [...]  ctypeslib  dtypes.py          exceptions.py   f2py [...]

numpy.libs/:
libgfortran-040039e1-0352e75f.so.5.0.0  libquadmath-96973f99-934c22de.so.0.0.0  libscipy_openblas64_-ff84a88b.so

Relative Paths: NumPy’s Python modules (e.g., numpy/core/_multiarray_umath.cpython-*.so) are compiled extensions that link to the shared libraries in numpy.libs/. These extensions are built with relative paths (e.g., @loader_path/../numpy.libs/libnpymath.so on macOS, or @rpath/$ORIGIN on Linux/Windows)

This is what the AI has to say about the loading mechanism. I.e. it is based on relative paths (I also thought rpath was a platform-independent standard, but apparently it is not...).

I guess this is not happening with CuQuantum since it is actually relying on system libraries, and not distributing them itself.
However, it yields that it is not a CuQuantum problem, but more related to the Nvidia libraries installation. Which is then part of the user environment (almost as managing their own Linux distribution).

I would not make this part of Qibotn, since it is not at all specific to the installation of anything Qibo related, but it is just about the GPU libs installation. Which should be referred to the docs of those libraries themselves. And there is no strong reason to expect a downstream package to maintain a guide specific for them

[liweintu]

Thanks @scarrazza and @alecandido for the ideas.

Indeed, CuQuantum does ship compiled extensions. The target .so file for linking, libcutensornet, lives in my conda environment: ~/miniforge3/envs/qibotn_cuda12_py312/lib/python3.12/site-packages/cuquantum/lib. From this path, it seems that it's CuQuantum who installed the libcutensornet library, not Linux or Nvidia's other toolkits.

So, I understand that it's not part of Qibotn, it's just about complying with libs installation and locations. But, I did try updating my PYTHONPATH to include this directory, however, it didn't work, still couldn't find the target library file. So, I guess we have to resort back to adding some document about updating the environment variables for locating the libraries? If agreeable, which environment variable will you recommend? @alecandido

[alecandido]

PYTHONPATH is only suitable to locate Python packages. And these libraries are not Python packages, but dynamic libraries, which are then located at runtime by dlopen().
dlopen() itself is provided by the libdl library which is part of glibc (or any equivalent C library distribution).
And that's why you need to use LD_LIBRARY_PATH, unless the path itself is registered in the binaries you are using (which is not, if you're locating it at runtime).
It is a good practice to avoid as much as possible to rely on runtime libraries' location. But it's not your choice, in this case, since you're not compiling the binaries of CuQuantum yourself.

Then, since it is down to CuQuantum, this boils down to CuQuantum installation instructions, and they should document it.
Since we're now introducing it as a transitive, the most we should do it is to link their documentation in ours, saying to our users: "if you want to use Qibotn this way, you will use CuQuantum - make sure to follow their instructions for a correct installation", and possibly link.

We should avoid working out ourselves the way to install CuQuantum and its dependencies, since:

1. it should already be done (why repeating the same job twice?)
2. it may change over time, and our guide may become outdated
3. it may be different for different platforms - and it may have been documented for all of them in their docs, but most likely we will report only one single way, which may simplistic, and lead to further confusion

In conclusion: just link the CuQuantum installation instructions, and we are done on our side.

[liweintu]

Yeah, that's fair enough. I'll add the link to cuQuantum installation instructions. Anyway, this issue has been documented here as a reference solution in case anyone falling into the same issue in future wants a quick fix. Thanks @alecandido !

[alecandido]

Thank you for your investigation @liweintu!

Yeah, that's fair enough. I'll add the link to cuQuantum installation instructions. Anyway, this issue has been documented here as a reference solution in case anyone falling into the same issue in future wants a quick fix. Thanks @alecandido !

Still for the sake of logging it: do you believe the information in cuQuantum installation's guide could be enough (possibly with a mild effort from the user) to achieve a consistent environment?

It is supposed to be, but this does not mean that it is. And I did not confirm it myself.

[liweintu]

Another issue is encountered below when running pytest test_cuquantum_cutensor_backend.py:

FAILED test_cuquantum_cutensor_backend.py::test_expectation[2] - AttributeError: 'NumpyBackend' object has no attribute 'calculate_expectation_state'

Again, Gemini came to the rescue, and found the cause below.

The error is caused by a change in qibo's NumpyBackend API: the calculate_expectation_state method has been replaced with expectation_value in newer versions of Qibo.

A quick fix of replacing calls to numpy_backend.calculate_expectation_state with numpy_backend.expectation_value resolves this issue. The changes have been applied to these following files, after which, pytest can pass successfully.

• test_expectation.py
• test_cuquantum_cutensor_backend.py
• test_cuquantum_cutensor_mpi_backend.py

[liweintu]

Still for the sake of logging it: do you believe the information in cuQuantum installation's guide could be enough (possibly with a mild effort from the user) to achieve a consistent environment?

It is supposed to be, but this does not mean that it is. And I did not confirm it myself.

From my prior experience of installing cuQuantum, I haven't encountered this linking issue, so I assume its installation has taken care of properly locating the needed libraries. If some one who follows their installation guide unfortunately does land up hitting this linking issue, then yes, some mild effort is needed for him/her to set the environment right to link to the libraries, I guess.