"*** stack smashing detected ***: terminated" when visualising

[lam206]

Edit: I have tried option 2 for getting the dependencies from the docs, namely a virtual env (I used conda instead of pip) on the host directly and that allows me to -visualise without the error. I will continue developing with the conda environment, so don't need a fix for this any longer. I will leave this discussion up in case in any case.

python3 -m rrc_2022_datasets.evaluate_pre_stage push rrc2022.example.TorchPushPolicy --output results_push.json -v causes the stack smashing error in the title. --visualization causes the same error. For both the push as well as the pull task with the respective example policies. On the other hand, everything works without the -v flag.

I've followed the instructions on the wiki on how to setup apptainer. I can get a shell and am isolating it. The example visualisation from the ros2 trifinger_simulation demo_trifinger_platform.py (are these remains from last years challenge?) work fine. I ran source /setup.bash and installed the example package in a venv with the newest pip (pip install -U pip inside the venv inside apptainer).

I've only come across this error message and hacking in C. This happens when one overrides the canary, which is checked before a function returns in a binary with canary on. I have never, however, encountered this error in Python and I guess some of the underlying libraries in the image must have some sort of buffer overflow going on. Anyone else had this issue?

The full output from python3 -m rrc_2022_datasets.evaluate_pre_stage push rrc2022.example.TorchPushPolicy --output results_push.json -v is:

INFO:root:import TorchPushPolicy from rrc2022.example
/home/pytorch/rrc22/my_env/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
Using flattened observations
/usr/local/lib/python3.8/dist-packages/gym/spaces/box.py:112: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
Start episode 0
startThreads creating 1 threads.
starting thread 0
started thread 0 
argc=2
argv[0] = --unused
argv[1] = --start_demo_name=Physics Server
ExampleBrowserThreadFunc started
X11 functions dynamically loaded using dlopen/dlsym OK!
X11 functions dynamically loaded using dlopen/dlsym OK!
*** stack smashing detected ***: terminated
Aborted (core dumped)```

[lam206]

Note that I have a very old NVIDIA GPU that modern PyTorch does not support. This is also seen in the error message. But for visualisation purposes this shouldn't be an issue. The ros2 trifinger_simulation demo as well as the policy evaluation without -v worked. Edit: I will only be using the CPU for PyTorch on this machine.

[luator]

Hello,
Did you run Apptainer with the --nv flag, if not, can you try if this solves the problem? It makes Nvidia drivers/libraries of the host system available in the container and is typically needed for GPU-related things to work.

[lam206]

Yes, I'm certain I did spawn the Apptainer shell with the --nv flag. I will double-check when I get home this evening.