bug(core): unable to run ray_tune operation with optuna using the latest ado image

[VassilisVassiliadis]

Issue Description

I'm unable to create ray_tune operations, with optuna, when I use the latest ado image (ado>=2.0.0).

I tested the following two tags and confirmed that both of them raise the error below:

• 2.0.0-ofed-py311-cu121-ofed2410v1140
• main-ofed-py311-cu129-ofed2410v1140

How to reproduce

Steps to reproduce the behavior:

1. Create a ray_tune/optuna ado space
2. Create the operation
3. The operation fails

Expected behaviour

The operation should work. It works on my MacOs just fine.

Screenshots/Logs

2026-07-27 03:50:09,849 DEBUG MainThread orchestrate_core: _run_operation_harness: Operation ray_tune@2.0.4-optuna-1fcb7a exited normally with status message='Unexpected exception when running RayTune operation: \x1b[36mray::tune()\x1b[39m (pid=14952, ip=10.48.27.233)\n File "/path/virtualenv/lib/python3.11/site-packages/ado_ray_tune/operator.py", line 611, in tune\n raise e\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ado_ray_tune/operator.py", line 608, in tune\n results = tuner.fit()\n ^^^^^^^^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/tuner.py", line 368, in fit\n return self._local_tuner.fit()\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py", line 530, in fit\n analysis = self._fit_internal(trainable, param_space)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py", line 646, in _fit_internal\n analysis = run(\n ^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/tune.py", line 866, in run\n callbacks = _create_default_callbacks(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/utils/callback.py", line 130, in _create_default_callbacks\n callbacks.append(TBXLoggerCallback())\n ^^^^^^^^^^^^^^^^^^^\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/ray/tune/logger/tensorboardx.py", line 38, in init\n from tensorboardX import SummaryWriter\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/tensorboardX/init.py", line 4, in \n from .global_writer import GlobalSummaryWriter\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/tensorboardX/global_writer.py", line 4, in \n from .writer import SummaryWriter\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/tensorboardX/writer.py", line 17, in \n from .event_file_writer import EventFileWriter\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/tensorboardX/event_file_writer.py", line 26, in \n from .record_writer import RecordWriter, directory_check\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/tensorboardX/record_writer.py", line 14, in \n import boto3\n File "/home/ray/anaconda3/lib/python3.11/site-packages/boto3/init.py", line 17, in \n from boto3.session import Session\n File "/home/ray/anaconda3/lib/python3.11/site-packages/boto3/session.py", line 17, in \n import botocore.session\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/session.py", line 26, in \n import botocore.client\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/client.py", line 15, in \n from botocore import waiter, xform_name\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/waiter.py", line 18, in \n from botocore.docs.docstring import WaiterDocstring\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/docs/init.py", line 15, in \n from botocore.docs.service import ServiceDocumenter\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/docs/service.py", line 14, in \n from botocore.docs.client import (\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/docs/client.py", line 18, in \n from botocore.docs.example import ResponseExampleDocumenter\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/docs/example.py", line 13, in \n from botocore.docs.shape import ShapeDocumenter\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/docs/shape.py", line 19, in \n from botocore.utils import is_json_value_header\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/utils.py", line 38, in \n import botocore.httpsession\n File "/home/ray/anaconda3/lib/python3.11/site-packages/botocore/httpsession.py", line 45, in \n from urllib3.contrib.pyopenssl import (\n File "/tmp/ray/session_2026-07-27_02-55-07_737123_1/runtime_resources/uv/5f926b2b24049d694b7d381f49a4a6c7bc4dbced/virtualenv/lib/python3.11/site-packages/urllib3/contrib/pyopenssl.py", line 43, in \n import OpenSSL.SSL # type: ignore[import-not-found]\n ^^^^^^^^^^^^^^^^^^\n File "/home/ray/anaconda3/lib/python3.11/site-packages/OpenSSL/init.py", line 8, in \n from OpenSSL import SSL, crypto\n File "/home/ray/anaconda3/lib/python3.11/site-packages/OpenSSL/SSL.py", line 42, in \n from OpenSSL.crypto import (\n File "/home/ray/anaconda3/lib/python3.11/site-packages/OpenSSL/crypto.py", line 783, in \n class X509Extension:\n File "/home/ray/anaconda3/lib/python3.11/site-packages/OpenSSL/crypto.py", line 867, in X509Extension\n _lib.GEN_EMAIL: "email",\n ^^^^^^^^^^^^^^\nAttributeError: module 'lib' has no attribute 'GEN_EMAIL'' recorded_at=datetime.datetime(2026, 7, 27, 10, 50, 8, 424180, tzinfo=datetime.timezone.utc) event=<OperationResourceEventEnum.FINISHED: 'finished'> exit_state=<OperationExitStateEnum.ERROR: 'error'>

Python/ado/system info

Please include the output of:

python --version=3.11.20
ado version=2.0.0
Your OS=linux

Additional information

I think this is related to the openssl package in the base container layer of the ado image.

[michael-johnston]

I encountered this error a few weeks ago.

At some stage a package - something pulling cryptography - did not set the correct dependency on PyOpenSSL. So uv sees the base env PyOpenSSL and thinks its okay, but its not, leading to this error.

The solution is to install the correct version of PyOpenSSL and possible boto3 in the runtime-env. However the problem disappeared for me on Morrigan so I never had to do it.

Do you have the same problem on Morrigan?

[VassilisVassiliadis]

I haven't tested Morrigan. Alessandro also suggested I install PyOpenSSL myself in the ray-venv and it's on my to-do list.