Issue with mounting agent script when running Beobench inside dind container

[rdnfn]

Problem

If the Beobench repo is mounted inside a docker-inside-docker/docker in docker/dind container, and then a beobench run command is run inside this dind container, the agent script is not mounted properly. This is likely due to Docker requiring the original host mount paths instead of the paths inside the dind container. See this and this stackoverflow answer. Note that this does not appear to affect dind instances with their own docker daemon (e.g. the Beobench dev container).

The output from GitHub actions that uses dind container (below, https://github.com/rdnfn/beobench/runs/7187183104?check_suite_focus=true) shows that this failure is cryptic. The failure seems to work as follows:

• It says it can't find __main__ module, because the mounted agent script is mounted as an empty directory
• Mounting as an empty directory is the default behaviour of docker if the file path is incorrect
• The file path is incorrect because the path needs to be correct on the host machine not the dind container (but there is not obvious way to know this inside the container)

Output:

Beobench⚡️Experiment gym image build finished.
Beobench⚡️Absolute agent file path: /__w/beobench/beobench/beobench/data/agents/random_action.py
Beobench⚡️Executing docker command: docker run -v /__w/beobench/beobench/beobench_results:/root/beobench_results --rm --shm-size=4gb --name auto_beobench_experiment_52f4f6 -v /__w/beobench/beobench/beobench_results/tmp/config.yaml:/root/beobench_configs/config.yaml:ro -v /__w/beobench/beobench/beobench/data/agents/random_action.py:/root/beobench_configs/random_action.py:ro beobench_energym_complete:0.5.2 /bin/bash -c export WANDB_API_KEY= && beobench run --config="{'agent': {'origin': 'random_action', 'config': {'run_or_experiment': 'PPO', 'stop': {'timesteps_total': 2}, 'config': {'lr': 0.005, 'model': {'fcnet_activation': 'relu', 'fcnet_hiddens': [256, 256, 256, 256], 'post_fcnet_activation': 'tanh'}, 'batch_mode': 'complete_episodes', 'gamma': 0.999, 'horizon': 96, 'soft_horizon': True, 'metrics_smoothing_episodes': 5, 'framework': 'torch', 'num_workers': 1}}}, 'env': {'gym': 'energym', 'config': {'days': 365, 'name': 'Apartments2Thermal-v0', 'gym_kwargs': {'max_episode_length': 35040, 'normalize': True, 'step_period': 15, 'ignore_reset': True}, 'weather': 'ESP_CT_Barcelona'}}, 'wrappers': [], 'general': {'local_dir': './beobench_results', 'wandb_project': None, 'wandb_entity': None, 'wandb_group': None, 'mlflow_name': None, 'use_gpu': False, 'docker_shm_size': '4gb', 'force_build': True, 'use_no_cache': False, 'dev_path': None, 'docker_flags': None, 'beobench_extras': 'extended', 'num_samples': 1, 'version': '0.5.2'}, 'autogen': {'run_id': '7eedbd9885084bf499cfd2d8922b3b6e', 'random_seed': 8718186}}" --no-additional-container && bash
Beobench⚡️container:Beobench⚡️Starting experiment run ...
Beobench⚡️container:Beobench⚡️Running experiment in container with 'Apartments2Thermal-v0' environment from 'energym' gym.
Beobench⚡️container:Beobench⚡️Using agent from random_action. Sample 1 of 1.
Beobench⚡️container:Beobench⚡️Running agent script.
Beobench⚡️container: /usr/local/bin/python: can't find '__main__' module in '/root/beobench_configs/random_action.py'
Beobench⚡️container: Traceback (most recent call last):
Beobench⚡️container:   File "/usr/local/bin/beobench", line 8, in <module>
Beobench⚡️container:     sys.exit(cli())
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
Beobench⚡️container:     return self.main(*args, **kwargs)
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
Beobench⚡️container:     rv = self.invoke(ctx)
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1657, in invoke
Beobench⚡️container:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
Beobench⚡️container:     return ctx.invoke(self.callback, **ctx.params)
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
Beobench⚡️container:     return __callback(*args, **kwargs)
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/beobench/cli.py", line 122, in run
Beobench⚡️container:     beobench.experiment.scheduler.run(
Beobench⚡️container:   File "/usr/local/lib/python3.8/dist-packages/beobench/experiment/scheduler.py", line 175, in run
Beobench⚡️container:     subprocess.check_call(args)
Beobench⚡️container:   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
Beobench⚡️container:     raise CalledProcessError(retcode, cmd)
Beobench⚡️container: subprocess.CalledProcessError: Command '['python', '/root/beobench_configs/random_action.py']' returned non-zero exit status 1.

Potential Solution

This is difficult problem to solve reliably. It might be possible to pass a relative path to Beobench to correct any mounting paths. Overall, this is not a priority right now as this is a fairly special use-case.