From db2e310fe7383ce75abee61b8bba523c49f9c26f Mon Sep 17 00:00:00 2001 From: Donghwan Shin Date: Thu, 18 Dec 2025 16:28:53 +0000 Subject: [PATCH 01/27] final pass --- paper.md | 32 +++++++++++--------------------- 1 file changed, 11 insertions(+), 21 deletions(-) diff --git a/paper.md b/paper.md index 4f4a383..298b62c 100644 --- a/paper.md +++ b/paper.md @@ -77,40 +77,30 @@ Therefore, `CAWSR` is designed to minimise such nondeterminism throughout the ev The evaluation pipeline is engineered to be fully synchronous, minimising unintentional non-determinism to facilitate reproducible results. However, it is noted that minor variations may still persist due to inherent non-determinism in upstream dependencies, such as the driving simulator or the driving agent itself [@9793395; @osikowicz2025empirically]. -![Internal component diagram of CAWSR.](./docs/resources/component_diagram.pdf) +![Internal component diagram of CAWSR.\label{fig:components}](./docs/resources/component_diagram.pdf) -Figure 1 shows architecture of `CAWSR` with the fundamental components of the framework. **CarlaClient**, a native CARLA PythonAPI class, establishes a TCP connection to the simulator via a host *IP* and *port*. As the framework’s sole communication link with CARLA, it allows `CAWSR` modules to extract data and spawn entities by interacting with the internal server. +\autoref{fig:components} illustrates the `CAWSR` architecture and its fundamental components. The framework operates through four primary modules: -**JSON parser** translates *scenario_definition* into a Behavior Tree (BT) by extracting the route and its associated trigger events. These trees are constructed using Scenario Runner’s `Atomic Behaviours` and `Atomic Conditions`. Serving as the framework's building blocks, these elements represent discrete CARLA actions and variables—such as spawning a pedestrian—to define the scenario's logic. +- CarlaClient: A native CARLA PythonAPI class that establishes a TCP connection (via host IP and port). It serves as the framework's exclusive interface for extracting simulation data and spawning entities. -**ScenarioManager** manages the setup and execution loop, using **CarlaClient** to spawn entities. During each loop, it executes the behavior tree to update actor states and evaluate conditions. It then triggers a simulation tick, advancing CARLA’s internal clock and generating a snapshot. This snapshot is passed to **Agent**, which monitors Autoware’s internal state and route data. Upon initialisation, Agent connects to Autoware via ROS2. Subsequently, at each step, **CarlaBridge** [@carlaautowarebridge] extracts snapshot data, transforms sensor inputs into Autoware’s coordinate system, and publishes them. Finally, Autoware processes this data and issues control commands, which are applied to the ego vehicle. +- JSON Parser: Translates the *scenario_definition* (see \autoref{fig:scenario_domain}) into a Behavior Tree (BT). It utilises Scenario Runner's *Atomic Behaviours* and *Atomic Conditions* as modular primitives to define discrete actions (e.g., spawning pedestrians) and logic triggers. -The internal loop within ScenarioManager continues executing until one of the following termination conditions is met, as defined by the CARLA Leaderboard evaluation criteria [@carla_leaderboard], shown in Table 1. +- ScenarioManager: Orchestrates the simulation loop by evaluating the BT to update actor states and triggering CARLA simulation ticks. Execution terminates based on CARLA Leaderboard criteria [@carla_leaderboard], as summarised in \autoref{tab:termination_criteria}. Post-execution, the module calculates the Driving Score (DS) according to the official leaderboard metrics. + +- Agent and CarlaBridge: The Agent manages the ROS2 connection to Autoware. At each timestep, the CarlaBridge [@carlaautowarebridge] transforms CARLA snapshots and sensor data into the Autoware coordinate system. Autoware processes these inputs to issue control commands, which the Agent then applies to the ego vehicle. | Termination Criteria | Description | |----------------------|---------------------------------------------------| | Route_Completion | Agent reached the end of the route. | | Actor_Blocked | Agent is blocked, not moving for 180s. | | Simulation_Timeout | No client-server communication established (30s). | -: Termination Criteria of each scenario within CAWSR. - -The same set of standard evalutaion critera is employed to calculate the driving score (DS) for each scenario execution, shown in Table 2. - -| Evaluation Criteria | -|---------------------------------------| -| Collisions_with_pedestrians | -| Collisions_with_other_vehicles | -| Collisions_with_static_elements | -| Running_a_red_light | -| Failure_to_yield_to_emergency_vehicle | -| Running_a_stop_sign | -: Evaluation Criteria of each scenario within CAWSR, per the CARLA Leaderboard [@carla_leaderboard]. Each criteria applies a fixed penality to the DS. +: Termination Criteria of each scenario within CAWSR.\label{tab:termination_criteria} -![Scenario definition domain model.](./docs/resources/scenario_domain.pdf) - -To facilitate development, we introduce a new domain model for the definition of route-based scenarios within CARLA, described in Figure 4, alongside a `JSON` implementation. +To facilitate development, we introduce a new domain model for the definition of route-based scenarios within CARLA, described in \autoref{fig:scenario_domain}, alongside a `JSON` implementation. This model is based on the format introduced by Scenario Runner, facilitating support between both frameworks. +![Scenario definition domain model.\label{fig:scenario_domain}](./docs/resources/scenario_domain.pdf) + # Conclusion To summarise, `CAWSR` provides ADS testing research community an easy to use Autoware evaluation pipeline. From 5adf588ae232b09ffe628066dccdc27a4fb0fa42 Mon Sep 17 00:00:00 2001 From: Donghwan Shin Date: Thu, 18 Dec 2025 16:34:30 +0000 Subject: [PATCH 02/27] Add GitHub Actions workflow for PDF draft generation --- .github/workflows/draft-pdf.yml | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 .github/workflows/draft-pdf.yml diff --git a/.github/workflows/draft-pdf.yml b/.github/workflows/draft-pdf.yml new file mode 100644 index 0000000..ffbc079 --- /dev/null +++ b/.github/workflows/draft-pdf.yml @@ -0,0 +1,30 @@ +name: Draft PDF +on: [push] + +jobs: + paper: + runs-on: ubuntu-latest + name: Paper Draft + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Build draft PDF + uses: openjournals/openjournals-draft-action@master + with: + journal: joss + # This should be the path to the paper within your repo. + paper-path: paper.md + - name: Upload + uses: actions/upload-artifact@v4 + with: + name: paper + # This is the output path where Pandoc will write the compiled + # PDF. Note, this should be the same directory as the input + # paper.md + path: paper.pdf + - name: Commit PDF to repository + uses: EndBug/add-and-commit@v9 + with: + message: '(auto) Paper PDF Draft' + # This should be the path to the paper within your repo. + add: 'paper.pdf' # 'paper/*.pdf' to commit all PDFs in the paper directory From 1b62f74a983b7b6ac8c91df02022d4ffa8fb142a Mon Sep 17 00:00:00 2001 From: Olek Osikowicz Date: Fri, 19 Dec 2025 16:46:05 +0000 Subject: [PATCH 03/27] Refraze introduction to blackbox ADS testing --- paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paper.md b/paper.md index 298b62c..1038896 100644 --- a/paper.md +++ b/paper.md @@ -44,7 +44,7 @@ Consequently, simulation-based testing has become essential, allowing researcher Among these tools, CARLA [@carla_sim] has become the de-facto standard in the research community due to its rich ecosystem of open-source tools, benchmarks, and documentation. Currently, the standard for evaluating ADS in CARLA is the CARLA Leaderboard and its engine, Scenario Runner (SR) [@carla_scenario_runner_2025]. -This framework is typically used to test "black-box" driving agents, such as those based on Vision Language Models or Reinforcement Learning (DS: add refs here). +This framework is typically used to test "black-box" driving agents, such as ML-based systems which expose only sensor-level inputs and driving control outputs. By running a set of predefined, challenging driving scenarios, researchers can systematically assess agent performance using common metrics like driving score, infractions, and route completion. However, applying this testing framework to industry-grade ADS, such as Autoware [@kato2018autoware] or Apollo [@apollo], remains difficult. Although communication bridges exist between CARLA and these systems [@guardstrikelab_2023_carla; @carlaautowarebridge], they lack native support for scenario execution engines, which limits their utility for scenario-based testing. From 04569012031b6ec584edd7c909bf9213123266a3 Mon Sep 17 00:00:00 2001 From: Olek Osikowicz Date: Fri, 19 Dec 2025 17:04:05 +0000 Subject: [PATCH 04/27] Clarify note on deployment --- paper.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/paper.md b/paper.md index 1038896..6eb8182 100644 --- a/paper.md +++ b/paper.md @@ -44,7 +44,7 @@ Consequently, simulation-based testing has become essential, allowing researcher Among these tools, CARLA [@carla_sim] has become the de-facto standard in the research community due to its rich ecosystem of open-source tools, benchmarks, and documentation. Currently, the standard for evaluating ADS in CARLA is the CARLA Leaderboard and its engine, Scenario Runner (SR) [@carla_scenario_runner_2025]. -This framework is typically used to test "black-box" driving agents, such as ML-based systems which expose only sensor-level inputs and driving control outputs. +This framework is typically used to test "black-box" driving agents, such as those based on Vision Language Models or Reinforcement Learning (DS: add refs here). By running a set of predefined, challenging driving scenarios, researchers can systematically assess agent performance using common metrics like driving score, infractions, and route completion. However, applying this testing framework to industry-grade ADS, such as Autoware [@kato2018autoware] or Apollo [@apollo], remains difficult. Although communication bridges exist between CARLA and these systems [@guardstrikelab_2023_carla; @carlaautowarebridge], they lack native support for scenario execution engines, which limits their utility for scenario-based testing. @@ -70,7 +70,7 @@ Therefore, `CAWSR` is designed to minimise such nondeterminism throughout the ev # Tool Overview -`CAWSR` is a fully synchronous testing framework that directly integrates the CARLA simulator, Scenario Runner (as the scenario executor), and Autoware (as the System Under Test) to facilitate autonomous driving testing research. The tool is distributed as a Docker container designed and currently supports two modes of operation: +`CAWSR` is a fully synchronous testing framework that directly integrates the CARLA simulator, Scenario Runner (as the scenario executor), and Autoware (as the System Under Test) to facilitate autonomous driving testing research. The tool is distributed as a containerized deployment using Docker and currently supports two modes of operation: 1. *Scenario Generation Mode:* Enables the dynamic generation and execution of scenarios (e.g. iterative scenario generation) provided by a user-defined algorithm. This is particularly useful for assessing the performance of new simulation-based ADS testing techniques. 2. *Benchmark Mode:* Allows the execution of a predefined set of scenario definitions provided by the user. This is useful for standardised evaluations and comparisons between different driving agents. From 095b0f481f4f8e9c7661429f09356f5ca30aaa69 Mon Sep 17 00:00:00 2001 From: Olek Osikowicz Date: Fri, 19 Dec 2025 17:05:39 +0000 Subject: [PATCH 05/27] Reversing accidental change --- paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paper.md b/paper.md index 6eb8182..ab46dfb 100644 --- a/paper.md +++ b/paper.md @@ -44,7 +44,7 @@ Consequently, simulation-based testing has become essential, allowing researcher Among these tools, CARLA [@carla_sim] has become the de-facto standard in the research community due to its rich ecosystem of open-source tools, benchmarks, and documentation. Currently, the standard for evaluating ADS in CARLA is the CARLA Leaderboard and its engine, Scenario Runner (SR) [@carla_scenario_runner_2025]. -This framework is typically used to test "black-box" driving agents, such as those based on Vision Language Models or Reinforcement Learning (DS: add refs here). +This framework is typically used to test "black-box" driving agents, such as ML-based systems which expose only sensor-level inputs and driving control outputs. By running a set of predefined, challenging driving scenarios, researchers can systematically assess agent performance using common metrics like driving score, infractions, and route completion. However, applying this testing framework to industry-grade ADS, such as Autoware [@kato2018autoware] or Apollo [@apollo], remains difficult. Although communication bridges exist between CARLA and these systems [@guardstrikelab_2023_carla; @carlaautowarebridge], they lack native support for scenario execution engines, which limits their utility for scenario-based testing. From 6e9793cc968746e315bc4a98c0a1660a064df739 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 22 Dec 2025 21:27:04 +0000 Subject: [PATCH 06/27] updated QoS depth for some topics; marked old code for removal --- srunner/autoagents/autoware_agent.py | 6 ++-- .../autoware_carla_interface/carla_ros.py | 29 ++++++------------- 2 files changed, 12 insertions(+), 23 deletions(-) diff --git a/srunner/autoagents/autoware_agent.py b/srunner/autoagents/autoware_agent.py index c6979ee..5178c1e 100644 --- a/srunner/autoagents/autoware_agent.py +++ b/srunner/autoagents/autoware_agent.py @@ -44,10 +44,10 @@ def setup(self, config: EnvironmentConfig) -> None: """ rclpy.init(args=None) - + self.config = config - self._node = rclpy.create_node('cawsr_bridge') + self._node = rclpy.create_node("cawsr_bridge") self.autoware_state = autoware_state.AutowareState("ego_vehicle", None) @@ -171,7 +171,7 @@ def run_step(self) -> None: if self.initialised: logger.info("Set agent route!") - # check if the current route is set + # check if the current route is set and we can publish engage if self.autoware_state.route_set() and not self.autoware_state.sent_engage: self.autoware_node.publish_engage(True) diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 1776dd2..2e857ed 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -55,8 +55,6 @@ SensorInterface, ) -from srunner.tools.CARLA_manager import CARLAManager - class carla_ros2_interface(object): def __init__(self, node): @@ -86,26 +84,26 @@ def __init__(self, node): sensor: datetime.datetime.now() for sensor in self.sensor_frequencies } - self.game_time_offset = ( - CARLAManager.FIXED_DELTA_SECONDS * 3 - ) # offset to account for initilisation ticks - frac, whole = math.modf(self.game_time_offset) + # old code, to be removed -> currently tested + # self.game_time_offset = ( + # CARLAManager.FIXED_DELTA_SECONDS * 3 + # ) # offset to account for initilisation ticks + # frac, whole = math.modf(self.game_time_offset) self.ros2_node = node - # Publish clock with larger queue to prevent drops + # publish clock with larger queue to prevent drops self.clock_publisher = self.ros2_node.create_publisher(Clock, "/clock", 50) obj_clock = Clock() - obj_clock.clock = Time(sec=int(whole), nanosec=int(frac * 1e9)) + obj_clock.clock = Time(sec=int(0)) self.clock_publisher.publish(obj_clock) - # Sensor Config (Edit your sensor here) + # load sensor config and create publishers sensors_config = pathlib.Path( "srunner/autoagents/autoware_carla_interface/objects/sensors.json" ) self.sensors = json.load(open(sensors_config.absolute())) - # Subscribing Autoware Control messages and converting to CARLA control self.sub_control = self.ros2_node.create_subscription( ActuationCommandStamped, "/control/command/actuation_cmd", @@ -168,9 +166,6 @@ def __init__(self, node): ) pass - # add to multi threaded executor instead - # self.spin_thread = threading.Thread(target=rclpy.spin, args=(self.ros2_node,)) - def __call__(self): input_data = self.sensor_interface.get_data() timestamp = GameTime.get_time() @@ -487,16 +482,13 @@ def ego_status(self): def run_step(self, input_data, timestamp): self.timestamp = timestamp - # Publish clock FIRST to update transform system before sensor data arrives - # This prevents "extrapolation into the future" errors seconds = int(self.timestamp) nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) obj_clock = Clock() obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) self.clock_publisher.publish(obj_clock) - # Small delay to allow clock to propagate to transform system - time.sleep(0.005) + time.sleep(0.05) # publish data of all sensors for key, data in input_data.items(): @@ -512,10 +504,7 @@ def run_step(self, input_data, timestamp): else: self.ros2_node.get_logger().info("No Publisher for [{key}] Sensor") - # Publish ego vehicle status self.ego_status() - - # Small delay to ensure large messages (LiDAR) are fully transmitted time.sleep(0.005) return self.current_control From 96a3f5c5b94d115f82b92b4f080f0b63ac1aae58 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 22 Dec 2025 22:33:10 +0000 Subject: [PATCH 07/27] reverted QoS depth --- README.md | 5 +++-- .../autoware_carla_interface/carla_ros.py | 18 +++++++++--------- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index b0c94f8..1f730e2 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,9 @@ docker pull ghcr.io/intelligent-testing-lab/autoware:latest Autoware and ROS use a custom messaging interface for communcation, known as DDS. They support various implementations, but they all rely on specific network settings to enable maximum data transfer. Save the following command in `setup.sh`, allow it to be executable `chmod +x setup.sh` and run. ```bash -# Increase the maximum receive buffer size for network packets +# Increase the maximum receive and send buffer size for network packets, allowing our containers to communicate sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB +sudo sysctl -w net.core.rmem_max=2147483647 # IP fragmentation settings sudo sysctl -w net.ipv4.ipfrag_time=3 # in seconds, default is 30 s @@ -85,7 +86,7 @@ Scenario Definition We use a custom implementation of a scenario definition in JSON. We have included a scenario domain model, as well as plenty of examples in the CAWSR Workspace repository `scenarios/examples/`. Domain Model: -![Domain Model](./docs/resources/scenario_domain.png) +![Domain Model](./docs/resources/scenario_domain.pdf) Contributing ------------ diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 2e857ed..546aa03 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -118,32 +118,32 @@ def __init__(self, node): self.current_control = carla.VehicleControl() self.pub_pose_with_cov = self.ros2_node.create_publisher( - PoseWithCovarianceStamped, "/sensing/gnss/pose_with_covariance", 10 + PoseWithCovarianceStamped, "/sensing/gnss/pose_with_covariance", 1 ) self.pub_vel_state = self.ros2_node.create_publisher( - VelocityReport, "/vehicle/status/velocity_status", 10 + VelocityReport, "/vehicle/status/velocity_status", 1 ) self.pub_steering_state = self.ros2_node.create_publisher( - SteeringReport, "/vehicle/status/steering_status", 10 + SteeringReport, "/vehicle/status/steering_status", 1 ) self.pub_ctrl_mode = self.ros2_node.create_publisher( - ControlModeReport, "/vehicle/status/control_mode", 10 + ControlModeReport, "/vehicle/status/control_mode", 1 ) self.pub_gear_state = self.ros2_node.create_publisher( - GearReport, "/vehicle/status/gear_status", 10 + GearReport, "/vehicle/status/gear_status", 1 ) self.pub_actuation_status = self.ros2_node.create_publisher( - ActuationStatusStamped, "/vehicle/status/actuation_status", 10 + ActuationStatusStamped, "/vehicle/status/actuation_status", 1 ) for sensor in self.sensors["sensors"]: self.id_to_sensor_type_map[sensor["id"]] = sensor["type"] if sensor["type"] == "sensor.camera.rgb": self.pub_camera = self.ros2_node.create_publisher( - Image, "/sensing/camera/traffic_light/image_raw", 10 + Image, "/sensing/camera/traffic_light/image_raw", 1 ) self.pub_camera_info = self.ros2_node.create_publisher( - CameraInfo, "/sensing/camera/traffic_light/camera_info", 10 + CameraInfo, "/sensing/camera/traffic_light/camera_info", 1 ) elif sensor["type"] == "sensor.lidar.ray_cast": if sensor["id"] in self.sensor_frequencies: @@ -158,7 +158,7 @@ def __init__(self, node): ) elif sensor["type"] == "sensor.other.imu": self.pub_imu = self.ros2_node.create_publisher( - Imu, "/sensing/imu/tamagawa/imu_raw", 10 + Imu, "/sensing/imu/tamagawa/imu_raw", 1 ) else: self.ros2_node.get_logger().info( From 78bdcf5a2b37b73d3d1826e68336440884b7bb73 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 22 Dec 2025 22:54:40 +0000 Subject: [PATCH 08/27] removed old code, updated QoS depth and timings --- README.md | 2 +- .../autoware_carla_interface/carla_ros.py | 15 +++------------ 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 1f730e2..b6bfcf2 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Autoware and ROS use a custom messaging interface for communcation, known as DDS ```bash # Increase the maximum receive and send buffer size for network packets, allowing our containers to communicate sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB -sudo sysctl -w net.core.rmem_max=2147483647 +sudo sysctl -w net.core.wmem_max=2147483647 # IP fragmentation settings sudo sysctl -w net.ipv4.ipfrag_time=3 # in seconds, default is 30 s diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 546aa03..6f219a4 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -84,16 +84,9 @@ def __init__(self, node): sensor: datetime.datetime.now() for sensor in self.sensor_frequencies } - # old code, to be removed -> currently tested - # self.game_time_offset = ( - # CARLAManager.FIXED_DELTA_SECONDS * 3 - # ) # offset to account for initilisation ticks - # frac, whole = math.modf(self.game_time_offset) - self.ros2_node = node - # publish clock with larger queue to prevent drops - self.clock_publisher = self.ros2_node.create_publisher(Clock, "/clock", 50) + self.clock_publisher = self.ros2_node.create_publisher(Clock, "/clock", 10) obj_clock = Clock() obj_clock.clock = Time(sec=int(0)) self.clock_publisher.publish(obj_clock) @@ -150,7 +143,7 @@ def __init__(self, node): self.pub_lidar[sensor["id"]] = self.ros2_node.create_publisher( PointCloud2, f"/sensing/lidar/{sensor['id']}/pointcloud_before_sync", - 10, + 5, # lower qos depth as using best_reliability ) else: self.ros2_node.get_logger().info( @@ -488,7 +481,7 @@ def run_step(self, input_data, timestamp): obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) self.clock_publisher.publish(obj_clock) - time.sleep(0.05) + time.sleep(0.005) # publish data of all sensors for key, data in input_data.items(): @@ -505,8 +498,6 @@ def run_step(self, input_data, timestamp): self.ros2_node.get_logger().info("No Publisher for [{key}] Sensor") self.ego_status() - time.sleep(0.005) - return self.current_control def shutdown(self): From a0783fd2d6af5145e7aa4bb37259c43439151e8f Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 22 Dec 2025 23:14:10 +0000 Subject: [PATCH 09/27] swapped ticking order --- .../autoware_carla_interface/carla_ros.py | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 6f219a4..82ba6a6 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -475,14 +475,6 @@ def ego_status(self): def run_step(self, input_data, timestamp): self.timestamp = timestamp - seconds = int(self.timestamp) - nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) - obj_clock = Clock() - obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) - self.clock_publisher.publish(obj_clock) - - time.sleep(0.005) - # publish data of all sensors for key, data in input_data.items(): sensor_type = self.id_to_sensor_type_map[key] @@ -497,7 +489,16 @@ def run_step(self, input_data, timestamp): else: self.ros2_node.get_logger().info("No Publisher for [{key}] Sensor") + time.sleep(0.01) # slight delay to ensure published messages are received + + seconds = int(self.timestamp) + nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) + obj_clock = Clock() + obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) + self.clock_publisher.publish(obj_clock) + self.ego_status() + return self.current_control def shutdown(self): From bb06fb2166615175ed2c24deec9f313f05da1eaf Mon Sep 17 00:00:00 2001 From: David Gasinski <118130264+david-gasinski@users.noreply.github.com> Date: Tue, 23 Dec 2025 00:22:04 +0000 Subject: [PATCH 10/27] Add GITHUB_TOKEN to draft PDF workflow --- .github/workflows/draft-pdf.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/draft-pdf.yml b/.github/workflows/draft-pdf.yml index ffbc079..3a490c9 100644 --- a/.github/workflows/draft-pdf.yml +++ b/.github/workflows/draft-pdf.yml @@ -8,6 +8,8 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + token: ${{ secrets.GITHUB_TOKEN }} - name: Build draft PDF uses: openjournals/openjournals-draft-action@master with: From 7063c3526974600dacf21fe2e0c28f2825faaaee Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Tue, 23 Dec 2025 00:24:10 +0000 Subject: [PATCH 11/27] updated README and minor fixes --- README.md | 10 +++++++--- docker/cyclonedds.xml | 1 - .../autoagents/autoware_carla_interface/carla_ros.py | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index b6bfcf2..f1a520c 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ docker pull ghcr.io/intelligent-testing-lab/autoware-scenario-runner:latest docker pull ghcr.io/intelligent-testing-lab/autoware:latest ``` -Autoware and ROS use a custom messaging interface for communcation, known as DDS. They support various implementations, but they all rely on specific network settings to enable maximum data transfer. Save the following command in `setup.sh`, allow it to be executable `chmod +x setup.sh` and run. +Autoware and ROS use a custom messaging interface for communication, known as DDS. They support various implementations, but they all rely on specific network settings to enable maximum data transfer. Save the following command in `setup.sh`, allow it to be executable `chmod +x setup.sh` and run. ```bash # Increase the maximum receive and send buffer size for network packets, allowing our containers to communicate sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB @@ -43,7 +43,7 @@ xhost +local:docker Using CAWSR ------------------------ -After completiting the prerequisite steps, clone the CAWSR workspace repository. To launch CAWSR, navigate to the CAWSR workspace and run `docker compose up`. +After completiting the prerequisite steps, clone the [CAWSR workspace](https://github.com/Intelligent-Testing-Lab/cawsr_workspace) repository. To launch CAWSR, navigate to the CAWSR workspace and run `docker compose up`. The structure of the workspace is as follows. ``` @@ -79,7 +79,6 @@ is called. To implement a custom algorithm, create a class than inherits from `B The algorithm will execute **runs** times. - Scenario Definition ------------------- @@ -88,6 +87,10 @@ We use a custom implementation of a scenario definition in JSON. We have include Domain Model: ![Domain Model](./docs/resources/scenario_domain.pdf) +Notes +------------ +Currently, traffic light recognition is disabled due to an issue with the [CARLA map format](https://github.com/autowarefoundation/autoware_universe/tree/main/simulator/autoware_carla_interface#traffic-light-recognition). This is a time consuming process, as each new traffic light requires the creation of a new objects within the CARLA Lanelet2 file that match the position of the PCD exactly. Once finished, we'll publish an updated Autoware image accordingly. + Contributing ------------ @@ -103,3 +106,4 @@ License ------- ScenarioRunner specific code is distributed under MIT License. +CAWSR specific code is distributed under MIT License. diff --git a/docker/cyclonedds.xml b/docker/cyclonedds.xml index b163dd7..7584c0f 100644 --- a/docker/cyclonedds.xml +++ b/docker/cyclonedds.xml @@ -14,7 +14,6 @@ 1MB - inf warning diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 82ba6a6..62d325f 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -489,7 +489,7 @@ def run_step(self, input_data, timestamp): else: self.ros2_node.get_logger().info("No Publisher for [{key}] Sensor") - time.sleep(0.01) # slight delay to ensure published messages are received + time.sleep(0.05) # 50ms delay to ensure published messages are received seconds = int(self.timestamp) nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) From 9a4dd92935124bd0498b1ee8627c0ded43cd82b7 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Thu, 25 Dec 2025 20:06:47 +0000 Subject: [PATCH 12/27] cleanup code + fix missing dois --- cawsr.py | 26 +++++++++++++++---- paper.bib | 12 ++++++--- .../autoware_carla_interface/carla_ros.py | 16 +++++------- srunner/scenariomanager/scenario_manager.py | 12 +++------ 4 files changed, 39 insertions(+), 27 deletions(-) diff --git a/cawsr.py b/cawsr.py index 4293d18..0306983 100644 --- a/cawsr.py +++ b/cawsr.py @@ -27,6 +27,7 @@ from typing import Optional, Union, Callable from srunner.scenariomanager.scenario_manager import ScenarioManager +from srunner.scenariomanager.timer import GameTime from srunner.tools.results_manager import ScenarioDefinitionManager from srunner.scenarios.route_scenario import RouteScenario from srunner.scenariomanager.carla_data_provider import CarlaDataProvider @@ -86,7 +87,7 @@ def __init__(self, cawsr_config: dict, carla_conf: CARLA) -> None: # manages results directories self.results_manager = ScenarioDefinitionManager() - # capture SIGINT for cleanp + # capture SIGINT for cleanup self._shutdown_requested = False if sys.platform != "win32": @@ -163,7 +164,8 @@ def run_scenario( self.ego_vehicles.append(actor) logger.info(f"Spawned ego with id: {actor.id}") - self.carla_world.tick() # client must tick to spawn actors + # client must tick to spawn actors + self._tick_carla() logger.info("Initialising Autoware...") agent_class_name = self.module_aw_agent.__name__.title().replace("_", "") @@ -188,7 +190,7 @@ def run_scenario( ego.prepare_ego(route[0][0]) # set location to first waypoint - self.carla_world.tick() + self._tick_carla() logger.info("Loading Traffic Manager...") tm_port = int(self._carla.TRAFFIC_MANAGER.PORT) # type: ignore @@ -265,6 +267,18 @@ def run_scenario( result_.put(result_dict) + def _tick_carla(self) -> None: + timestamp = None + world = CarlaDataProvider.get_world() + if world: + snapshot = world.get_snapshot() + if snapshot: + timestamp = snapshot.timestamp + if timestamp: + CarlaDataProvider.get_world().tick() + CarlaDataProvider.on_carla_tick() + GameTime.on_carla_tick(timestamp) + def _load_alg(self) -> type[BasicAlgorithm]: """Load an algorithm instance from mounted docker volume algorithms/ @@ -285,10 +299,12 @@ def _load_alg(self) -> type[BasicAlgorithm]: ) def run_algorithm(self) -> None: - """Executes CAWSR in algorithm mode. Every scenario""" + """Executes CAWSR in algorithm mode""" + # load the algorithm and scenario defintion optimisation_algorithm = self._load_alg() scenario = pathlib.Path(self._conf["algorithm"]["initial_definition"]) + # add some code here # if scenario = null (initial definition not given) # run the algorithm to generate a new, random scenario @@ -317,7 +333,7 @@ def run_algorithm(self) -> None: self.results_manager.last_scenario, env_config )[ 0 - ] # route id. Multiple routes currently aren't supported, so use first route + ] # route id. Multiple routes currently aren't supported, so use first route -> fix to use config json_definition = self._cawsr_process( route_config=route_config, diff --git a/paper.bib b/paper.bib index 90274ee..0eff17b 100644 --- a/paper.bib +++ b/paper.bib @@ -9,6 +9,7 @@ @inproceedings{carla_sim @inproceedings{osikowicz2025empirically, title = {Empirically evaluating flaky tests for autonomous driving systems in simulated environments}, author = {Osikowicz, Olek and McMinn, Phil and Shin, Donghwan}, + doi = {10.1109/FTW66604.2025.00009}, booktitle = {2025 IEEE/ACM International Flaky Tests Workshop (FTW)}, pages = {13--20}, year = {2025}, @@ -18,6 +19,7 @@ @inproceedings{osikowicz2025empirically @article{Jaeger2023ICCV, title = {Hidden Biases of End-to-End Driving Models}, author = {Bernhard Jaeger and Kashyap Chitta and Andreas Geiger}, + doi = {10.1109/iccv51070.2023.00757}, booktitle = {Proc. of the IEEE International Conf. on Computer Vision (ICCV)}, year = {2023} } @@ -26,6 +28,7 @@ @inproceedings{kato2018autoware title = {Autoware on board: Enabling autonomous vehicles with embedded systems}, author = {Kato, Shinpei and Tokunaga, Shota and Maruyama, Yuya and Maeda, Seiya and Hirabayashi, Manato and Kitsukawa, Yuki and Monrroy, Abraham and Ando, Tomohito and Fujii, Yusuke and Azumi, Takuya}, booktitle = {2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS)}, + doi = {10.1109/iccps.2018.00035}, pages = {287--296}, year = {2018}, organization = {IEEE} @@ -74,6 +77,7 @@ @article{carlaautowarebridge @inproceedings{tehrani2025pcla, title = {PCLA: A Framework for Testing Autonomous Agents in the CARLA Simulator}, author = {Tehrani, Masoud Jamshidiyan and Kim, Jinhan and Tonella, Paolo}, + doi = {10.1145/3696630.3728577}, booktitle = {Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering}, pages = {1040--1044}, year = {2025} @@ -131,8 +135,8 @@ @article{tang2023survey @ARTICLE{9793395, author={Chance, Greg and Ghobrial, Abanoub and McAreavey, Kevin and Lemaignan, Séverin and Pipe, Tony and Eder, Kerstin}, - journal={IEEE Transactions on Intelligent Transportation Systems}, - title={On Determinism of Game Engines Used for Simulation-Based Autonomous Vehicle Verification}, + journal={IEEE Transactions on Intelligent Transportation Systems}, + title={On Determinism of Game Engines Used for Simulation-Based Autonomous Vehicle Verification}, year={2022}, volume={23}, number={11}, @@ -142,8 +146,8 @@ @ARTICLE{9793395 @INPROCEEDINGS{9294422, author={Rong, Guodong and Shin, Byung Hyun and Tabatabaee, Hadi and Lu, Qiang and Lemke, Steve and Možeiko, Mārtiņš and Boise, Eric and Uhm, Geehoon and Gerow, Mark and Mehta, Shalin and Agafonov, Eugene and Kim, Tae Hyung and Sterner, Eric and Ushiroda, Keunhae and Reyes, Michael and Zelenkovsky, Dmitry and Kim, Seonman}, - booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)}, - title={LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving}, + booktitle={2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)}, + title={LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving}, year={2020}, volume={}, number={}, diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 62d325f..911d83e 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -14,7 +14,6 @@ import json import math -import time # pylint: disable=import-error from autoware_vehicle_msgs.msg import ControlModeReport @@ -475,6 +474,12 @@ def ego_status(self): def run_step(self, input_data, timestamp): self.timestamp = timestamp + seconds = int(self.timestamp) + nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) + obj_clock = Clock() + obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) + self.clock_publisher.publish(obj_clock) + # publish data of all sensors for key, data in input_data.items(): sensor_type = self.id_to_sensor_type_map[key] @@ -489,16 +494,7 @@ def run_step(self, input_data, timestamp): else: self.ros2_node.get_logger().info("No Publisher for [{key}] Sensor") - time.sleep(0.05) # 50ms delay to ensure published messages are received - - seconds = int(self.timestamp) - nanoseconds = int((self.timestamp - int(self.timestamp)) * 1000000000.0) - obj_clock = Clock() - obj_clock.clock = Time(sec=seconds, nanosec=nanoseconds) - self.clock_publisher.publish(obj_clock) - self.ego_status() - return self.current_control def shutdown(self): diff --git a/srunner/scenariomanager/scenario_manager.py b/srunner/scenariomanager/scenario_manager.py index 0555f89..fa3e040 100644 --- a/srunner/scenariomanager/scenario_manager.py +++ b/srunner/scenariomanager/scenario_manager.py @@ -178,22 +178,18 @@ def _tick_scenario(self, timestamp): if self._debug_mode: print("\n--------- Tick ---------\n") - + if self._agent is not None: + self._agent() # pylint: disable=not-callable _tick_carla_start = time.perf_counter_ns() / 1e6 if self._sync_mode and self._watchdog.get_status(): CarlaDataProvider.get_world().tick() + GameTime.on_carla_tick(timestamp) + CarlaDataProvider.on_carla_tick() MetricsCollector.update_key( "carla_time", (time.perf_counter_ns() / 1e6) - _tick_carla_start ) - - if self._agent is not None: - self._agent() # pylint: disable=not-callable - - # Update game time and actor information - GameTime.on_carla_tick(timestamp) - CarlaDataProvider.on_carla_tick() # Tick scenario _scenario_tick_start = time.perf_counter_ns() / 1e6 From 856d840352f2a88916598febbfcdb7f8ea56588f Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Thu, 25 Dec 2025 20:23:07 +0000 Subject: [PATCH 13/27] removed tick() causing desync --- srunner/objects/ego_vehicle.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/srunner/objects/ego_vehicle.py b/srunner/objects/ego_vehicle.py index 03a4b5b..120efdf 100644 --- a/srunner/objects/ego_vehicle.py +++ b/srunner/objects/ego_vehicle.py @@ -35,8 +35,6 @@ def spawn(self) -> carla.Actor: self.ego_model, self.ego_spawn, self.ego_name ) - CarlaDataProvider.get_world().tick() - if self._actor is None: logger.warning( "Failed to spawn EgoVehicle. This is likely an issue with CARLA." From e9b631f9aec823341d6dfb7fa9dffa5cb3f52f75 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Fri, 26 Dec 2025 09:37:33 +0000 Subject: [PATCH 14/27] moved executor to 2 threads; improves reliability --- cawsr.py | 2 +- srunner/autoagents/autoware_agent.py | 42 ++++++++++++------- .../autoware_carla_interface/carla_ros.py | 2 +- 3 files changed, 29 insertions(+), 17 deletions(-) diff --git a/cawsr.py b/cawsr.py index 0306983..8823d4e 100644 --- a/cawsr.py +++ b/cawsr.py @@ -276,8 +276,8 @@ def _tick_carla(self) -> None: timestamp = snapshot.timestamp if timestamp: CarlaDataProvider.get_world().tick() - CarlaDataProvider.on_carla_tick() GameTime.on_carla_tick(timestamp) + CarlaDataProvider.on_carla_tick() def _load_alg(self) -> type[BasicAlgorithm]: """Load an algorithm instance from mounted docker volume algorithms/ diff --git a/srunner/autoagents/autoware_agent.py b/srunner/autoagents/autoware_agent.py index 5178c1e..8551b54 100644 --- a/srunner/autoagents/autoware_agent.py +++ b/srunner/autoagents/autoware_agent.py @@ -47,26 +47,38 @@ def setup(self, config: EnvironmentConfig) -> None: self.config = config - self._node = rclpy.create_node("cawsr_bridge") - + # initialise autoware state object self.autoware_state = autoware_state.AutowareState("ego_vehicle", None) + self._node = rclpy.create_node("cawsr_bridge") + self._node_state = rclpy.create_node("autoware_state_node") + self.carla_interface = InitializeInterface(self.config, self._node) - self.state_node = state_node.StateNode(self.autoware_state, self._node) + self.state_node = state_node.StateNode(self.autoware_state, self._node_state) self.state_node.reset_autoware(self.config.town, self.config.ego_name) - self.route_node = route_node.RouteNode(self.autoware_state, self._node) - self.autoware_node = autoware_node.AutowareNode(self.autoware_state, self._node) + self.route_node = route_node.RouteNode(self.autoware_state, self._node_state) + self.autoware_node = autoware_node.AutowareNode( + self.autoware_state, self._node_state + ) - self._single_thread_executor = rclpy.executors.SingleThreadedExecutor() + # run state note and cawsr bridge in separate executors + self._executors = [ + rclpy.executors.SingleThreadedExecutor(), + rclpy.executors.SingleThreadedExecutor(), + ] - self._single_thread_executor.add_node(self._node) + self._executors[0].add_node(self._node) + self._executors[1].add_node(self._node_state) - self._executor_thread = threading.Thread( - target=self._single_thread_executor.spin, daemon=True - ) - self._executor_thread.start() + self._executor_threads = [ + threading.Thread(target=self._executors[0].spin, daemon=True), + threading.Thread(target=self._executors[1].spin, daemon=True), + ] + + for thread in self._executor_threads: + thread.start() self.sent_route = False self.initialised = False @@ -110,11 +122,11 @@ def destroy(self) -> None: logger.info("Waiting for shutdown. Starting Node cleanup") time.sleep(1) # sleep for 1 second for sanity try: - self.autoware_node.destroy_node() - self.state_node.destroy_node() - self.route_node.destroy_node() + self._node.destroy_node() + self._node_state.destroy_node() rclpy.shutdown() - self._executor_thread.join() + for thread in self._executor_threads: + thread.join() except RuntimeError: logger.info("Failed to clean up executor thread...") diff --git a/srunner/autoagents/autoware_carla_interface/carla_ros.py b/srunner/autoagents/autoware_carla_interface/carla_ros.py index 911d83e..398f759 100644 --- a/srunner/autoagents/autoware_carla_interface/carla_ros.py +++ b/srunner/autoagents/autoware_carla_interface/carla_ros.py @@ -142,7 +142,7 @@ def __init__(self, node): self.pub_lidar[sensor["id"]] = self.ros2_node.create_publisher( PointCloud2, f"/sensing/lidar/{sensor['id']}/pointcloud_before_sync", - 5, # lower qos depth as using best_reliability + 10, # lower qos depth as using best_reliability ) else: self.ros2_node.get_logger().info( From ea016a5e36268b632133b46edb8864b9f81be972 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Fri, 26 Dec 2025 12:05:18 +0000 Subject: [PATCH 15/27] added new dds configs for local and dist --- Dockerfile | 1 - docker/{cyclonedds.xml => cyclonedds_dist.xml} | 0 docker/cyclonedds_local.xml | 18 ++++++++++++++++++ 3 files changed, 18 insertions(+), 1 deletion(-) rename docker/{cyclonedds.xml => cyclonedds_dist.xml} (100%) create mode 100644 docker/cyclonedds_local.xml diff --git a/Dockerfile b/Dockerfile index b416b3e..9b75a4f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -50,7 +50,6 @@ RUN mkdir /cyclonedds && \ mv /autoware_scenario_runner/docker/cyclonedds.xml /cyclonedds/ && \ rm -rf /autoware_scenario_runner/docker && \ echo "export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp" >> ~/.bashrc && \ - echo "export CYCLONEDDS_URI=file:///cyclonedds/cyclonedds.xml" >> ~/.bashrc && \ echo "alias rossrc='source ${AUTOWARE_MSG_PKG} && source ${ROS_PKG} && echo Sourced'" >> ~/.bashrc && \ source ~/.bashrc diff --git a/docker/cyclonedds.xml b/docker/cyclonedds_dist.xml similarity index 100% rename from docker/cyclonedds.xml rename to docker/cyclonedds_dist.xml diff --git a/docker/cyclonedds_local.xml b/docker/cyclonedds_local.xml new file mode 100644 index 0000000..9319c27 --- /dev/null +++ b/docker/cyclonedds_local.xml @@ -0,0 +1,18 @@ + + + + + + + + default + 65500B + + + + + 500kB + + + + From e18183ebb8d435e197d462763ca6329223c97146 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Fri, 26 Dec 2025 12:12:01 +0000 Subject: [PATCH 16/27] fixed bug on dockerfile --- Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 9b75a4f..7290eb1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -47,7 +47,8 @@ RUN python3 -m pip install -r requirements.txt && \ # update CYCLONE DDS Config for ROS RUN mkdir /cyclonedds && \ - mv /autoware_scenario_runner/docker/cyclonedds.xml /cyclonedds/ && \ + mv /autoware_scenario_runner/docker/cyclonedds_local.xml /cyclonedds/ && \ + mv /autoware_scenario_runner/docker/cyclonedds_dist.xml /cyclonedds/ && \ rm -rf /autoware_scenario_runner/docker && \ echo "export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp" >> ~/.bashrc && \ echo "alias rossrc='source ${AUTOWARE_MSG_PKG} && source ${ROS_PKG} && echo Sourced'" >> ~/.bashrc && \ From c8c1a009be9ff56a65bf8bf0dd489a969c3d90da Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Sun, 28 Dec 2025 14:03:34 +0000 Subject: [PATCH 17/27] updated DDS configs --- Dockerfile | 2 +- ...ds_dist.xml => cyclonedds_distributed.xml} | 22 ++++++++++++------- 2 files changed, 15 insertions(+), 9 deletions(-) rename docker/{cyclonedds_dist.xml => cyclonedds_distributed.xml} (58%) diff --git a/Dockerfile b/Dockerfile index 7290eb1..3c814b0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -48,7 +48,7 @@ RUN python3 -m pip install -r requirements.txt && \ # update CYCLONE DDS Config for ROS RUN mkdir /cyclonedds && \ mv /autoware_scenario_runner/docker/cyclonedds_local.xml /cyclonedds/ && \ - mv /autoware_scenario_runner/docker/cyclonedds_dist.xml /cyclonedds/ && \ + mv /autoware_scenario_runner/docker/cyclonedds_distributed.xml /cyclonedds/ && \ rm -rf /autoware_scenario_runner/docker && \ echo "export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp" >> ~/.bashrc && \ echo "alias rossrc='source ${AUTOWARE_MSG_PKG} && source ${ROS_PKG} && echo Sourced'" >> ~/.bashrc && \ diff --git a/docker/cyclonedds_dist.xml b/docker/cyclonedds_distributed.xml similarity index 58% rename from docker/cyclonedds_dist.xml rename to docker/cyclonedds_distributed.xml index 7584c0f..6540087 100644 --- a/docker/cyclonedds_dist.xml +++ b/docker/cyclonedds_distributed.xml @@ -3,20 +3,26 @@ - + - default + + false + 65500B + + + + + + auto + + - - + - 1MB + 500kB - - warning - From 7600b5f5b0d81cbe6d2b62cf0e3a7dc736e7d36c Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Sun, 28 Dec 2025 18:03:36 +0000 Subject: [PATCH 18/27] changed DDS config to autodetermine network interface --- docker/cyclonedds_distributed.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/cyclonedds_distributed.xml b/docker/cyclonedds_distributed.xml index 6540087..ed688cc 100644 --- a/docker/cyclonedds_distributed.xml +++ b/docker/cyclonedds_distributed.xml @@ -3,7 +3,7 @@ - + false From f149f53ab5f9d9dae5e29c522574c3c7bbb082ec Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Sun, 28 Dec 2025 20:34:52 +0000 Subject: [PATCH 19/27] updated DDS config --- docker/cyclonedds_distributed.xml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docker/cyclonedds_distributed.xml b/docker/cyclonedds_distributed.xml index ed688cc..45fe1d7 100644 --- a/docker/cyclonedds_distributed.xml +++ b/docker/cyclonedds_distributed.xml @@ -3,21 +3,20 @@ - + - false - 65500B + auto + 10 - From bcee0227c61507eca0874e79e53dca8a6b78be15 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 29 Dec 2025 22:08:37 +0000 Subject: [PATCH 20/27] updated LICENSE, README --- CONTRIBUTING.md | 17 ++++ LICENSE Autoware | 201 ++++++++++++++++++++++++++++++++++++++ NOTICE | 25 +++++ README.md | 135 +++++++++++++++++++++----- cawsr.py | 6 +- example_scenario.json | 219 ------------------------------------------ 6 files changed, 357 insertions(+), 246 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 LICENSE Autoware create mode 100644 NOTICE delete mode 100644 example_scenario.json diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..0f7da7b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,17 @@ +# Contributing to CAWSR + +Thank you for your interest in this project! We welcome contributions from the community. + +## How to Contribute +1. **Fork the repository** on GitHub. +2. **Create a new branch** for your feature or bug fix. +3. **Submit a Pull Request (PR)** with a clear description of your changes. + +## Reporting Issues +If you encounter any bugs or unexpected behaviour, please [open an issue](https://github.com/Intelligent-Testing-Lab/cawsr) on GitHub. Please include: +* Steps to reproduce the error. +* Your environment details (OS, CARLA version, Autoware version). +* Any error logs or screenshots. + +## Seeking Support +If you have questions about how to use the software, please open an issue with the "question" label or contact the authors directly. diff --git a/LICENSE Autoware b/LICENSE Autoware new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE Autoware @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..a5ee11a --- /dev/null +++ b/NOTICE @@ -0,0 +1,25 @@ +Scenario Runner for Autoware +Copyright 2025 University of Sheffield + +This product includes software based on Scenario Runner by CARLA. +Copyright (c) Intel Corporation / CARLA Team. + +------------------------------------------------------------------------- + +This product includes software developed by The Autoware Foundation +(https://www.autoware.org/) and its contributors. + +FROM AUTOWARE UNIVERSE (autoware_carla_interface): +Copyright 2021 The Autoware Foundation + +This product includes software developed at +The Autoware Foundation (https://www.autoware.org/). + +This product includes code developed by TIER IV. +Copyright 2017 TIER IV, Inc. + +This product includes code developed by AutoCore. +Copyright 2022 AutoCore Technology (Nanjing) Co., Ltd. + +This product includes code developed by Leo Drive. +Copyright 2022 Leo Drive Teknoloji A.Ş. diff --git a/README.md b/README.md index f1a520c..85fdaeb 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,13 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -![GitHub tag (latest SemVer)](https://img.shields.io/github/tag/carla-simulator/scenario_runner.svg) CAWSR: ScenarioRunner for CARLA with support for Autoware ======================== -This repository contains scenario definition and an execution engine -for CARLA. Support has been added to run route-based scenarios with the ego being controlled by [Autoware](https://autoware.org/autoware-overview/) + +CAWSR (Carla Autoware Scenario Runner) is a scenario execution engine built for the testing of [Autoware](https://autoware.org/autoware-overview/) in route-based scenarios. Prerequisites --------------------------- -Both CARLA and Autoware require a high-spec computer with a high-end Nvidia GPU. It is also possible to run a [**distributed**]() setup with multiple machines to help ease the workload. Currently, only Linux is supported (guide was written on Ubuntu 24.04). +Both CARLA and Autoware require a high-spec computer with a high-end Nvidia GPU. It is also possible to run a [**distributed**]() setup with multiple machines to help ease the workload, or run the entire stack locally. Currently, only Linux is supported (guide was written on Ubuntu 24.04). Ensure the target machine(s) have the [Docker Engine]() and [Nvidia Container toolkit]() installed to enable gpu accelerated workflows in Docker. @@ -23,7 +22,7 @@ docker pull ghcr.io/intelligent-testing-lab/autoware-scenario-runner:latest docker pull ghcr.io/intelligent-testing-lab/autoware:latest ``` -Autoware and ROS use a custom messaging interface for communication, known as DDS. They support various implementations, but they all rely on specific network settings to enable maximum data transfer. Save the following command in `setup.sh`, allow it to be executable `chmod +x setup.sh` and run. +Autoware and ROS use a custom messaging interface for communication, known as DDS. For maximum performance, configure your network settings as follows. If not configured, you will see [heavy performance issues](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning) as the default ubuntu buffer sizes fill up fast, especially when running over lossy networks such as WiFi. ```bash # Increase the maximum receive and send buffer size for network packets, allowing our containers to communicate sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB @@ -33,6 +32,7 @@ sudo sysctl -w net.core.wmem_max=2147483647 sudo sysctl -w net.ipv4.ipfrag_time=3 # in seconds, default is 30 s sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # 128 MiB, default is 256 KiB ``` +Save the following commands in `setup.sh`, allow it to be executable `chmod +x setup.sh`. These settings are **temporary** and will revert on restart. To allow GUI applications (like Autoware and CARLA) to run through Docker, you must allow xhost connections from the `docker` group. @@ -40,24 +40,95 @@ To allow GUI applications (like Autoware and CARLA) to run through Docker, you m xhost +local:docker ``` -Using CAWSR +Running CAWSR ------------------------ -After completiting the prerequisite steps, clone the [CAWSR workspace](https://github.com/Intelligent-Testing-Lab/cawsr_workspace) repository. To launch CAWSR, navigate to the CAWSR workspace and run `docker compose up`. +CAWSR can both be ran `locally` or `distributed`. Due to the high-spec requirements, it is recommended to run distributed if you do not meet the following minimum specs: +- At least **10GB** VRAM and a modern GPU (2080 ti or newer) +- At least **32GB** RAM +- A modern Intel or AMD CPU with at least 8 cores. + +### Locally + +To run locally, set *MODE* in `.env` +```env +[Network] +MODE=local +``` + +Ensure multicast is enabled for the *localhost* interface: +```bash +sudo ip link show lo +1: lo: +``` +If **MUTLICAST** is not present, you can enable it with `sudo ip link set lo multicast on`. + +Run the entire stack +```bash +docker compose up +``` + +### Distributed + +When running distributed, we use *unicast* to enable compatibility with all networks. This requires some extra configuration. + +Running CAWSR distributed using the following setup: +- **Machine A**: Carla and CAWSR +- **Machine B**: Autoware + +Configure the `.env` and ensure is it the same across both machines +``` +[Network] +MODE=local # or distributed (caps sensitive) +ROS_DOMAIN_ID=0 + +# For distributed mode +HOST_IP=127.0.0.1 # CAWSR and CARLA +AUTOWARE_IP=127.0.0.1 # Autoware +``` +The `ROS_DOMAIN_ID` *must* match, otherwise the ROS2 nodes will not be able to find each other. Once configured, start CAWSR and Carla on Machine A +``` +docker compose up carla cawsr +``` +and Autoware on Machine B +``` +docker compose up autoware-latest +``` + +If you are experiencing problems, you can trouble shoot with the following command, replacing `eth0` and `192.168.1.20` with your network interface and destination IP: +```bash +ip addr show # find the network interface used +sudo tcpdump -i eth0 udp and src 192.168.1.20 +``` +This will tell you if packets are flowing between Machine A and Machine B. If no packets are flowing, it is likely an issue with your network configuration. +Using CAWSR +------------------------ +After completing the prerequisite steps, clone the [CAWSR workspace](https://github.com/Intelligent-Testing-Lab/cawsr_workspace) repository. The structure of the workspace is as follows. ``` scenarios/ -> this folder holds all the scenario configurations configs/ -> this folder holds all user config files results/ -> results from runs are stored here algorithms/ -> holds all custom algorithm scripts +docker_compose.yml +.env ``` All folders are mounted as Docker volumes into the CAWSR container, so any changes persist between host and container. -In CAWSR, there are two modes you can configure `algorithm` or `benchmark`. To set the mode, modify `mode: 'benchmark' # benchmark or algorithm` in a `config.yaml` file. You can create multiple configuration files in `configs/`. To use a specific config, modify the **CAWSR_CONFIG** ENV variable in the `docker-compose.yaml`, pointing it to the path of your config file. **All files use relative paths from the CAWSR root directory**. +## Configuring CAWSR + +CAWSR is designed to be highly configurable and supports easy swapping of config files. +1. Create a config file in `configs/` based on one of the examples. +2. Modify the `CAWSR_CONFIG` environmental variable in `.env` to point towards the selected file. **All files use relative paths from the CAWSR root directory**. + +## Execution Mode -**Algorithm** -Algorithm config: +In CAWSR, there are two modes you can configure `algorithm` or `benchmark`. To set the mode, modify the **mode** variable in `config.yaml`. + +### Algorithm + +This mode enables the use of a custom algorithm to modify / optimize the scenario definition after execution. ```yaml algorithm: initial_definition: scenarios/examples/example_scenario.json # can be null @@ -69,41 +140,57 @@ algorithm: ``` Included in `algorithms/basic_algorithm.py` is the BasicAlgorithm class, from which all algorithms inherit. The algorithm is ran on every -iteration of the scenario, modifying the defintion based on the result of the previous scenario. At beginning of every iteration, the method +iteration of the scenario, modifying the definition based on the result of the previous scenario. At beginning of every iteration, the method ```python def _scenario_callback( self, scenario_definition: dict, driving_score: float ) -> dict: ``` -is called. To implement a custom algorithm, create a class than inherits from `BasicAlgorithm` and implements the function `scenario_callback`. The function must follow the signature above, returning a new scenario definition. To use outside resources, such as loading a lanelet file (see example config), pass them in via the args config variable. This gets converted into a python dictionary and passed to the algorithm class when initialised. Algorithms are run sync, so CAWSR will wait for completion. +is called. + +#### Implementing a custom algorithm +Create a class than inherits `BasicAlgorithm` and implements the function `scenario_callback`. The function must accept the current scenario_definition and the driving score, returning a new scenario definition. To use outside resources, such as loading a lanelet file (see example config), pass them in via the args config variable. This gets converted into a python dictionary and passed to the algorithm class when initialised. Algorithms are run synchronously, so CAWSR will wait for completion. The algorithm will execute **runs** times. +### Benchmark + +Benchmark simply executes all scenario definitions in a given directory. Set `scenarios` to a path containing `.json` scenario definitions, and enable / disable random sampling. If enabled, CAWSR will executed each scenario once in a random order. + + Scenario Definition ------------------- We use a custom implementation of a scenario definition in JSON. We have included a scenario domain model, as well as plenty of examples in the CAWSR Workspace repository `scenarios/examples/`. Domain Model: -![Domain Model](./docs/resources/scenario_domain.pdf) +![Domain Model](./docs/resources/scenario_domain.png) Notes ------------ -Currently, traffic light recognition is disabled due to an issue with the [CARLA map format](https://github.com/autowarefoundation/autoware_universe/tree/main/simulator/autoware_carla_interface#traffic-light-recognition). This is a time consuming process, as each new traffic light requires the creation of a new objects within the CARLA Lanelet2 file that match the position of the PCD exactly. Once finished, we'll publish an updated Autoware image accordingly. +Currently, traffic light recognition is disabled due to an issue with the [CARLA map format](https://github.com/autowarefoundation/autoware_universe/tree/main/simulator/autoware_carla_interface#traffic-light-recognition). The updated LaneLet files (as well as the Autoware images) will be published accordingly once development has finished. Contributing ------------ -Please take a look at our [Contribution guidelines](https://carla.readthedocs.io/en/latest/#contributing). - -FAQ ------- - -If you run into problems, check our -[FAQ](http://carla.readthedocs.io/en/latest/faq/). +Please take a look at our [Contribution guidelines](). License ------- - -ScenarioRunner specific code is distributed under MIT License. -CAWSR specific code is distributed under MIT License. +### 1. CAWSR +Core CAWSR logic and Autoware integration. +* **Copyright:** © 2025 University of Sheffield +* **License File:** [`LICENSE Sheffield`](./LICENSE%20Sheffield) + +### 2. Scenario Runner (CARLA) (MIT) +Scenario execution engine for CARLA. +* **Copyright:** © Intel Corporation / CARLA Team +* **License File:** [`LICENSE Carla`](./LICENSE%20Carla) + +### 3. Autoware Carla Interface (Apache 2.0) +Autoware communication bridge. +**Apache License 2.0**. +* **Copyright:** © The Autoware Foundation / Tier IV, Inc. / AutoCore / Leo Drive +* **License File:** [`LICENSE-APACHE`](./LICENSE-APACHE) + +**Notices:** See [`NOTICE`](./NOTICE) for the full list of required attributions. diff --git a/cawsr.py b/cawsr.py index 8823d4e..10e621a 100644 --- a/cawsr.py +++ b/cawsr.py @@ -232,7 +232,6 @@ def run_scenario( # logger.info("Successfully initialised agent; route set.") self.scenario_manager.run_scenario() - self.carla_client.stop_recorder() result = True except Exception: traceback.print_exc() @@ -241,6 +240,7 @@ def run_scenario( ) result = False + self.carla_client.stop_recorder() # stop the MetricsCollector thread MetricsCollector.reset() @@ -324,7 +324,7 @@ def run_algorithm(self) -> None: logger.info("Starting CARLA container....") CARLAManager.restart_carla() - time.sleep(5) # allow CARLA to load + time.sleep(10) # allow CARLA to load env_config = EnvironmentParser.parse_scenario_env( self.results_manager.fetch_scenario_xml() @@ -363,7 +363,7 @@ def run_benchmark(self) -> None: logger.info("Starting CARLA container....") CARLAManager.restart_carla() - time.sleep(5) # allow CARLA to load + time.sleep(10) # allow CARLA to load if self._conf["benchmark"]["random_sampling"]: scenario = random.choice(scenarios) diff --git a/example_scenario.json b/example_scenario.json deleted file mode 100644 index f8d15fd..0000000 --- a/example_scenario.json +++ /dev/null @@ -1,219 +0,0 @@ -{ - "routes": [ - { - "route": { - "id": 0, - "weathers": [ - { - "weather": { - "route_percentage": 0.0, - "precipitation": 100.0, - "cloudiness": 0.0, - "precipitation_deposits": 100.0, - "wetness": 100.0, - "wind_intensity": 100.0, - "sun_azimuth_angle": -1.0, - "sun_altitude_angle": 90.0, - "fog_density": 2.0 - } - }, - { - "weather": { - "route_percentage": 100.0, - "precipitation": 0.0, - "cloudiness": 0.0, - "precipitation_deposits": 0.0, - "wetness": 0.0, - "wind_intensity": 0.0, - "sun_azimuth_angle": -1.0, - "sun_altitude_angle": 90.0, - "fog_density": 2.0 - } - } - ], - "waypoints": [ - { - "position": { - "x": 88.4, - "y": 82.2, - "z": 0.0 - } - }, - { - "position": { - "x": 230.0, - "y": 133.5, - "z": 0.0 - } - } - ], - "scenarios": [ - { - "scenario": { - "name": "PedestrianCrossing_1", - "type": "PedestrianCrossing", - "trigger_point": { - "x": 160.0, - "y": 133.5, - "z": 0.0, - "yaw": 0.0 - } - } - } - ] - } - }, - { - "route": { - "id": 1, - "weathers": [ - { - "weather": { - "route_percentage": 0.0, - "cloudiness": 0.0, - "precipitation": 100.0, - "precipitation_deposits": 100.0, - "wetness": 100.0, - "wind_intensity": 100.0, - "sun_azimuth_angle": -1.0, - "sun_altitude_angle": 90.0, - "fog_density": 2.0 - } - }, - { - "weather": { - "route_percentage": 50.0, - "cloudiness": 50.0, - "precipitation": 0.0, - "precipitation_deposits": 0.0, - "wetness": 0.0, - "wind_intensity": 0.0, - "sun_azimuth_angle": -1.0, - "sun_altitude_angle": 90.0, - "fog_density": 2.0 - } - }, - { - "weather": { - "route_percentage": 100.0, - "cloudiness": 100.0, - "precipitation": 0.0, - "precipitation_deposits": 0.0, - "wetness": 0.0, - "wind_intensity": 0.0, - "sun_azimuth_angle": -1.0, - "sun_altitude_angle": 90.0, - "fog_density": 3.0 - } - } - ], - "waypoints": [ - { - "position": { - "x": 983.5, - "y": 5382.2, - "z": 371 - } - }, - { - "position": { - "x": 1234, - "y": 5.2, - "z": 3123471 - } - }, - { - "position": { - "x": 123412343.5, - "y": 512341234.2, - "z": 3712341234 - } - } - ], - "scenarios": {} - } - } - ], - "scenarios": [ - { - "scenario": { - "town": "Town01", - "ego_vehicle": { - "x": 312, - "y": 129, - "z": 0, - "yaw": 180, - "model": "vehicle.toyota.prius", - "name": "ego_vehicle", - "sensor_configuration": [ - { - "sensor": { - "type": "sensor.camera.rgb", - "id": "rgb_front", - "spawn_point": { - "x": 0.7, - "y": 0.0, - "z": 1.6, - "roll": 0.0, - "pitch": 0.0, - "yaw": 0.0 - }, - "image_size_x": 1920, - "image_size_y": 1080, - "fov": 90.0 - } - }, - { - "sensor": { - "type": "sensor.lidar.ray_cast", - "id": "top", - "spawn_point": { - "x": 0.0, - "y": 0.0, - "z": 3.1, - "roll": 0.0, - "pitch": 0.0, - "yaw": 0.0 - }, - "range": 100, - "channels": 64, - "points_per_second": 300000, - "upper_fov": 10.0, - "lower_fov": -30.0, - "rotation_frequency": 20 - } - }, - { - "sensor": { - "type": "sensor.other.gnss", - "id": "gnss", - "spawn_point": { - "x": 0.0, - "y": 0.0, - "z": 1.6, - "roll": 0.0, - "pitch": 0.0, - "yaw": 0.0 - } - } - }, - { - "sensor": { - "type": "sensor.other.imu", - "id": "imu", - "spawn_point": { - "x": 0.0, - "y": 0.0, - "z": 1.6, - "roll": 0.0, - "pitch": 0.0, - "yaw": 0.0 - } - } - } - ] - } - } - } - ] -} From 018edeed1523b3891428e47498c63105a6ac4ee2 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 29 Dec 2025 22:32:13 +0000 Subject: [PATCH 21/27] added log statements; sometimes recording does not copy properly --- srunner/tools/CARLA_manager.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/srunner/tools/CARLA_manager.py b/srunner/tools/CARLA_manager.py index e1fddeb..50e788f 100644 --- a/srunner/tools/CARLA_manager.py +++ b/srunner/tools/CARLA_manager.py @@ -28,7 +28,7 @@ class CARLAManager(object): def _load_config(config: CARLA) -> None: CARLAManager.port = config.PORT CARLAManager.fidelity = config.FIDELITY - CARLAManager.FIXED_DELTA_SECONDS = config.FIXED_DELTA_SECONDS + CARLAManager.FIXED_DELTA_SECONDS = config.FIXED_DELTA_SECONDS CARLAManager.run_command = [ f'docker run -dt --gpus all --net=host -v /tmp/.X11-unix:/tmp/.X11-unix:rw -e DISPLAY=$DISPLAY -e NVIDIA_DRIVER_CAPABILITIES=all -e XDG_RUNTIME_DIR=/tmp carlasim/carla:0.9.15 /bin/bash -c "./CarlaUE4.sh -carla-rpc-port={CARLAManager.port} -quality-level={CARLAManager.fidelity}"' @@ -108,6 +108,8 @@ def fetch_file(path: str, dest: str): if not result.returncode == 0: logger.info(f"Failed to copy {path} to {dest}") + logger.info(f"stdout: {result.stdout.strip()}") + logger.info(f"stderr: {result.stderr.strip()}") @staticmethod def restart_carla(): From 42ab5da4713924bf2cca6f416384003fbef0ce5f Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 29 Dec 2025 22:47:08 +0000 Subject: [PATCH 22/27] added sleep to ensure file is written --- cawsr.py | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/cawsr.py b/cawsr.py index 10e621a..903acee 100644 --- a/cawsr.py +++ b/cawsr.py @@ -183,6 +183,9 @@ def run_scenario( logger.info("Loading route...") + # TO DO + # interpolate route at a larger distance (i.e 5m) to reduce waypoints + # remove segment sampling from awagent.set_route gps_route, route = route_manipulation.interpolate_trajectory( route_config.keypoints ) @@ -190,6 +193,18 @@ def run_scenario( ego.prepare_ego(route[0][0]) # set location to first waypoint + # allow the agent X ticks to initialize sensors and set the route + logger.info("Initialising agent route...") + budget = self._conf["initialisation_budget"] + status = False + for tick in range(1, budget + 1): + status = self.aw_agent.run_step_init() # type: ignore + self._tick_carla() + if not status: + logger.info("Agent failed to initialise route") + else: + logger.info("Successfully initialised agent; route set.") + self._tick_carla() logger.info("Loading Traffic Manager...") @@ -218,19 +233,6 @@ def run_scenario( self.scenario_manager.load_scenario( scenario, self.aw_agent, follow_ego=True ) - - # logger.info("Initialising agent route...") - # allow the agent to localise and set the route - # budget = self._conf["initialisation_budget"] - # status = False - # for tick in range(1, budget + 1): - # status = self.aw_agent.run_step_init() # type: ignore - # CarlaDataProvider.get_world().tick() - # if not status: - # logger.info("Agent failed to initialise route") - # else: - # logger.info("Successfully initialised agent; route set.") - self.scenario_manager.run_scenario() result = True except Exception: @@ -441,6 +443,7 @@ def _cawsr_process( self.results_manager.cleanup_xml() # copy over the recording from CARLA container + time.sleep(1) # ensure file is written CARLAManager.fetch_file( "/home/carla/recording.log", self.results_manager.last_scenario, From 5a726834474e38a61c11ad1f89c04cf628e7cd60 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Mon, 29 Dec 2025 23:10:31 +0000 Subject: [PATCH 23/27] cleaned up initialisation code --- cawsr.py | 2 -- srunner/autoagents/autoware_agent.py | 13 +++++++------ 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/cawsr.py b/cawsr.py index 903acee..80955f6 100644 --- a/cawsr.py +++ b/cawsr.py @@ -205,8 +205,6 @@ def run_scenario( else: logger.info("Successfully initialised agent; route set.") - self._tick_carla() - logger.info("Loading Traffic Manager...") tm_port = int(self._carla.TRAFFIC_MANAGER.PORT) # type: ignore CarlaDataProvider.set_traffic_manager_port(tm_port) diff --git a/srunner/autoagents/autoware_agent.py b/srunner/autoagents/autoware_agent.py index 8551b54..3b346ee 100644 --- a/srunner/autoagents/autoware_agent.py +++ b/srunner/autoagents/autoware_agent.py @@ -138,7 +138,6 @@ def run_step_init(self) -> bool: Ticks CARLA and Autoware, allowing the agent to localise and plan the route. Operates on a fixed tick budget to ensure determinism. If the agent goes over the budget, it is treated as a failure. - """ if not self.agent_set_route: @@ -166,6 +165,8 @@ def run_step_init(self) -> bool: if self.autoware_state.route_set() and not self.autoware_state.sent_engage: return True + self.carla_interface.tick_bridge() + return False def run_step(self) -> None: @@ -177,11 +178,11 @@ def run_step(self) -> None: ) self.last_tick = time.perf_counter_ns() - if not self.initialised: - self.initialised = self.run_step_init() - - if self.initialised: - logger.info("Set agent route!") + # if not self.initialised: + # self.initialised = self.run_step_init() + # + # if self.initialised: + # logger.info("Set agent route!") # check if the current route is set and we can publish engage if self.autoware_state.route_set() and not self.autoware_state.sent_engage: From 3a611b6ca2d4a88aa98bd44e7c00c3d3bb89345c Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Tue, 30 Dec 2025 09:42:01 +0000 Subject: [PATCH 24/27] moved agent initialisation --- cawsr.py | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/cawsr.py b/cawsr.py index 80955f6..35c394d 100644 --- a/cawsr.py +++ b/cawsr.py @@ -193,18 +193,6 @@ def run_scenario( ego.prepare_ego(route[0][0]) # set location to first waypoint - # allow the agent X ticks to initialize sensors and set the route - logger.info("Initialising agent route...") - budget = self._conf["initialisation_budget"] - status = False - for tick in range(1, budget + 1): - status = self.aw_agent.run_step_init() # type: ignore - self._tick_carla() - if not status: - logger.info("Agent failed to initialise route") - else: - logger.info("Successfully initialised agent; route set.") - logger.info("Loading Traffic Manager...") tm_port = int(self._carla.TRAFFIC_MANAGER.PORT) # type: ignore CarlaDataProvider.set_traffic_manager_port(tm_port) @@ -225,12 +213,26 @@ def run_scenario( logger.info("Could not load Route Scenario") traceback.print_exc() + # allow the agent X ticks to initialize sensors and set the route + logger.info("Initialising agent route...") + budget = self._conf["initialisation_budget"] + status = False + for tick in range(1, budget + 1): + status = self.aw_agent.run_step_init() # type: ignore + self._tick_carla() + if not status: + logger.info("Agent failed to initialise route") + else: + logger.info("Successfully initialised agent; route set.") + logger.info("Starting scenario...") + try: self.carla_client.start_recorder("/home/carla/recording.log", True) self.scenario_manager.load_scenario( scenario, self.aw_agent, follow_ego=True ) + self.scenario_manager.run_scenario() result = True except Exception: From dc853c93eddcaee0cae034ebf52a1ddfbb05a9a8 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Tue, 30 Dec 2025 10:02:28 +0000 Subject: [PATCH 25/27] removed GameTime Restart from load_scenario --- README.md | 8 +++--- cawsr.py | 27 ++++++++++++--------- srunner/scenariomanager/scenario_manager.py | 1 - 3 files changed, 19 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 85fdaeb..8149fc9 100644 --- a/README.md +++ b/README.md @@ -73,22 +73,22 @@ docker compose up When running distributed, we use *unicast* to enable compatibility with all networks. This requires some extra configuration. Running CAWSR distributed using the following setup: -- **Machine A**: Carla and CAWSR +- **Machine A**: CAWSR and CARLA - **Machine B**: Autoware Configure the `.env` and ensure is it the same across both machines ``` [Network] -MODE=local # or distributed (caps sensitive) +MODE=distributed ROS_DOMAIN_ID=0 # For distributed mode HOST_IP=127.0.0.1 # CAWSR and CARLA AUTOWARE_IP=127.0.0.1 # Autoware ``` -The `ROS_DOMAIN_ID` *must* match, otherwise the ROS2 nodes will not be able to find each other. Once configured, start CAWSR and Carla on Machine A +The `ROS_DOMAIN_ID` *must* match, otherwise the ROS2 nodes will not be able to find each other. Once configured, start CAWSR on Machine A ``` -docker compose up carla cawsr +docker compose up cawsr ``` and Autoware on Machine B ``` diff --git a/cawsr.py b/cawsr.py index 35c394d..eb952c1 100644 --- a/cawsr.py +++ b/cawsr.py @@ -155,6 +155,9 @@ def run_scenario( logger.info(f"{settings.__str__()}") + logger.info("Restarting GameTime...") + GameTime.restart() + # update the world CarlaDataProvider.set_world(self.carla_world) @@ -193,6 +196,18 @@ def run_scenario( ego.prepare_ego(route[0][0]) # set location to first waypoint + # allow the agent X ticks to initialize sensors and set the route + logger.info("Initialising agent route...") + budget = self._conf["initialisation_budget"] + status = False + for tick in range(1, budget + 1): + status = self.aw_agent.run_step_init() # type: ignore + self._tick_carla() + if not status: + logger.info("Agent failed to initialise route") + else: + logger.info("Successfully initialised agent; route set.") + logger.info("Loading Traffic Manager...") tm_port = int(self._carla.TRAFFIC_MANAGER.PORT) # type: ignore CarlaDataProvider.set_traffic_manager_port(tm_port) @@ -213,18 +228,6 @@ def run_scenario( logger.info("Could not load Route Scenario") traceback.print_exc() - # allow the agent X ticks to initialize sensors and set the route - logger.info("Initialising agent route...") - budget = self._conf["initialisation_budget"] - status = False - for tick in range(1, budget + 1): - status = self.aw_agent.run_step_init() # type: ignore - self._tick_carla() - if not status: - logger.info("Agent failed to initialise route") - else: - logger.info("Successfully initialised agent; route set.") - logger.info("Starting scenario...") try: diff --git a/srunner/scenariomanager/scenario_manager.py b/srunner/scenariomanager/scenario_manager.py index fa3e040..bb1c7dc 100644 --- a/srunner/scenariomanager/scenario_manager.py +++ b/srunner/scenariomanager/scenario_manager.py @@ -77,7 +77,6 @@ def _reset(self): self.scenario_duration_game = 0.0 self.start_system_time = None self.end_system_time = None - GameTime.restart() def cleanup(self): """ From 3324679dc936651e680fdb966815a6c730ef111b Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Tue, 30 Dec 2025 10:55:20 +0000 Subject: [PATCH 26/27] fixed bug where bridge wouldn't tick after route set --- srunner/autoagents/autoware_agent.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/srunner/autoagents/autoware_agent.py b/srunner/autoagents/autoware_agent.py index 3b346ee..5692b91 100644 --- a/srunner/autoagents/autoware_agent.py +++ b/srunner/autoagents/autoware_agent.py @@ -140,6 +140,8 @@ def run_step_init(self) -> bool: Operates on a fixed tick budget to ensure determinism. If the agent goes over the budget, it is treated as a failure. """ + self.carla_interface.tick_bridge() + if not self.agent_set_route: self.set_route() @@ -165,8 +167,6 @@ def run_step_init(self) -> bool: if self.autoware_state.route_set() and not self.autoware_state.sent_engage: return True - self.carla_interface.tick_bridge() - return False def run_step(self) -> None: From 7169a571e911929ab6c441809d385acadd605263 Mon Sep 17 00:00:00 2001 From: David Gasinski Date: Tue, 6 Jan 2026 11:58:01 +0000 Subject: [PATCH 27/27] added new requirement --- requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements.txt b/requirements.txt index f982bbd..f1e531e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,3 +14,4 @@ simple-watchdog-timer antlr4-python3-runtime==4.10 graphviz lanelet2 +allpairspy