Perception

The perception system operates as a multithreaded pipeline, with each thread responsible for a specific stage in processing camera data for autonomous driving. Thread communication and data exchange are managed by a synchronization class using mutexes and condition variables.

Threads Overview

1. Camera Thread

   • Captures frames from the front-facing camera
   • Passes frames to the synchronization class for downstream access

2. Lane Detection Thread

   • Retrieves frames from the synchronization class
   • Runs deep learning model inference to produce a binary mask (lane vs non-lane)
   • Stores lane mask in the synchronization class for use by other threads

3. Object Detection Thread

   • Retrieves frames from the synchronization class
   • Runs deep learning model inference to produce a multi-class mask (vehicles, pedestrians, traffic signs, traffic lights, drivable area)
   • Stores object mask in the synchronization class for use by other threads

4. Traffic Sign/Light Classification Thread

   • Retrieves object mask from the synchronization class
   • Crops regions where traffic signs or lights are detected
   • Runs secondary model inference to classify detected signs/lights
   • Publishes classification results for use by control modules

5. TrajectoryDefinition Thread

   • Retrieves lane mask and object mask from the synchronization class
   • Performs:
     • Inverse Perspective Mapping (IPM) for top-down view
     • Connected Components analysis to identify lane clusters
     • Lane validation and matching:
       • Validates clusters against history and distance constraints
       • If two valid lanes detected: maintains both, updates lane width history
       • If only one lane detected: creates synthetic second lane using width history
       • If no valid matches: defaults to clusters closest to frame center
     • Midcurve generation to create trajectory line
   • Publishes:
     • Midpoint error for PID steering control
     • Trajectory curve for MPC and control modules
   • Analyzes trajectory for obstacles and non-road regions:
     • Triggers emergency braking if needed
     • In SAE_4 mode, adjusts trajectory to avoid obstacles if possible
   • Computes ACC inputs (distance and velocity of moving obstacles in trajectory)
   • Performs LKAS analysis (difference from lanes to mid for lateral assist)

Integration

• All threads operate in real-time and synchronize data via the synchronization class for consistent decision-making.
• Outputs are published to the control system and other