openscenario-osint-generator

Research artefact for our 2024 bachelor thesis at the University of Gothenburg and Chalmers University of Technology. The prototype generates ASAM OpenSCENARIO 1.2.0 XML test scenarios for Autonomous Driving and ADAS from open web sources and Large Language Models.

Paper: [A New Approach to AD/ADAS Test Scenario Generation Using Open-Source Intelligence and Large Language Models]

• Authors: Aleksey Zorin, Louis Mercier
• Supervisor: Muhammed Çağrı Kaya
• Examiner: Christian Berger

What it does

The system takes a plain-text description of a traffic incident and produces an OpenSCENARIO 1.2.0 XML file (.xosc) that a simulator can load. Every generated file is validated against the official ASAM XSD before it is written to disk. Invalid scenarios are flagged, so the output is guaranteed to conform the schema.

The artefact consists of three components:

1. Scraper (scraper/). BeautifulSoup and Requests for HTML, feedparser for RSS. Enforces robots.txt via urllib.robotparser on every URL. Persists raw text to MongoDB with source URL and timestamp.
2. Generator (generator/). A regex parser extracts entities, weather, road type, and event sequences from a structured plain-text description. An OpenSCENARIOGenerator class builds the XML tree with lxml and validates against schemas/OpenSCENARIO_1_2.xsd.
3. LLM completion (ollama_pipeline/). Two custom Ollama modelfiles built on Llama 3 8B, run in sequence: scenario-structure refactors natural text into the OpenSCENARIO structure, scenario-completion fills gaps with plausible values. GPT-4 was used in the thesis for comparison.

Screenshots

The Scenario Generator, where a plain-text description is submitted and the generator produces a validated .xosc file.

The Scenario Maker, with the URL scraping at the top, including the prompt pipeline and scenario templates. The chosen input generates data from scraped text, and formulates two prompts for the LLM.

Tech stack

• Python 3.11, Flask
• BeautifulSoup, Requests, feedparser (scraping and parsing)
• MongoDB via PyMongo (data storage)
• lxml with the ASAM OpenSCENARIO 1.2.0 XSD (XML generation and validation)
• OpenAI GPT-4 and Meta Llama 3 8B via Ollama (scenario completion)

Input format

The generator expects a plain-text description in the following structure. This is a known limitation. The regex is strict and does not tolerate reordering or missing fields.

• Scenario: <free text describing the incident>.
• Weather: <rainy|sunny|snowy|foggy|clear>,
• Precipitation: <dry|rain|snow>,
• Wind: <int> m/s,
• TimeOfDay: <morning|afternoon|evening|night|unspecified>,
• Traffic: <light|moderate|heavy>,
• RoadType: <motorway|urban|rural|...>,
• Vehicles: <Name>, <type>, <category>; <Name>, <type>, <category>,
• Pedestrians: <none|low|moderate|high>,
• Events: <event 1>; <event 2>,

Getting started

Installation

To install the project, you can follow these steps:

1. Clone the repository:

   git clone https://github.com/louismercier/12-web-scraping-prototype.git

2. Start a new virtual environment

   python3 -m venv env

   And active the virtual environment:

   source env/bin/activate

   Or use the following command for Windows

   env\Scripts\activate

3. Navigate to the project directory:

   cd 12-web-scraping-prototype

4. Install the required dependencies:

   pip install -r requirements.txt

Run the project

To run the project, you can follow these steps:

1. Make sure you are in the project directory:

   cd 12-web-scraping-prototype

2. Execute the main script:

   python app.py

   NOTE: If you have multiple versions of Python installed on your system, make sure to run the command with the appropriate Python version. For example, if your project requires Python 3.11, you should run python3.11 app.py.

3. The project will run on http://127.0.0.1:5000

4. A simple UI provides the section where the link for the target website should be inserted

Setting up the project on Windows

1. Clone the project.

2. Navigate to the project directory.

3. Before creating a virtual environment, you might need to adjust your PowerShell execution policy. Open a PowerShell as Administrator and run the following command:

   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

   This command allows scripts signed by a trusted publisher to run on your system.

4. Create a virtual environment:

   python -m venv env

5. Activate the virtual environment:

   env\Scripts\activate

6. Install the required packages:

   pip install -r requirements.txt

Remember to set your execution policy back to its original setting when you're done. You can do this with the following command:

Set-ExecutionPolicy -ExecutionPolicy Restricted -Scope CurrentUser

Handle database

After the first POST request to the Flask app that triggers save_to_database, you can check if the data is inserted correctly by connecting to the MongoDB shell (mongo) and using the following commands:

db.scraped_data.find()

This will display the documents inserted into the scraped_data collection.

Design Goals

Original functional requirements from the artefact planning phase, annotated with what was actually implemented:

Implemented

• Link Input Handling: The prototype must accept user input in the form of URLs, which will be the target for data scraping.
• HTTP Request Handling: Implement functionality to perform HTTP requests to the provided URLs using the Requests library (or Scrapy if we need advanced crawling). (Requests used; Scrapy was not implemented)
• Content Parsing: Utilize BeautifulSoup to parse the HTML/XML content returned from the HTTP requests to extract relevant data. (Extended with feedparser)
• Data Extraction: Define and implement extraction rules to identify and collect necessary data points (such as incident descriptions, timestamps, tags, etc.) from the parsed content.
• Scraping Ethics Compliance: Include checks against robots.txt to ensure compliance with the websites' scraping policies.
• Basic UI: Develop a simple Flask-based UI to input URLs, initiate scraping, and display basic status updates or results.

Partial

• Error Handling: Ensure robust error handling for network issues, invalid URLs, and unexpected webpage structures to minimize crashes or incorrect data scraping. (HTTP status codes and MongoDB insert exceptions are handled; malformed HTML and unexpected page structures are not)

Changed during implementation

• Data Structuring: Use Pandas to structure the scraped data into a readable and analyzable format, such as a DataFrame. (Changed in favour of MongoDB)
• Data Storage: Store the structured data in an SQLite database. (Implemented in MongoDB through PyMongo instead)

Not implemented

• Rate Limiting: Implement rate limiting to prevent overwhelming the servers of the targeted websites and comply with their scraping policies.
• User-Agent Management: Rotate user agents to avoid getting blocked by the websites' anti-scraping mechanisms.
• Scheduling: Implement Cron Jobs to automate scraping at scheduled intervals if continuous data collection is necessary. (we might use this in the future for automation)

Ethical and legal notes

The paper documents which sources are and are not scrapable, based on each site's Terms of Service (ToS) and robots.txt. The scraper enforces robots.txt for the * user-agent. Do not use it against sources whose ToS forbid automated collection.

Known limitations

• The parser expects a rigid input format. Freeform text is not handled.
• The generator implements a subset of the OpenSCENARIO schema. Some property elements and advanced constructs are not covered.
• The Ollama pipeline runs standalone rather than through the Flask UI.
• Not all fields in the plain-text description map to XML output; a few are parsed and then ignored downstream.

Citation

@thesis{zorin_mercier_2024,
  author = {Zorin, Aleksey and Mercier, Louis},
  title  = {A New Approach to AD/ADAS Test Scenario Generation Using
            Open-Source Intelligence and Large Language Models},
  school = {University of Gothenburg and Chalmers University of Technology},
  year   = {2024}
}