diff --git a/coda_lm_eval/README.md b/coda_lm_eval/README.md new file mode 100644 index 00000000000..9288d73413c --- /dev/null +++ b/coda_lm_eval/README.md @@ -0,0 +1,163 @@ +# CODA-LM +[![arXiv](https://img.shields.io/badge/arXiv-2404.10595-b31b1b.svg?style=plastic)](https://arxiv.org/abs/2404.10595) [![arXiv](https://img.shields.io/badge/Web-CODA_LM-blue.svg?style=plastic)](https://coda-dataset.github.io/coda-lm/) + +This repository contains the implementation of the paper: + +> Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases
+> [Kai Chen](https://kaichen1998.github.io)\*, [Yanze Li]()\*, [Wenhua Zhang]()\*, [Yanxin Liu](), [Pengxiang Li](https://scholar.google.com/citations?user=rUp_4RgAAAAJ&hl=en), [Ruiyuan Gao](https://gaoruiyuan.com/), [Lanqing Hong](https://scholar.google.com.sg/citations?user=2p7x6OUAAAAJ&hl=en)†, [Meng Tian](), [Xinhai Zhao](), [Zhenguo Li](https://scholar.google.com/citations?user=XboZC1AAAAAJ&hl=en&oi=ao), [Dit-Yan Yeung](https://sites.google.com/view/dyyeung), [Huchuan Lu](https://scholar.google.com/citations?user=D3nE0agAAAAJ&hl=en), [Xu Jia](https://stephenjia.github.io/)†
+> *Equal Contribution †Corresponding Author + +

+ +

+ +## Data Preparation + +The instructions for downloading CODA-LM are listed as follows: + +1. Download the image files following the CODA official instructions [here](https://coda-dataset.github.io/download.html#instructions) +2. Download the CODA-LM annotation files and then decompress them in the same root directory. + +| Split | Size | Image Source | Download | +| :---: | :--: | :-----------: | :--------: | +| Train | 4884 | CODA2022 val | [HF Hub](https://huggingface.co/datasets/KaiChen1998/coda-lm) | +| Val | 4384 | CODA2022 test | [HF Hub](https://huggingface.co/datasets/KaiChen1998/coda-lm) | +| Test | 500 | CODA2022 test | [HF Hub](https://huggingface.co/datasets/KaiChen1998/coda-lm) | +| Mini | 50 | CODA2022 test | [HF Hub](https://huggingface.co/datasets/KaiChen1998/coda-lm) | + +Note that: + +1. Images of **CODA-LM train** set come from **CODA2022 val** set, while images of **CODA-LM val and test** sets come from **CODA2022 test** set. +2. **CODA-LM mini** set is a 50-image subset of **CODA-LM val** set for demonstration. + +After decompression, the data organization is listed as follows: + +``` +├── val -- CODA2022 val (we only use images) +│ │── images +│ │ │── *.jpg +├── test -- CODA2022 test (we only use images) +│ │── images +│ │ │── *.jpg +├── CODA-LM +│ │── Train -- CODA-LM train (we use 4884 images from CODA2022 val) +│ │ │── val_*.json +│ │── Val -- CODA-LM val (we use 4384 images from CODA2022 test) +│ │ │── test_*.json +│ │── Test -- CODA-LM test (we use 500 images from CODA2022 test) +│ │ │── test_*.json +│ │── Mini -- CODA-LM mini (a 50-image subset of CODA-LM val) +│ │ │── test_*.json +``` + + + +## Data Format + +The annotation files contains question-answering pairs for all three tasks as following, + +``` +{ + "general_perception":{ + "vehicles": [ -- list containing information on all vehicles + { + "description": , -- description about a single vehicle + "explanation": " -- explanation why it affects the ego car + }, + "vulnerable_road_users": [...], -- list containing information on all VRUs + "traffic signs": [...], -- list containing information on all traffic signs + "traffic lights": [...], -- list containing information on all traffic lights + "traffic cones": [...], -- list containing information on all traffic cones + "barriers": [...], -- list containing information on all barriers + "other objects": [...], -- list containing information on all other objects + "description and explanation": -- summarization of information on all categories + }, + "region_perception":{ + "1": { -- region index + "description and explanation": , -- description of road users in the specific region with explanation on why it affects the ego car + "box": , -- xywh coordinates + "category_name": -- object category + }, + "2": {...}, + "3": {...} + }, + "driving_suggestion": , +} +``` + +## Data Usage +To help users better understand the structure of CODA-LM, we provide a python script to convert our annotations to basic VQA formats, as follows: + +1. Download the data and make sure the directory organization follows [Data Prepration](https://github.com/DLUT-LYZ/CODA-LM?tab=readme-ov-file#data-preparation). + +2. Run `convert2vqa.py` as follows: + + ```bash + # English + python convert2vqa.py --coda_root $CODA_ROOT --codalm_ann_name CODA-LM + + # Chinese + python convert2vqa.py --coda_root $CODA_ROOT --codalm_ann_name CODA-LM-chinese + ``` + +3. After that, the resulting data organization will be like this: + + ``` + ├── val + │ │── images + │ │── images_w_bboxes -- Images with bboxes drawn for region perception + │ │ │── *.jpg + ├── test + │ │── images + │ │── images_w_bboxes -- Images with bboxes drawn for region perception + │ │ │── *.jpg + ├── CODA-LM + │ │── Train + │ │ │── vqa_anno + │ │ │ │── general_perception.jsonl -- VQA annotations for general perception + │ │ │ │── region_perception.jsonl -- VQA annotations for region perception + │ │ │ │── driving_suggestion.jsonl -- VQA annotations for driving suggestion + │ │── Val + │ │ │── vqa_anno + │ │── Test + │ │ │── vqa_anno + │ │── Mini + │ │ │── vqa_anno + ``` + +4. The basic VQA format saves data sample simply with a dictionary containing `question_id`, `image`, `question`, and `answer`, as follows: + + ``` + {"question_id": 0, "image": val/images/0001.jpg, "question": , "answer": } + {"question_id": 1, "image": val/images/0002.jpg, "question": , "answer": } + {"question_id": 2, "image": val/images/0003.jpg, "question": , "answer": } + ... + ``` + +5. Note that for regional perception, there are various possible manners to utilize the bbox annotations. Here we provide a simple implementation by drawing the bboxes with red rectangles on images, which are saved in the `images_w_bboxes` directory. + + ![img](images/visual.png) + + +## Data Annotation + +Check [CODA-LM Annotation Tool](./labeling) for more details. + + +# Evaluation + +Check [CODA-LM Evaluation](./evaluation) for more details. + + + +## Citation + +```bibtex +@article{li2024automated, + title={Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases}, + author={Li, Yanze and Zhang, Wenhua and Chen, Kai and Liu, Yanxin and Li, Pengxiang and Gao, Ruiyuan and Hong, Lanqing and Tian, Meng and Zhao, Xinhai and Li, Zhenguo and others}, + journal={arXiv preprint arXiv:2404.10595}, + year={2024} +} +``` + diff --git a/coda_lm_eval/convert2vqa.py b/coda_lm_eval/convert2vqa.py new file mode 100644 index 00000000000..5e74fd8b1e6 --- /dev/null +++ b/coda_lm_eval/convert2vqa.py @@ -0,0 +1,115 @@ +from PIL import Image, ImageDraw +import os +import numpy as np +import json +import argparse +from tqdm import tqdm + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument("--coda_root", type=str, default="./") + parser.add_argument("--codalm_ann_name", type=str, default="CODA-LM", choices=["CODA-LM", "CODA-LM-chinese"]) + args = parser.parse_args() + + # user notice for directory structure + print("=======================\nBefore started, please make sure your directory has been organized as in https://github.com/DLUT-LYZ/CODA-LM?tab=readme-ov-file#data-preparation\n=======================\n") + + ######################## + # Start pre-processing + ######################## + ann_root = os.path.join(args.coda_root, args.codalm_ann_name) + for split in os.listdir(ann_root): + assert split in ['Train', 'Val', 'Test', 'Mini'] + split_root = os.path.join(ann_root, split) + json_list = sorted([each for each in os.listdir(split_root) if each.endswith('.json')]) + + stage1_index = -1 + stage2_index = -1 + stage3_index = -1 + stage1_all_data = [] + stage2_all_data = [] + stage3_all_data = [] + for json_name in tqdm(json_list, desc=split): + with open(os.path.join(split_root, json_name), 'r', encoding='utf-8') as f: + json_data = json.load(f) + img_dir = json_name.split("_")[0] + img_name = json_name.split("_")[1][:-5] + '.jpg' + + ######################## + # Step 1: Prepare general perception data + ######################## + stage1_index += 1 + stage1_data = dict( + question_id=stage1_index, + image=os.path.join(img_dir, 'images', img_name), + question="There is an image of traffic captured from the perspective of the ego car. Focus on objects influencing the ego car's driving behavior: vehicles (cars, trucks, buses, etc.), vulnerable road users (pedestrians, cyclists, motorcyclists), traffic signs (no parking, warning, directional, etc.), traffic lights (red, green, yellow), traffic cones, barriers, miscellaneous(debris, dustbin, animals, etc.). You must not discuss any objects beyond the seven categories above. Please describe each object's appearance, position, direction, and explain why it affects the ego car's behavior." + ) + if split != 'Test': + stage1_data['answer'] = json_data['general_perception']['description and explanation'] + stage1_all_data.append(stage1_data) + + ######################## + # Step 2: Prepare driving suggestion data + ######################## + stage2_index += 1 + stage2_data = dict( + question_id=stage2_index, + image=os.path.join(img_dir, 'images', img_name), + question="There is an image of traffic captured from the perspective of the ego car. Focus on objects influencing the ego car's driving behavior: vehicles (cars, trucks, buses, etc.), vulnerable road users (pedestrians, cyclists, motorcyclists), traffic signs (no parking, warning, directional, etc.), traffic lights (red, green, yellow), traffic cones, barriers, miscellaneous(debris, dustbin, animals, etc.). You must not discuss any objects beyond the seven categories above. Please provide driving suggestions for the ego car based on the current scene." + ) + if split != 'Test': + stage2_data['answer'] = json_data['driving_suggestion'] + stage2_all_data.append(stage2_data) + + ######################## + # Step 3: Prepare region perception data + ######################## + regional_data = json_data["region_perception"] + for key, value in regional_data.items(): + # preprare image paths + output_root = os.path.join(args.coda_root, img_dir, 'images_w_boxes') + output_path = os.path.join(output_root, "{}_object_{}.jpg".format(json_name.split("_")[1][:-5], key)) + os.makedirs(output_root, exist_ok=True) + + # prepare images + if not os.path.exists(output_path): + img = Image.open(os.path.join(args.coda_root, img_dir, 'images', img_name)) + draw = ImageDraw.Draw(img) + rect = [value['box'][0], value['box'][1], value['box'][0] + value['box'][2], value['box'][1] + value['box'][3]] + draw.rectangle(rect, outline="red", width=2) + img.save(output_path) + + # prepare annotation + stage3_index += 1 + stage3_data = dict( + question_id=stage3_index, + image=os.path.join(img_dir, 'images_w_boxes', "{}_object_{}.jpg".format(json_name.split("_")[1][:-5], key)), + question="Please describe the object inside the red rectangle in the image and explain why it affect ego car driving." + ) + if split != 'Test': + stage3_data['answer'] = value['description and explanation'] + stage3_all_data.append(stage3_data) + + ######################## + # Step 4: Save annotation + ######################## + save_root = os.path.join(split_root, 'vqa_anno') + os.makedirs(save_root, exist_ok=True) + + # save stage1 + with open(os.path.join(save_root, 'general_perception.jsonl'), 'w') as file: + for entry in stage1_all_data: + json_str = json.dumps(entry) + file.write(json_str + '\n') + + # save stage2 + with open(os.path.join(save_root, 'driving_suggestion.jsonl'), 'w') as file: + for entry in stage2_all_data: + json_str = json.dumps(entry) + file.write(json_str + '\n') + + # save stage3 + with open(os.path.join(save_root, 'region_perception.jsonl'), 'w') as file: + for entry in stage3_all_data: + json_str = json.dumps(entry) + file.write(json_str + '\n') diff --git a/coda_lm_eval/evaluation/README.md b/coda_lm_eval/evaluation/README.md new file mode 100644 index 00000000000..6413ad3e266 --- /dev/null +++ b/coda_lm_eval/evaluation/README.md @@ -0,0 +1,50 @@ +# CODA-LM Evaluation + +## Data Preparation + +1. First of all, follow the instructions on [CODA-LM submission](https://coda-dataset.github.io/w-coda2024/track1/index.html#submission) to run inference with your LVLM and save your results as requested. Your results should be saved in three `jsonl` files for three tasks separately. +2. If you are using a subset of CODA-LM validation set (e.g., CODA-LM Mini set) for evaluation, you need to copy the corresponding JSON files in a separate `$ROOT_TO_GT` directory like `./CODA_LM/Mini`. Directly setting `$ROOT_TO_GT=./CODA-LM/Val`, but not running inference on the whole validation set will result in a running error. +3. Now the data organization will be like: + +``` +├── CODA_LM +│ ├── Train +│ ├── Val +│ ├── Test +│ ├── ROOT_TO_GT +│ │ │── test_*.json +├── ROOT_TO_RESULTS +│ │── general_perception_answer.jsonl +│ │── region_perception_answer.jsonl +│ │── driving_suggestion_answer.jsonl +``` + + + +## Instructions + +0. Install additional dependencies for evaluation via pip. + + ```bash + pip install OpenAI + ``` + +1. Insert the ground truth `label name` of each corner case object collected in regional perception. + + ```bash + # Results will be saved in $ROOT_TO_RESULTS/region_perception_answer_w_label.jsonl + python convert2eval.py --reference_path ./CODA-LM/$ROOT_TO_GT --prediction_path $ROOT_TO_RESULTS/region_perception_answer.jsonl + ``` + +2. Run evaluation for each task separately. By default, we prompt `gpt-4o-2024-05-13` for evaluation. + + ```bash + # General perception + python stage1_eval_batch.py --reference_path ./CODA-LM/$ROOT_TO_GT --prediction_path $ROOT_TO_RESULTS/general_perception_answer.jsonl --save_path eval/general_perception_answer --model_name gpt-4o-2024-05-13 --api_key $OPENAI_KEY + + # Driving suggestion + python stage2_eval_batch.py --reference_path ./CODA-LM/$ROOT_TO_GT --prediction_path $ROOT_TO_RESULTS/driving_suggestion_answer.jsonl --save_path eval/driving_suggestion_answer --model_name gpt-4o-2024-05-13 --api_key $OPENAI_KEY + + # Regional perception + python stage3_eval_batch.py --reference_path ./CODA-LM/$ROOT_TO_GT --prediction_path $ROOT_TO_RESULTS/region_perception_answer_w_label.jsonl --save_path eval/region_perception_answer_w_label --model_name gpt-4o-2024-05-13 --api_key $OPENAI_KEY + ``` diff --git a/coda_lm_eval/evaluation/convert2eval.py b/coda_lm_eval/evaluation/convert2eval.py new file mode 100644 index 00000000000..c9f1c6b74f8 --- /dev/null +++ b/coda_lm_eval/evaluation/convert2eval.py @@ -0,0 +1,39 @@ +import os +import json +import argparse +from tqdm import tqdm + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--reference_path", type=str, default="./CODA-LM/Test") + parser.add_argument("--prediction_path", type=str, default="./region_perception_answer.jsonl") + args = parser.parse_args() + + ###################### + # load ground truth + ###################### + reference_data = {} + for each in tqdm(os.listdir(args.reference_path), desc='Loading GT'): + if not each.endswith('.json'): + continue + with open(os.path.join(args.reference_path, each), "r", encoding='utf-8') as f: + each_data = json.load(f) + reference_data[each[:-5]] = each_data + + ###################### + # load prediction + ###################### + prediction_data = [json.loads(q) for q in open(os.path.expanduser(args.prediction_path), "r", encoding='utf-8')] + for each in tqdm(prediction_data, desc='Processing prediction'): + image_name = each['image'].split('/')[-1] + json_name = 'test_' + image_name.split('_')[0] + object_id = image_name.split('_')[-1][:-4] + assert object_id.isdigit() + each['label_name'] = reference_data[json_name]['region_perception'][object_id]['category_name'] + + ###################### + # save converted prediction + ###################### + with open(args.prediction_path[:-6] + '_w_label.jsonl', "w", encoding='utf-8') as file: + for each in prediction_data: + file.write(json.dumps(each) + "\n") \ No newline at end of file diff --git a/coda_lm_eval/evaluation/scene_few_shot/high.json b/coda_lm_eval/evaluation/scene_few_shot/high.json new file mode 100644 index 00000000000..0d11a779375 --- /dev/null +++ b/coda_lm_eval/evaluation/scene_few_shot/high.json @@ -0,0 +1,5 @@ +{ + "reference": "There is a black SUV in the opposite direction to the ego car in the left lane and a white van in front of the ego car in the ego car lane. A motorcyclist wearing yelow hat is visible on the right side of the ego car, moving through the intersection. Motorcyclists are vulnerable, potentially leading to safety-critical situations. A directional sign indicating lanes for straight travel, assisting the ego car in navigating its route. A green traffic light is visible straight ahead, allowing traffic to proceed through the intersection. On the side of the road, the orange barrier marks a risky area, alerting the ego car to potential hazards and the need for caution.", + "prediction": "The green traffic light ahead informs the ego car that it has the right of way to proceed. The presence of a pedestrian crossing indicates that drivers should be alert for pedestrians. A black sedan and a sliver car in front of th ego car directly influences the ego car's driving behavior, requiring the driver to adjust speed and maintain a safe following distance. To the right of the image, there is a person riding a scooter. This is a vulnerable road user who may behave unpredictably. There are barriers set up along the edge of the road on the right side, indicating some form of road work or a closed-off area. The ego car should stay clear of these barriers to avoid entering a potentially unsafe area.", + "response": "The prediction text correctly identifies the green traffic light, barriers, two vehicles and the person riding a scooter. Although the colors and types of vehicles are slightly different, silver and white colors are similar, both the reference text and the prediction text describe small vehicles (SUV, sedan, van, and car are all equivalent), so SUV is predicted to be sedan, and van is predicted to be car. While there is no explicit mention of crosswalk markings in the reference text, they usually appear at intersections, so this is not an illusion. However, it did not mention the directional sign indicating straight line travel and the relative position of the vehicle was incorrect. So, Rating: [[8]]." +} \ No newline at end of file diff --git a/coda_lm_eval/evaluation/scene_few_shot/low.json b/coda_lm_eval/evaluation/scene_few_shot/low.json new file mode 100644 index 00000000000..d34a75b47ab --- /dev/null +++ b/coda_lm_eval/evaluation/scene_few_shot/low.json @@ -0,0 +1,5 @@ +{ + "reference": "Numerous cars are parked on both sides of the road, resulting in a narrower space available for driving, which requires drivers to pay close attention. Ahead, there is a black truck traveling in the same lane in front of ego car, necessitating the maintenance of a safe driving distance. On the right, there are motorcyclists and cyclists moving in the same direction as the ego car, with a significant number of motorcycles and bicycles parked on the road. Traffic cones placed around indicate potentially hazardous areas. The driver's vehicle should recognize these to avoid obstacles, reducing the possibility of damage or accidents. A traffic light is visible ahead, but its color is unclear. Traffic lights control the flow of traffic, and the driver's vehicle must respond correctly to their signals to ensure safe operation. There is a no-entry sign ahead.", + "prediction": "In this scenario, several objects could influence the driver's behavior. Motorcycles parked along the road require the driver to be cautious and maintain a safe distance from them to prevent accidents. Ahead, there is a black truck, and the driver should keep a proper distance. Pedestrians are present on the sidewalk ahead, and the driver should pay attention to them and be prepared to stop or slow down as necessary to ensure driving safety. There are many trees and buildings on both sides, which should not distract the driver's attention. To avoid being late, the driver wants to pass through the traffic light before it turns red, intending to overtake the vehicles ahead.", + "response": "The prediction text successfully mentions the motorcycles parked by the road and the truck ahead. However, it completely overlooks the vehicles parked on both sides mentioned in the reference text, the people riding motorcycles and bicycles on the right side, and ignores the traffic cones and the no-entry sign. Additionally, the prediction text hallucinates the presence of pedestrians. As for the trees and buildings, they actually do not have an impact and do not need to be considered. The prediction text imagines the driver's psychological thoughts, which, although not present in the reference text, are acceptable if reasonable. So, Rating: [[2]]." +} \ No newline at end of file diff --git a/coda_lm_eval/evaluation/stage1_eval_batch.py b/coda_lm_eval/evaluation/stage1_eval_batch.py new file mode 100644 index 00000000000..2a9041b133c --- /dev/null +++ b/coda_lm_eval/evaluation/stage1_eval_batch.py @@ -0,0 +1,226 @@ +from openai import OpenAI +from concurrent.futures import ThreadPoolExecutor, wait +from functools import partial +from tqdm import tqdm +import time +import os +import json +import argparse + +class GPTBatcher: + """ + Borrow from https://github.com/fengsxy/gpt_batch + + Parameters: + api_key (str): API key for authenticating requests to the OpenAI API. + model_name (str, optional): Specifies the GPT model version. Default is 'gpt-3.5-turbo-0125'. + system_prompt (str, optional): Initial text or question to seed the model with. Default is empty. + temperature (float, optional): Sets the creativity of the responses. Default is 1. + num_workers (int, optional): Number of parallel workers for request handling. Default is 64. + timeout_duration (int, optional): Timeout for API responses in seconds. Default is 60. + retry_attempts (int, optional): How many times to retry a failed request. Default is 2. + """ + + def __init__(self, + api_key, + model_name="gpt-3.5-turbo", + system_prompt="", + temperature=0, + num_workers=32, + timeout_duration=60, + retry_attempts=2, + api_base_url=None): + + self.client = OpenAI(api_key=api_key, base_url = api_base_url) + self.model_name = model_name + self.system_prompt = "You are an impartial judge tasked with evaluating the quality of predicted text provided by autonomous driving AI assistant. You will compare this prediction text to a reference text, focusing on the description of objects that influence the driving behavior of ego car, and the explanation of why these objects impact. Your evaluation criteria should include accuracy(checking if the predicted text correctly identifies objects mentioned the reference text), suppression hallucination(ensuring that objects not mentioned in the reference text are not erroneously included in the predicted text), correlation(sessing if the reasons for the objects' impact on the ego car's driving behavior are consistent between the reference and predicted text). Be as objective as possible. Do not allow the length of the predicted text to influence your evaluation. Maximize your text comprehension capabilities to freely match objects with high similarity, appropriately ignoring the relative positions and color attributes of the objects. After providing your short explanation, you must rate the response on a scale from 1 to 10 by strictly following this format: \"[[rating]]\", for example: \"Rating: [[10]]\"." + self.temperature = temperature + self.num_workers = num_workers + self.timeout_duration = timeout_duration + self.retry_attempts = retry_attempts + self.miss_index =[] + if api_base_url: + self.client.base_url = api_base_url + + + def create_messages(self, message): + ret = [] + # system prompt + ret.append({ + "role": "system", + "content": self.system_prompt + }) + + # few shot example + few_shot = os.path.join(os.path.dirname(os.path.abspath(__file__)), "scene_few_shot") + with open(os.path.join(few_shot, "high.json")) as f: + high_data = json.load(f) + with open(os.path.join(few_shot, "low.json")) as f: + low_data = json.load(f) + + template = "[The Start of Reference Text]\n{}\n[The End of Reference Text]\n\n[The Start of Prediction Text]\n{}\n[The End of Prediction Text]" + + # high example + ret.append({ + "role": "user", + "content": template.format(high_data["reference"], high_data["prediction"]) + }) + ret.append({ + "role": "assistant", + "content": high_data["response"] + }) + + # low example + ret.append({ + "role": "user", + "content": template.format(low_data["reference"], low_data["prediction"]) + }) + ret.append({ + "role": "assistant", + "content": low_data["response"] + }) + + ret.append({ + "role": "user", + "content": template.format(message["reference"], message["prediction"]) + }) + return ret + + def get_attitude(self, ask_text): + index, ask_text = ask_text + try: + completion = self.client.chat.completions.create( + model=self.model_name, + messages=ask_text, + temperature=self.temperature, + ) + return (index, completion.choices[0].message.content) + except Exception as e: + print(f"Error occurred: {e}") + self.miss_index.append(index) + return (index, None) + + def process_attitude(self, message_list): + new_list = [] + num_workers = self.num_workers + timeout_duration = self.timeout_duration + retry_attempts = self.retry_attempts + + executor = ThreadPoolExecutor(max_workers=num_workers) + message_chunks = list(self.chunk_list(message_list, num_workers)) + try: + for chunk in tqdm(message_chunks, desc="Processing messages"): + future_to_message = {executor.submit(self.get_attitude, message): message for message in chunk} + for _ in range(retry_attempts): + done, not_done = wait(future_to_message.keys(), timeout=timeout_duration) + for future in not_done: + future.cancel() + new_list.extend(future.result() for future in done if future.done()) + if len(not_done) == 0: + break + future_to_message = {executor.submit(self.get_attitude, future_to_message[future]): future for future in not_done} + except Exception as e: + print(f"Error occurred: {e}") + finally: + executor.shutdown(wait=False) + return new_list + + def complete_attitude_list(self,attitude_list, max_length): + completed_list = [] + current_index = 0 + for item in attitude_list: + index, value = item + # Fill in missing indices + while current_index < index: + completed_list.append((current_index, None)) + current_index += 1 + # Add the current element from the list + completed_list.append(item) + current_index = index + 1 + while current_index < max_length: + print("Filling in missing index", current_index) + self.miss_index.append(current_index) + completed_list.append((current_index, None)) + current_index += 1 + return completed_list + + def chunk_list(self, lst, n): + """Yield successive n-sized chunks from lst.""" + for i in range(0, len(lst), n): + yield lst[i:i + n] + + def handle_message_list(self,message_list): + indexed_list = [(index, data) for index, data in enumerate(message_list)] + max_length = len(indexed_list) + attitude_list = self.process_attitude(indexed_list) + attitude_list.sort(key=lambda x: x[0]) + attitude_list = self.complete_attitude_list(attitude_list, max_length) + attitude_list = [x[1] for x in attitude_list] + return attitude_list + + def get_miss_index(self): + return self.miss_index + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--reference_path", type=str, default="ann") + parser.add_argument("--prediction_path", type=str, default="prediction/xx.jsonl") + parser.add_argument("--save_path", type=str, default="eval_res/xx") + parser.add_argument("--num_workers", type=int, default=32) + parser.add_argument("--model_name", type=str, default="gpt-3.5-turbo") + parser.add_argument("--api_key", type=str, default=None) + parser.add_argument("--api_base_url", type=str, default=None) + parser.add_argument("--retry_attempts", type=int, default=16) + args = parser.parse_args() + + os.makedirs(args.save_path, exist_ok=True) + json_list = sorted([each for each in os.listdir(args.reference_path) if each.endswith(".json")]) + answers = [json.loads(q) for q in open(os.path.expanduser(args.prediction_path), "r", encoding='utf-8')] + answers.sort(key=lambda element: element['question_id']) + assert len(json_list) == len(answers) + batcher = GPTBatcher( + api_key=args.api_key, + model_name=args.model_name, + num_workers=args.num_workers, + retry_attempts=args.retry_attempts, + api_base_url=args.api_base_url) + + all_score = [] + rets = [] + for idx, json_name in tqdm(enumerate(json_list)): + assert idx == int(answers[idx]['question_id']) + message= dict() + message["prediction"] = answers[idx]['answer'] + + with open(os.path.join(args.reference_path, json_name), "r", encoding='utf-8') as f: + data = json.load(f) + general_data = data["general_perception"] + message["reference"] = general_data['description and explanation'] + ret = batcher.create_messages(message) + rets.append(ret) + + results = batcher.handle_message_list(rets) + for idx, json_name in tqdm(enumerate(json_list)): + output = results[idx] + txt_name = json_name.replace(".json", ".txt") + if output == None: + continue + print(f"Missing {json_name} output") + + try: + all_score.append(int(output.split("Rating: [[")[1].split("]]")[0])) + except: + try: + all_score.append(int(output.split("rating is: [[")[1].split("]]")[0])) + except: + try: + all_score.append(int(output.split("[[")[1].split("]]")[0])) + except: + print(f"Missing extract score from {txt_name}") + with open(os.path.join(args.save_path, txt_name), "w", encoding='utf-8') as f: + f.write(output) + + # cal score + print(f"Stage1_score: {sum(all_score)/len(all_score)}") + with open(os.path.join(args.save_path, "all_score.txt"), "w", encoding='utf-8') as f: + f.write(f"Stage1_score: {sum(all_score)/len(all_score)}") \ No newline at end of file diff --git a/coda_lm_eval/evaluation/stage2_eval_batch.py b/coda_lm_eval/evaluation/stage2_eval_batch.py new file mode 100644 index 00000000000..4bf443cf101 --- /dev/null +++ b/coda_lm_eval/evaluation/stage2_eval_batch.py @@ -0,0 +1,223 @@ +from openai import OpenAI +from concurrent.futures import ThreadPoolExecutor, wait +from functools import partial +from tqdm import tqdm +import time +import os +import json +import argparse + +class GPTBatcher: + """ + Borrow from https://github.com/fengsxy/gpt_batch + + Parameters: + api_key (str): API key for authenticating requests to the OpenAI API. + model_name (str, optional): Specifies the GPT model version. Default is 'gpt-3.5-turbo-0125'. + system_prompt (str, optional): Initial text or question to seed the model with. Default is empty. + temperature (float, optional): Sets the creativity of the responses. Default is 1. + num_workers (int, optional): Number of parallel workers for request handling. Default is 64. + timeout_duration (int, optional): Timeout for API responses in seconds. Default is 60. + retry_attempts (int, optional): How many times to retry a failed request. Default is 2. + """ + + def __init__(self, + api_key, + model_name="gpt-3.5-turbo-0125", + system_prompt="", + temperature=0, + num_workers=32, + timeout_duration=60, + retry_attempts=2, + api_base_url=None): + + self.client = OpenAI(api_key=api_key, base_url = api_base_url) + self.model_name = model_name + self.system_prompt = "You are an impartial judge tasked with evaluating the quality of predicted text provided by autonomous driving AI assistant. You will compare this prediction to a reference text, focusing on the ego car driving suggestion. Your evaluation should consider rationality, relevance, level of detail of the response. Predicted text should be specific and actionable, rather than vague or overly broad. Identify and correct any mistakes. Do not allow the length of the predicted text to influence your evaluation. Do not allow the length of the predicted text to influence your evaluation. Maximize your text comprehension capabilities to freely match objects with high similarity, appropriately ignoring the relative positions and color attributes of the objects. After providing your short explanation, you must rate the response on a scale from 1 to 10 by strictly following this format: \"[[rating]]\", for example: \"Rating: [[10]]\"." + self.temperature = temperature + self.num_workers = num_workers + self.timeout_duration = timeout_duration + self.retry_attempts = retry_attempts + self.miss_index =[] + if api_base_url: + self.client.base_url = api_base_url + + + def create_messages(self, message): + ret = [] + # system prompt + ret.append({ + "role": "system", + "content": self.system_prompt + }) + + # few shot example + few_shot = os.path.join(os.path.dirname(os.path.abspath(__file__)), "suggestion_few_shot") + with open(os.path.join(few_shot, "high.json")) as f: + high_data = json.load(f) + with open(os.path.join(few_shot, "low.json")) as f: + low_data = json.load(f) + + template = "[The Start of Reference Text]\n{}\n[The End of Reference Text]\n\n[The Start of Prediction Text]\n{}\n[The End of Prediction Text]" + + # high example + ret.append({ + "role": "user", + "content": template.format(high_data["reference"], high_data["prediction"]) + }) + ret.append({ + "role": "assistant", + "content": high_data["response"] + }) + + # low example + ret.append({ + "role": "user", + "content": template.format(low_data["reference"], low_data["prediction"]) + }) + ret.append({ + "role": "assistant", + "content": low_data["response"] + }) + + ret.append({ + "role": "user", + "content": template.format(message["reference"], message["prediction"]) + }) + return ret + + def get_attitude(self, ask_text): + index, ask_text = ask_text + try: + completion = self.client.chat.completions.create( + model=self.model_name, + messages=ask_text, + temperature=self.temperature, + ) + return (index, completion.choices[0].message.content) + except Exception as e: + print(f"Error occurred: {e}") + self.miss_index.append(index) + return (index, None) + + def process_attitude(self, message_list): + new_list = [] + num_workers = self.num_workers + timeout_duration = self.timeout_duration + retry_attempts = self.retry_attempts + + executor = ThreadPoolExecutor(max_workers=num_workers) + message_chunks = list(self.chunk_list(message_list, num_workers)) + try: + for chunk in tqdm(message_chunks, desc="Processing messages"): + future_to_message = {executor.submit(self.get_attitude, message): message for message in chunk} + for _ in range(retry_attempts): + done, not_done = wait(future_to_message.keys(), timeout=timeout_duration) + for future in not_done: + future.cancel() + new_list.extend(future.result() for future in done if future.done()) + if len(not_done) == 0: + break + future_to_message = {executor.submit(self.get_attitude, future_to_message[future]): future for future in not_done} + except Exception as e: + print(f"Error occurred: {e}") + finally: + executor.shutdown(wait=False) + return new_list + + def complete_attitude_list(self,attitude_list, max_length): + completed_list = [] + current_index = 0 + for item in attitude_list: + index, value = item + # Fill in missing indices + while current_index < index: + completed_list.append((current_index, None)) + current_index += 1 + # Add the current element from the list + completed_list.append(item) + current_index = index + 1 + while current_index < max_length: + print("Filling in missing index", current_index) + self.miss_index.append(current_index) + completed_list.append((current_index, None)) + current_index += 1 + return completed_list + + def chunk_list(self, lst, n): + """Yield successive n-sized chunks from lst.""" + for i in range(0, len(lst), n): + yield lst[i:i + n] + + def handle_message_list(self,message_list): + indexed_list = [(index, data) for index, data in enumerate(message_list)] + max_length = len(indexed_list) + attitude_list = self.process_attitude(indexed_list) + attitude_list.sort(key=lambda x: x[0]) + attitude_list = self.complete_attitude_list(attitude_list, max_length) + attitude_list = [x[1] for x in attitude_list] + return attitude_list + + def get_miss_index(self): + return self.miss_index + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--reference_path", type=str, default="ann") + parser.add_argument("--prediction_path", type=str, default="prediction/xx.jsonl") + parser.add_argument("--save_path", type=str, default="eval_res/xx") + parser.add_argument("--num_workers", type=int, default=32) + parser.add_argument("--model_name", type=str, default="gpt-3.5-turbo") + parser.add_argument("--api_key", type=str, default=None) + parser.add_argument("--api_base_url", type=str, default=None) + parser.add_argument("--retry_attempts", type=int, default=10) + args = parser.parse_args() + + os.makedirs(args.save_path, exist_ok=True) + json_list = sorted([each for each in os.listdir(args.reference_path) if each.endswith(".json")]) + answers = [json.loads(q) for q in open(os.path.expanduser(args.prediction_path), "r", encoding='utf-8')] + answers.sort(key=lambda element: element['question_id']) + assert len(json_list) == len(answers) + batcher = GPTBatcher( + api_key=args.api_key, + model_name=args.model_name, + num_workers=args.num_workers, + api_base_url=args.api_base_url) + + all_score = [] + rets = [] + for idx, json_name in tqdm(enumerate(json_list)): + assert idx == int(answers[idx]['question_id']) + message= dict() + message["prediction"] = answers[idx]['answer'] + + with open(os.path.join(args.reference_path, json_name), "r", encoding='utf-8') as f: + data = json.load(f) + message["reference"] = data["driving_suggestion"] + ret = batcher.create_messages(message) + rets.append(ret) + + results = batcher.handle_message_list(rets) + for idx, json_name in tqdm(enumerate(json_list)): + output = results[idx] + txt_name = json_name.replace(".json", ".txt") + if output == None: + continue + print(f"Missing {json_name} output") + + try: + all_score.append(int(output.split("Rating: [[")[1].split("]]")[0])) + except: + try: + all_score.append(int(output.split("rating is: [[")[1].split("]]")[0])) + except: + try: + all_score.append(int(output.split("[[")[1].split("]]")[0])) + except: + print(f"Missing extract score from {txt_name}") + with open(os.path.join(args.save_path, txt_name), "w") as f: + f.write(output) + + print(f"Stage2_score: {sum(all_score)/len(all_score)}") + with open(os.path.join(args.save_path, "all_score.txt"), "w", encoding='utf-8') as f: + f.write(f"Stage2_score: {sum(all_score)/len(all_score)}") \ No newline at end of file diff --git a/coda_lm_eval/evaluation/stage3_eval_batch.py b/coda_lm_eval/evaluation/stage3_eval_batch.py new file mode 100644 index 00000000000..7cd1af87f97 --- /dev/null +++ b/coda_lm_eval/evaluation/stage3_eval_batch.py @@ -0,0 +1,282 @@ +from openai import OpenAI +from concurrent.futures import ThreadPoolExecutor, wait +from functools import partial +from tqdm import tqdm +import time +import os +import json +import argparse +from collections import defaultdict + +class RegionEval(object): + + def __init__(self, reference_path, prediction_path): + self.system_prompt = "You are an impartial judge tasked with evaluating text similarity and relevance of the reference text and autonomous driving AI assistant's predicted text. Be as objective as possible. Do not allow the length of the predicted text to influence your evaluation. After providing your short explanation, you must rate on a scale from 1 to 10 by strictly following this format: \"[[rating]]\", for example: \"Rating: [[10]]\"." + self.gt_data = defaultdict(list) + self.predict_data = defaultdict(list) + self.load_gt(reference_path) + self.load_predict(prediction_path) + # import pdb; pdb.set_trace() + for each in self.gt_data.keys(): + assert len(self.gt_data[each]) == len(self.predict_data[each]) + + def load_predict(self, prediction_path): + answers = [json.loads(q.strip()) for q in open(os.path.expanduser(prediction_path), "r", encoding='utf-8')] + answers.sort(key=lambda element: element['question_id']) + for line_data in answers: + new_label_name = self.convert_label(line_data["label_name"]) + self.predict_data[new_label_name].append({ + "question": line_data["question"], + "answer": line_data["answer"] + }) + + def load_gt(self, reference_path): + gt_list = sorted([each for each in os.listdir(reference_path) if each.endswith(".json")]) + for json_name in gt_list: + with open(os.path.join(reference_path, json_name), 'r', encoding='utf-8') as f: + json_data = json.load(f) + json_data = json_data["region_perception"] + for object_id, object_data in json_data.items(): + new_label_name = self.convert_label(object_data['category_name']) + + self.gt_data[new_label_name].append({ + 'image_name': f"{json_name.split('.')[0]}_object_{object_id}.jpg", + 'box_type': 'xywh', # top xy + 'bbox': object_data['box'], + 'answer': object_data['description and explanation'] + }) + + def convert_label(self, category_name): + label_dict = { + "vehicle": ["car", "truck", "tram", "tricycle","bus", "trailer", "construction_vehicle", "recreational_vehicle"], + "vru": ["pedestrian", "cyclist", "bicycle", "moped", "motorcycle", "stroller", "wheelchair", "cart"], + "traffic_sign": ["warning_sign", "traffic_sign"], + "traffic_light": ["traffic_light"], + "traffic_cone": ["traffic_cone"], + "barrier": ["barrier", "bollard"], + "miscellaneous": ["dog", "cat", "sentry_box", "traffic_box", "traffic_island", "debris", "suitcace", "dustbin", "concrete_block", "machinery", "chair", "phone_booth", "basket", "cardboard", "carton", "garbage", "garbage_bag", "plastic_bag", "stone", "tire", "misc"], + } + self.label_info = { + "vehicle": 0, + "vru": 1, + "traffic_sign": 2, + "traffic_light": 3, + "traffic_cone": 4, + "barrier": 5, + "miscellaneous": 6 + } + for new_name, label_info in label_dict.items(): + if category_name in label_info: + return new_name + + def get_class(self): + return self.label_info + + def create_messages(self, message): + ret = [] + # system prompt + ret.append({ + "role": "system", + "content": self.system_prompt + }) + + template = "[The Start of Reference Text]\n{}\n[The End of Reference Text]\n\n[The Start of Prediction Text]\n{}\n[The End of Prediction Text]" + + ret.append({ + "role": "user", + "content": template.format(message["reference"], message["prediction"]) + }) + + return ret + + def get_class_messages(self, label_name): + results = [] + txt_names = [] + gt, pred = self.gt_data[label_name], self.predict_data[label_name] # List[Dict] + for index in tqdm(range(len(gt))): + hypo = pred[index] + ref = gt[index] + message = dict() + message["prediction"] = hypo["answer"] + message["reference"] = ref["answer"] + results.append(self.create_messages(message)) + txt_names.append(f"{ref['image_name'].split('.')[0]}.txt") + + return results, txt_names + +class GPTBatcher: + """ + Borrow from https://github.com/fengsxy/gpt_batch + + Parameters: + api_key (str): API key for authenticating requests to the OpenAI API. + model_name (str, optional): Specifies the GPT model version. Default is 'gpt-3.5-turbo-0125'. + system_prompt (str, optional): Initial text or question to seed the model with. Default is empty. + temperature (float, optional): Sets the creativity of the responses. Default is 1. + num_workers (int, optional): Number of parallel workers for request handling. Default is 64. + timeout_duration (int, optional): Timeout for API responses in seconds. Default is 60. + retry_attempts (int, optional): How many times to retry a failed request. Default is 2. + """ + + def __init__(self, + api_key, + model_name="gpt-3.5-turbo-0125", + system_prompt="", + temperature=0, + num_workers=64, + timeout_duration=60, + retry_attempts=2, + api_base_url=None): + + self.client = OpenAI(api_key=api_key, base_url = api_base_url) + self.model_name = model_name + self.system_prompt = system_prompt + self.temperature = temperature + self.num_workers = num_workers + self.timeout_duration = timeout_duration + self.retry_attempts = retry_attempts + self.miss_index =[] + if api_base_url: + self.client.base_url = api_base_url + + def get_attitude(self, ask_text): + index, ask_text = ask_text + try: + completion = self.client.chat.completions.create( + model=self.model_name, + messages=ask_text, + temperature=self.temperature, + ) + return (index, completion.choices[0].message.content) + except Exception as e: + print(f"Error occurred: {e}") + self.miss_index.append(index) + return (index, None) + + def process_attitude(self, message_list): + new_list = [] + num_workers = self.num_workers + timeout_duration = self.timeout_duration + retry_attempts = self.retry_attempts + + executor = ThreadPoolExecutor(max_workers=num_workers) + message_chunks = list(self.chunk_list(message_list, num_workers)) + try: + for chunk in tqdm(message_chunks, desc="Processing messages"): + future_to_message = {executor.submit(self.get_attitude, message): message for message in chunk} + for _ in range(retry_attempts): + done, not_done = wait(future_to_message.keys(), timeout=timeout_duration) + for future in not_done: + future.cancel() + new_list.extend(future.result() for future in done if future.done()) + if len(not_done) == 0: + break + future_to_message = {executor.submit(self.get_attitude, future_to_message[future]): future for future in not_done} + except Exception as e: + print(f"Error occurred: {e}") + finally: + executor.shutdown(wait=False) + return new_list + + def complete_attitude_list(self,attitude_list, max_length): + completed_list = [] + current_index = 0 + for item in attitude_list: + index, value = item + # Fill in missing indices + while current_index < index: + completed_list.append((current_index, None)) + current_index += 1 + # Add the current element from the list + completed_list.append(item) + current_index = index + 1 + while current_index < max_length: + print("Filling in missing index", current_index) + self.miss_index.append(current_index) + completed_list.append((current_index, None)) + current_index += 1 + return completed_list + + def chunk_list(self, lst, n): + """Yield successive n-sized chunks from lst.""" + for i in range(0, len(lst), n): + yield lst[i:i + n] + + def handle_message_list(self, message_list): + indexed_list = [(index, data) for index, data in enumerate(message_list)] + max_length = len(indexed_list) + attitude_list = self.process_attitude(indexed_list) + attitude_list.sort(key=lambda x: x[0]) + attitude_list = self.complete_attitude_list(attitude_list, max_length) + attitude_list = [x[1] for x in attitude_list] + return attitude_list + + def get_miss_index(self): + return self.miss_index + + +if __name__ == "__main__": + + parser = argparse.ArgumentParser() + parser.add_argument("--reference_path", type=str, default="ann") + parser.add_argument("--prediction_path", type=str, default="prediction/xx.jsonl") + parser.add_argument("--save_path", type=str, default="eval/xx") + parser.add_argument("--num_workers", type=int, default=1) + parser.add_argument("--model_name", type=str, default="gpt-3.5-turbo") + parser.add_argument("--api_key", type=str, default=None) + parser.add_argument("--api_base_url", type=str, default=None) + parser.add_argument("--retry_attempts", type=int, default=10) + + args = parser.parse_args() + os.makedirs(args.save_path, exist_ok=True) + os.makedirs(os.path.join(args.save_path, "gpt_result"), exist_ok=True) + region_eval = RegionEval(args.reference_path, args.prediction_path) + batcher = GPTBatcher( + api_key=args.api_key, + model_name=args.model_name, + num_workers=args.num_workers, + retry_attempts=args.retry_attempts, + api_base_url=args.api_base_url) + + all_score = [] + label_info = region_eval.get_class() + for label_name in tqdm(label_info.keys()): + cls_score = [] + + rets, txt_names = region_eval.get_class_messages(label_name) + results = batcher.handle_message_list(rets) + + for idx, txt_name in tqdm(enumerate(txt_names)): + output = results[idx] + if output == None: + continue + print(f"Missing {txt_name} output") + + try: + cls_score.append(int(output.split("Rating: [[")[1].split("]]")[0])) + except: + try: + cls_score.append(int(output.split("rating is: [[")[1].split("]]")[0])) + except: + try: + cls_score.append(int(output.split("[[")[1].split("]]")[0])) + except: + print(f"Missing extract score from {txt_name}") + + with open(os.path.join(args.save_path, "gpt_result", txt_name), "w") as f: + f.write(output) + + with open(os.path.join(args.save_path, f"{label_name}.txt"), "w") as fp: + if len(cls_score) == 0: + fp.write(f"computing gpt score: 0.0\n") + else: + fp.write(f"computing gpt score: {sum(cls_score) / len(cls_score)}\n") + if len(cls_score) == 0: + print(f"Label: {label_name}, GPT-Score: 0.0") + else: + print(f"Label: {label_name}, GPT-Score: {sum(cls_score) / len(cls_score)}") + + all_score += cls_score + print(f"Stage3_score: {sum(all_score)/len(all_score)}") + with open(os.path.join(args.save_path, "all_score.txt"), "w", encoding='utf-8') as f: + f.write(f"Stage3_score: {sum(all_score)/len(all_score)}") \ No newline at end of file diff --git a/coda_lm_eval/evaluation/suggestion_few_shot/high.json b/coda_lm_eval/evaluation/suggestion_few_shot/high.json new file mode 100644 index 00000000000..1c0bac39ed0 --- /dev/null +++ b/coda_lm_eval/evaluation/suggestion_few_shot/high.json @@ -0,0 +1,5 @@ +{ + "reference": "Maintain the current lane and speed, carefully monitor the motorcyclist to the right, and prepare to react if the motorcyclist changes lanes or stops suddenly. Stay alert for the vehicles in the left lane that may enter your lane or affect the flow of traffic.", + "prediction": "The ego car should maintain its current speed and continue straight, while remaining vigilant of the bicycle rider, ensuring a safe distance to prevent accidents. Additionally, the car must be ready to adapt to sudden changes in traffic, such as other vehicles switching lanes.", + "response": "The predictive text only mistakenly substitutes \"motorcyclist\" for \"bicycle rider\" . It correctly suggests maintaining speed and being watchful for lane changes and reflects an understanding of safe driving. So, Rating: [[8]]" +} \ No newline at end of file diff --git a/coda_lm_eval/evaluation/suggestion_few_shot/low.json b/coda_lm_eval/evaluation/suggestion_few_shot/low.json new file mode 100644 index 00000000000..6b1e379215c --- /dev/null +++ b/coda_lm_eval/evaluation/suggestion_few_shot/low.json @@ -0,0 +1,5 @@ +{ + "reference": "The ego car should maintain a safe distance from the white SUV ahead, and prepare to navigate around the roadwork or damaged road surface. It should also observe for any instructions from the traffic lights once they become visible.", + "prediction": "The ego car should maintain a safe distance from the other cars and the truck to avoid collisions. It should be cautious of pedestrians and cyclists, as they might be crossing the street or sharing the road and follow the speed limits and any posted road signs to ensure a safe driving experience for all road users.", + "response": "The prediction text deviates from the reference text by introducing irrelevant elements like pedestrians, cyclists, road signs, and a truck, while omitting key details like roadwork and traffilc light. It always describes general safe driving practices, lacking the specific analysis required for the given scenario. So, Rating: [[2]]" +} \ No newline at end of file diff --git "a/coda_lm_eval/images/CODA-LM - \345\277\253\346\215\267\346\226\271\345\274\217.lnk" "b/coda_lm_eval/images/CODA-LM - \345\277\253\346\215\267\346\226\271\345\274\217.lnk" new file mode 100644 index 00000000000..c8e11652fd6 Binary files /dev/null and "b/coda_lm_eval/images/CODA-LM - \345\277\253\346\215\267\346\226\271\345\274\217.lnk" differ diff --git a/coda_lm_eval/images/overview.png b/coda_lm_eval/images/overview.png new file mode 100644 index 00000000000..76b2b012a02 Binary files /dev/null and b/coda_lm_eval/images/overview.png differ diff --git a/coda_lm_eval/images/visual.png b/coda_lm_eval/images/visual.png new file mode 100644 index 00000000000..4d9fc7ed060 Binary files /dev/null and b/coda_lm_eval/images/visual.png differ diff --git a/coda_lm_eval/labeling/README.md b/coda_lm_eval/labeling/README.md new file mode 100644 index 00000000000..f530dd444a2 --- /dev/null +++ b/coda_lm_eval/labeling/README.md @@ -0,0 +1,12 @@ +# CODA-LM Labeling Tool +## Gradio Web UI +The instructions for launching a Gradio Web UI of Labeling Tool are listed as follows: +1. Install dependencies. +``` +pip install gradio==4.31.4 +``` +2. Launch the Gradio Web UI. +``` +python codalm_labeling_tool.py --data_root $DATA_ROOT/CODA/CODA-LM --version Mini --save_path ./Refine +``` +The Gradio web interface is now up and running. You can use the URL shown on the screen to open the web interface. \ No newline at end of file diff --git a/coda_lm_eval/labeling/codalm_labeling_tool.py b/coda_lm_eval/labeling/codalm_labeling_tool.py new file mode 100644 index 00000000000..cdf2e4a55c4 --- /dev/null +++ b/coda_lm_eval/labeling/codalm_labeling_tool.py @@ -0,0 +1,192 @@ +import gradio as gr +import os +from PIL import Image +import json +import re +import random +import argparse +def remove_extra_commas(json_str): + fixed_json = re.sub(r',\s*([\]}])', r'\1', json_str) + return fixed_json + +def convert_label(cls_select): + if cls_select == "vru": + return "vulnerable_road_users" + elif cls_select == "traffic_signs": + return "traffic signs" + elif cls_select == "traffic_lights": + return "traffic lights" + elif cls_select == "traffic_cones": + return "traffic cones" + elif cls_select == "other_objects": + return "other objects" + else: + return cls_select + +def show(data_root, version, text, cls_select): + assert version in ['Train', 'Val', 'Test', 'Mini'] + if version in ['Train']: + images_type = 'val_' + else: + images_type = 'test_' + data_path = os.path.join(data_root, version) + img_name = images_type + text.zfill(4) + '.jpg' + json_name = images_type + text.zfill(4) + '.json' + img_root, img_id = img_name.replace(".jpg", "").split("_") + new_label = convert_label(cls_select) + img_path = os.path.join(data_root, "..",img_root, "images", img_id + ".jpg") + img = Image.open(img_path) + json_path = os.path.join(data_path, json_name) + custum_sum = 0 + info_list = [] + if os.path.exists(json_path): + with open(json_path, 'r') as f: + parsed_json = json.load(f) + suggestion = parsed_json['driving_suggestion'] + parsed_json = parsed_json['general_perception'] + parsed_json['suggestions'] = suggestion + if new_label in parsed_json.keys(): + info = parsed_json[new_label] + else: + info = parsed_json[cls_select] + if new_label == 'suggestions': + info_list.append(info) + else: + filtered_info = [d for d in info if d] + for idx, item in enumerate(filtered_info): + info_list.append("description: " + item['description'] + "\n" + "explanation: " + item['explanation']) + custum_sum += len(filtered_info) + info_list = info_list + remove_list = [""] * (14 - len(info_list)) + return img, *info_list, *remove_list + +def save_text(save_path, version, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, final_dro, text, cls_select): + d_list = [d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14] + assert version in ['Train', 'Val', 'Test', 'Mini'] + if version in ['Train']: + json_type = 'val_' + else: + json_type = 'test_' + good_text = "" + modify_text = "" + for d in d_list: + if d == "good": + good_text = good_text + '1 ' + else: + good_text = good_text + '0 ' + if d == 'modify': + modify_text = modify_text +'1 ' + else: + modify_text = modify_text +'0 ' + if final_dro == "complete": + complete_info = 1 + else: + complete_info = 0 + complete_text_name = 'complete_' + cls_select +'.txt' + good_txt_name = 'good_' + cls_select + '.txt' + modify_txt_name = 'modify_' + cls_select + '.txt' + refine_save_path = os.path.join(save_path, version, json_type + text.zfill(4)) + os.makedirs(refine_save_path, exist_ok=True) + with open(os.path.join(refine_save_path, good_txt_name), 'w') as fp: + fp.write(good_text) + with open(os.path.join(refine_save_path, modify_txt_name), 'w') as fp: + fp.write(modify_text) + with open(os.path.join(refine_save_path, complete_text_name), 'w') as fp: + fp.write(str(complete_info)) + return "Submit successfully!" + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--data_root", type=str, default="CODA/CODA-LM") + parser.add_argument("--version", type=str, default="Test") + parser.add_argument("--save_path", type=str, default="./Refine") + args = parser.parse_args() + rules_markdown = """ ### Guidelines +- Input the image number (ranging from 1 to 4884) along with the specific category you wish to examine, then press "Display" to reveal both the image and its associated annotation data. +- According to the annotation rules, select "good", "modify", or "delete" for each text box. +- Based on the selected good and modify annotations, assess whether the image completely describes the salient objects of that category, marking it as either "complete" or "incomplete". +- After making your selections, click "Submit". A "Submit successfully!" message will display upon successful submission. +- Click "Clear" to refresh the page and start a new round. +""" + with gr.Blocks() as demo: + data_root = gr.State(value = args.data_root) + version = gr.State(value = args.version) + save_path = gr.State(value = args.save_path) + gr.Markdown(rules_markdown) + with gr.Row():\ + + with gr.Column(scale=20): + text = gr.Textbox( + label="Image name", + placeholder="Input image name: 1" + ) + with gr.Column(scale=5): + cls_select = gr.Dropdown(["vehicles", "vru", "traffic_signs", "traffic_lights", "traffic_cones", "barriers", "other_objects", "suggestions"], label="Class selection") + with gr.Row(): + display_btn = gr.Button(value="Display") + + with gr.Row(): + with gr.Column(scale=1): + img1 = gr.Image() + with gr.Column(): + with gr.Row(): + text1 = gr.Textbox(scale=3, show_label = False) + tie_dro1 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text2 = gr.Textbox(scale=3, show_label = False) + tie_dro2 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text3 = gr.Textbox(scale=3, show_label = False) + tie_dro3 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text4 = gr.Textbox(scale=3, show_label = False) + tie_dro4 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text5 = gr.Textbox(scale=3, show_label = False) + tie_dro5 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text6 = gr.Textbox(scale=3, show_label = False) + tie_dro6 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text7 = gr.Textbox(scale=3, show_label = False) + tie_dro7 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text8 = gr.Textbox(scale=3, show_label = False) + tie_dro8 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text9 = gr.Textbox(scale=3, show_label = False) + tie_dro9 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text10 = gr.Textbox(scale=3, show_label = False) + tie_dro10 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text11 = gr.Textbox(scale=3, show_label = False) + tie_dro11 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text12 = gr.Textbox(scale=3, show_label = False) + tie_dro12 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text13 = gr.Textbox(scale=3, show_label = False) + tie_dro13 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + text14 = gr.Textbox(scale=3, show_label = False) + tie_dro14 = gr.Dropdown(["good", "modify", "delete"], label="Label selection", scale=1) + with gr.Row(): + final_dro = gr.Dropdown(["complete","incomplete"], label="Ann Complete choice", scale=2) + btn = gr.Button("Submit", scale=2) + text_output = gr.Textbox(label="Output", scale=2) + clear_btn = gr.Button(value="🗑️ Clear") + + img_list = [img1] + text_list = [text1, text2, text3, text4, text5, text6, text7, text8, text9, text10, text11, text12, text13, text14] + btn_list = [display_btn, btn, clear_btn] + drop_down_list = [tie_dro1, tie_dro2, tie_dro3, tie_dro4, tie_dro5, tie_dro6, tie_dro7, tie_dro8, tie_dro9, tie_dro10, tie_dro11, tie_dro12, tie_dro13, tie_dro14, final_dro] + display_btn.click(show, inputs=[data_root, version, text, cls_select], outputs=img_list + text_list) + btn.click(save_text, inputs=[save_path, version] + drop_down_list + [text, cls_select], outputs=text_output) + + clear_btn.click(lambda: ["", None, "", None, "", "", "", "", "", "", "", "", "", "", "", "", "", "", + None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, "Display", "Submit", "Clear"], + None, [text, cls_select, text_output] + img_list + text_list + drop_down_list + btn_list) + + demo.launch(share=True) \ No newline at end of file