TimeClipAI: Real-Time Action Classification and Time Segmentation in Full-Length Videos Using Anchor Transformers for Online Temporal Action Localization

Empowering full-length video analysis with Anchor Transformers for Online Temporal Action Localization

TimeClipAI is a cutting-edge framework for real-time action classification and temporal segmentation in videos. Leveraging the power of Anchor Transformers, it provides state-of-the-art performance for online temporal action localization. This repository contains everything you need to train and test the model and its post-processing network (OSN).

---

🌟 Features

• Real-Time Action Classification: Accurately classify actions in streaming or full-length videos.
• Temporal Segmentation: Precise time segmentation for action localization.
• Anchor Transformers: Novel architecture for efficient and robust processing.
• Support for Multiple Datasets: Compatible with EGTEA, EPIC-Kitchen 100, THUMOS'14 datasets.
• Pre-trained Features: Ready-to-use I3D features for seamless training and testing.

📐 Model Architecture

---

📋 Table of Contents

• Installation
• Training
• Testing
• Datasets and Features
• Configuration
• Contributing
• License

---

🛠 Installation

Prerequisites

Ensure your system meets the following requirements:

• OS: Ubuntu 20.04
• Python: 3.10.9
• CUDA: 12.0

Dependencies

Install the required Python libraries using the provided requirements.txt file:

pip install -r requirements.txt

Key dependencies include:

• pytorch==2.0.0
• numpy==1.23.5
• h5py==3.9.0

See requirements.txt for the full list.

---

🚀 Training

Datasets and Features

Download pre-trained Kinetics I3D features for the EGTEA dataset from this Google Drive link. Place the files in the data/ directory.

Other supported datasets:

• EPIC-Kitchen 100
• THUMOS'14

⚙️ Configuration

Configuration files for all datasets are included in the repository.

Training Our Model

Train Our Model with the following command:

python main.py --mode=train --split=<split_number> --batch_size=256 --epoch=5

Replace <split_number> with the dataset split (e.g., EGTEA has 4 splits).

Training the Post-Processing Network (OSN)

To train the OSN (Offset Scoring Network), run:

python supnet.py --mode=make --inference_subset=train --split=<split_number>
python supnet.py --mode=make --inference_subset=test --split=<split_number>
python supnet.py --mode=train --split=<split_number>

---

🧪 Testing

Test Our Model using the following command:

python main.py --mode=test --split=<split_number> --batch_size=256 --epoch=5

Replace <split_number> with the appropriate dataset split.

📈 Sample Results

---

📊 Datasets and Features

TimeClipAI supports multiple datasets for action localization tasks. Below are the links to download the necessary features:

Dataset Name	Source
EGTEA	Google Drive
EPIC-Kitchens 100	Google Drive
THUMOS'14	Google Drive
CricShot10	GitHub

Place all downloaded feature files in the data/ directory.

---

🤝 Contributing

We welcome contributions to improve TimeClipAI! To contribute:

1. Fork the repository.

2. Create a new branch:

   git checkout -b feature-branch

3. Make your changes and commit:

   git commit -m "Add new feature"

4. Push to your branch:

   git push origin feature-branch

5. Open a Pull Request.

Please ensure your code follows the project's coding standards and includes appropriate documentation.

---

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Built with ❤️ by the TimeClipAI team. Start analyzing videos in real-time today!