This project needs some reworking, primarily a ground up re-write of how tasks are started and managed. As an example, right now it is nearly architecturally impossible to add the ability to rename an output.
Tests were started to start to isolate different parts of the code in py-thread-test and the first implementation of a true multi-threaded web application was used for video-manager. These use a central SQL database for storing and retrieving data between threads.
Core Changes
- Eliminate the usage of multiprocessing Process where possible in favor of async tasks
- Use Quart for async web operations, templates to provide better web API to data and more maintainable interfaces
- Use SQLite DB to abstract data from code space. No media sample data should be stored in memory long term
- Create async action queue that will store tasks to perform. Can be used to limit active tasks
Database
Database should be wiped between launches. Media sample data should still be stored in JSON files within the project. This allows importing previous backups for further processing. Actions should be restartable on a media sample by simply being able to re-add it to the queue based on past data. A "completed" or not flag could be used to automatically restart any incomplete tasks.
Additional Changes
- Redo CSV support. Instead of CSVs being manged as a linear queue, just add each line as a grouped media sample. This will remove a lot of largely unused logic. Treat each CSV entry the same as the web API, each element ID can be a header in the CSV.
- Post processing, media samples should be selectable from web front end and have post processing tasks available, these would include renaming, compressing, trimming (removing decoded data used to validate backups), etc
This project needs some reworking, primarily a ground up re-write of how tasks are started and managed. As an example, right now it is nearly architecturally impossible to add the ability to rename an output.
Tests were started to start to isolate different parts of the code in py-thread-test and the first implementation of a true multi-threaded web application was used for video-manager. These use a central SQL database for storing and retrieving data between threads.
Core Changes
Database
Database should be wiped between launches. Media sample data should still be stored in JSON files within the project. This allows importing previous backups for further processing. Actions should be restartable on a media sample by simply being able to re-add it to the queue based on past data. A "completed" or not flag could be used to automatically restart any incomplete tasks.
Additional Changes