Running the extracted keyframes from the video through object detection and OCR can provide orienting information about the situation for the fact checkers.
We've got some reference implementations for these tasks, for example this notebook provided by @sohailahmedkhan: https://files.skivdal.no/api/public/dl/CH0qTPen/sohail-object-detection-and-ocr.ipynb
There should also be examples of this in the Lifelog system.
Translations for the text extracted with OCR is also needed, and can probably be done by the veridash_backend.worker.translation.Translation class with minimal modifications.
There's an existing, empty panel for this in the frontend, and the messages are handled and should be matched against here:
|
# might be cached |
|
match data["messageType"]: |
|
case "metadata": |
|
task_id: str = get_metadata.apply_async((data["videoId"], )).id # ignore the error on this line |
|
case "transcription": |
|
task_id: str = get_transcription.apply_async((data["videoId"], )).id # ignore the error on this line |
|
case "map": |
|
task_id: str = get_coordinates.apply_async((data["videoId"], )).id # ignore the error on this line |
|
case "keyframes": |
|
task_id: str = get_keyframes.apply_async((data["videoId"], )).id # ignore the error on this line |
|
case _: |
|
raise NotImplementedError(f"The messageType {data['messageType']} is not yet implemented") |
Note that object detection is dependent first on keyframe extraction, so likely, the actual job would be executed from the callback:
|
case "keyframes": |
|
pass # TODO: rerun object detection |
Then, the actual object detection job should be a task in the Celery application: https://github.com/skivdal/veridash/blob/main/backend/veridash_backend/worker/app.py
Keyframes need not be created, they can be collected from the database as results of running the keyframes task. Thus, this task is dependent on keyframes, and should not run if there are no keyframes present.
How this information is best displayed in the frontend would require some involved consideration.
Running the extracted keyframes from the video through object detection and OCR can provide orienting information about the situation for the fact checkers.
We've got some reference implementations for these tasks, for example this notebook provided by @sohailahmedkhan: https://files.skivdal.no/api/public/dl/CH0qTPen/sohail-object-detection-and-ocr.ipynb
There should also be examples of this in the Lifelog system.
Translations for the text extracted with OCR is also needed, and can probably be done by the
veridash_backend.worker.translation.Translationclass with minimal modifications.There's an existing, empty panel for this in the frontend, and the messages are handled and should be matched against here:
veridash/backend/veridash_backend/webserver/actions.py
Lines 53 to 64 in daec03a
Note that object detection is dependent first on keyframe extraction, so likely, the actual job would be executed from the callback:
veridash/backend/veridash_backend/webserver/app.py
Lines 95 to 96 in daec03a
Then, the actual object detection job should be a task in the Celery application: https://github.com/skivdal/veridash/blob/main/backend/veridash_backend/worker/app.py
Keyframes need not be created, they can be collected from the database as results of running the keyframes task. Thus, this task is dependent on keyframes, and should not run if there are no keyframes present.
How this information is best displayed in the frontend would require some involved consideration.