Currently, the map component is supplied with location data only through analyzing the video metadata.
Running some form of named entity recognition on the transcript of the video audio, finding geographical locations, along with checking with a geocoding API, could get us other location point to show: "locations mentioned in the video"
This should be a separate job, running after completion of the transcript job. See:
|
# TODO: handle dependent jobs |
|
match message_type: |
|
case "transcription": |
|
pass # TODO: rerun map task |
Extracting the locations from the transcript could be as easy as running it through ChatGPT, asking for locations. Running it on the English translation might be more accurate than the original language, due to Whisper having a relatively smaller vocabulary than GPT, and I've anecdotally seen the translation correct location names.
For geocoding (transforming location names to coordinates), the Google Maps API might be a nice option: https://developers.google.com/maps/documentation/geocoding/overview
OpenStreetMap might have some tooling as well.
Finally, the MapResponse type should be extended to reflect the difference between metadata location and transcript location(s):
|
export interface MapResponse extends BackendMessage { |
|
latlng: number[] | null; |
|
} |
The backend should then send data formatted as expected by the frontend. Note that the cache might need to be queried to resend the metadata location as part of this message.
This should then be appropriately visualized with distinct colors in the frontend map. Note that there's an issue #2 for changing this map to Google Maps, so UI changes should not be done in the Leaflet component (double work)
Currently, the map component is supplied with location data only through analyzing the video metadata.
Running some form of named entity recognition on the transcript of the video audio, finding geographical locations, along with checking with a geocoding API, could get us other location point to show: "locations mentioned in the video"
This should be a separate job, running after completion of the transcript job. See:
veridash/backend/veridash_backend/webserver/app.py
Lines 91 to 94 in daec03a
Extracting the locations from the transcript could be as easy as running it through ChatGPT, asking for locations. Running it on the English translation might be more accurate than the original language, due to Whisper having a relatively smaller vocabulary than GPT, and I've anecdotally seen the translation correct location names.
For geocoding (transforming location names to coordinates), the Google Maps API might be a nice option: https://developers.google.com/maps/documentation/geocoding/overview
OpenStreetMap might have some tooling as well.
Finally, the MapResponse type should be extended to reflect the difference between metadata location and transcript location(s):
veridash/frontend/src/useBackend.ts
Lines 55 to 57 in daec03a
The backend should then send data formatted as expected by the frontend. Note that the cache might need to be queried to resend the metadata location as part of this message.
This should then be appropriately visualized with distinct colors in the frontend map. Note that there's an issue #2 for changing this map to Google Maps, so UI changes should not be done in the Leaflet component (double work)