Ab/refactor youtube job - #3739
Conversation
OpenAPI ChangesNo changes detected Unexpected changes? Ensure your branch is up-to-date with |
There was a problem hiding this comment.
Pull request overview
Refactors YouTube ETL into asynchronous channel and playlist tasks to reduce per-worker workload and improve recovery.
Changes:
- Splits YouTube extraction and loading by channel and playlist.
- Adds removal/orphan unpublishing helpers.
- Updates task tests and management-command messaging.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
learning_resources/tasks.py |
Adds asynchronous YouTube task fan-out. |
learning_resources/tasks_test.py |
Tests new task orchestration. |
learning_resources/management/commands/backpopulate_youtube_data.py |
Reports queued channels. |
learning_resources/etl/youtube.py |
Splits channel and playlist extraction/transformation. |
learning_resources/etl/youtube_test.py |
Updates extraction tests. |
learning_resources/etl/pipelines.py |
Removes the synchronous YouTube pipeline. |
learning_resources/etl/loaders.py |
Adds granular loading and unpublishing helpers. |
learning_resources/etl/loaders_test.py |
Tests new unpublishing behavior. |
| # get_youtube_data is absent on purpose: it only queues the fan-out, whose | ||
| # writes land long after it returns, so it has nothing to invalidate |
There was a problem hiding this comment.
I think this is ok - before the refactor clear_views_cache ran before the search updates finished anyway. And the cache is renewed by itself eventually
| video_channel = loaders.upsert_video_channel( | ||
| youtube.transform_channel(channel_data) | ||
| ) | ||
| loaders.unpublish_removed_playlists( | ||
| video_channel, [playlist_data["id"] for playlist_data, _ in playlists] |
There was a problem hiding this comment.
That's true but fixing it would complicate the code more than it's worth. I think this situation is pretty unlikely to happen and even if it did it would be fixed in the next run
What are the relevant tickets?
Description (What does it do?)
Screenshots (if appropriate):
How can this be tested?
Additional Context