Serialize new-resource embedding fan-out

[feoh]

What are the relevant tickets?

Related: #3623 and #3639

Description (What does it do?)

embed_new_learning_resources runs every 30 minutes and looks back across a wider window for newly created resources. It previously dispatched every Qdrant embedding chunk in that window as one celery.group, immediately materializing the entire fan-out on the shared Redis broker. That creates a short-lived queue-depth burst, prompts KEDA to scale the embeddings worker fleet aggressively, and adds avoidable broker CPU pressure.

This changes the task to use the existing _replace_with_chain helper, dispatching one embedding chunk at a time. The full-catalog and by-ID embedding entrypoints already use this pattern following #3484, where sequential chains were introduced specifically to prevent large embedding workloads from overwhelming Qdrant.

The no-work path now completes directly instead of replacing the task with an empty group.

This is independent of #3639: that PR bounds OpenSearch reindex chords while preserving parallel batches and aggregate results; embedding chunks do not require chord result aggregation, so the existing sequential-chain mechanism is simpler and avoids chord bookkeeping entirely.

How can this be tested?

• Run pytest vector_search/tasks_test.py (44 passed locally).
• The updated test verifies the task creates a chain, does not create a group, and preserves the generated resource IDs.
• A new test verifies an empty lookback window does not dispatch an empty canvas.
• Pre-commit passes for both changed files.

For operational validation, observe the embeddings Celery queue during the next scheduled run: queue depth should no longer jump by the full number of lookback chunks at once, and KEDA should not scale the worker deployment solely for that fan-out burst.

Additional Context

This deliberately trades parallelism for bounded pressure, matching the existing Qdrant protection strategy used by start_embed_resources, embed_learning_resources_by_id, and content-file embedding tasks.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This pull request updates the vector_search scheduled embedding job to avoid a bursty Celery fan-out on the shared Redis broker by switching embed_new_learning_resources from a single celery.group(...) dispatch to the existing sequential _replace_with_chain(...) pattern used elsewhere in vector_search.

Changes:

• Replace celery.group(tasks) + self.replace(...) in embed_new_learning_resources with _replace_with_chain(self, tasks) to serialize chunk dispatch.
• Update the existing unit test to assert a celery.chain is created (and no group is created) while preserving the embedded resource IDs.
• Add a new unit test ensuring the no-work path returns None and does not dispatch an empty canvas (no chain/group/replace calls).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
vector_search/tasks.py	Switch embed_new_learning_resources from group fan-out to sequential chain replacement to reduce broker queue-depth bursts.
vector_search/tasks_test.py	Update and expand tests to validate chain behavior and the no-work early-exit path.