feat: updating concurrency limits

[TEJASNARAYANS]

What Changed?

Briefly describe what this PR changes.

Why?

Explain the problem this solves and why this approach was chosen.

How to Test?

List clear steps for reviewers to verify the change.

Release Label

Select one semantic version bump intent for this PR:

• major - breaking change, next release bumps major version
• minor - backward-compatible feature, next release bumps minor version
• fix - backward-compatible bug fix, next release bumps patch version
• No label (defaults to patch release)

If you do not have permission to apply labels, mention the intended release label here and a maintainer will set it.

Checklist

• I have read the CONTRIBUTING.md guide.
• My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• I have updated documentation where needed.

[greptile-apps]

Greptile Summary

This PR replaces the direct Celery fan-out for evaluation tasks with a Redis-backed fair-share dispatch system. Instead of enqueuing thousands of row tasks immediately, a single dispatch_fair_eval_rows_task round-robins across workspaces (up to batch_size rows per workspace turn) and enforces per-workspace, per-org, and global inflight limits via Lua-atomic Redis counters.

• New concurrency system (app/workers/concurrency/): limits.py manages Redis slot acquire/release with Lua scripts; eval_dispatch.py handles single-row dispatch (transcribe or evaluate); fair_dispatch.py implements the workspace round-robin loop and schedules follow-up dispatcher turns when tasks complete.
• Queue restructuring: transcribe_call_import_row moves to a diarization queue, eval tasks to an evaluations queue, and the worker now drains imports → diarization → evaluations with concurrency raised from 4 to 8.
• Frontend changes: Workspace-scoped query keys (adds activeWorkspaceId to useQuery/invalidateQueries calls) and a new workspaceNavigation.ts helper that redirects from detail pages to their parent list when switching workspaces.

Confidence Score: 3/5

Not safe to merge without addressing the infinite re-transcription loop in fair_dispatch.py.

The new round-robin dispatcher stores a transcribe_overwrite flag in Redis but never clears it after the initial transcription finishes. Every subsequent dispatcher turn reads the flag as still active and re-transcribes the same audio, repeating until the 20-minute TTL expires. In practice this means every audio row in a force-overwrite evaluation is transcribed ~10+ times before evaluation can begin. The rest of the system — slot acquire/release, the dispatched/skip/at_capacity loop logic, queue routing, and the frontend query-key scoping — all look correct.

app/workers/concurrency/fair_dispatch.py needs a clear_evaluation_transcribe_overwrite helper and a call to it inside _apply_eval_chain_transcribe_cleanup (or on successful dispatch in _dispatch_batch_for_workspace). app/workers/tasks/transcribe_call_import_row.py is the other half of this fix.

Important Files Changed

Filename	Overview
app/workers/concurrency/fair_dispatch.py	New round-robin workspace dispatcher — has a critical issue where store_evaluation_transcribe_overwrite is never cleared after transcription completes, causing repeated re-transcription until the 20-minute TTL expires.
app/workers/concurrency/eval_dispatch.py	New per-row dispatch helpers — slot acquire/release is correctly guarded; _try_dispatch_single_row returns distinct dispatched/skip/at_capacity values that fix the previous starvation bug.
app/workers/concurrency/limits.py	New Redis Lua-backed inflight-slot manager — acquire and release scripts are atomic and correctly guard against underflow.
app/workers/tasks/transcribe_call_import_row.py	Removes direct eval-chain enqueue and adds _apply_eval_chain_transcribe_cleanup in finally; cleanup correctly clears celery_task_id but does not clear the transcribe_overwrite flag, leading to repeated re-transcription.
app/workers/tasks/evaluate_call_import_row.py	Adds _eval_slot_task_id param and calls finish_eval_work_and_redispatch in finally; slot release is correctly handled for both new-style and legacy tasks.
app/api/v1/routes/call_import_evaluations.py	Replaces direct fan-out with a call to _enqueue_eval_rows_with_optional_transcribe + schedule_fair_dispatch; the refactoring is clean and removes the previous N-dispatcher-tasks-per-N-evaluations bug.
app/workers/config.py	Moves eval/transcribe tasks to dedicated evaluations/diarization queues and updates task routes; routing is consistent with explicit queue overrides in apply_async calls.

Comments Outside Diff (1)

1. app/workers/concurrency/fair_dispatch.py, line 807-812 (link)

   transcribe_overwrite key never cleared — causes infinite re-transcription loop

   store_evaluation_transcribe_overwrite writes the key with a 20-minute TTL, but nothing ever explicitly deletes it. After the first transcription completes, _apply_eval_chain_transcribe_cleanup clears eval_row.celery_task_id = None and returns the row to the pending pool. On the very next dispatcher turn evaluation_transcribe_overwrite(evaluation.id) still returns True, so _needs_transcribe_for_eval(..., transcribe_overwrite=True) returns True again (because not existing_dia or transcribe_overwrite is always True when the flag is set). A new transcription slot is acquired and the row is re-transcribed — this repeats every ~<transcription time> minutes until the 20-minute TTL expires, meaning each audio row can be transcribed ~10+ times before evaluation ever starts. Adding a clear_evaluation_transcribe_overwrite(evaluation_id) call inside _apply_eval_chain_transcribe_cleanup (or immediately after a successful dispatch in _dispatch_batch_for_workspace) would stop the loop.

_{Reviews (5): Last reviewed commit: "fix: some misses" | Re-trigger Greptile}