GKE high-concurrency worker-imports (128-256 threads)

[TEJASNARAYANS]

Summary

• Add Helm helpers for queue-aware default worker (celery,audio-metrics) and worker-imports (threads pool, configurable concurrency).
• Default chart values: worker queue list + worker-imports concurrency 32; new unit tests for worker and worker-imports deployments.
• Add examples/gke/values-gke-high-concurrency.yaml overlay (4–8 worker-imports pods × 32 threads = 128–256 Celery threads) and document usage in READMEs.

Test plan

• make deps — Helm dependencies updated successfully
• make lint — chart lint passed (0 failures)
• make unittest — worker + worker-imports suites pass; pre-existing gcs_config_test.yaml failure (unknown path data.config\.yml on ConfigMap)
• helm upgrade --install with layered values on a GKE cluster; confirm worker-imports HPA/replicas and Celery -Q imports / --pool=threads / --concurrency=32

Using the overlay (layer on GCS values)

1. Copy and customize GCS base values: cp examples/gke/values-gcs.yaml my-values.yaml
2. Install or upgrade with both files (later file wins for overlapping keys):

helm upgrade --install efficientai charts/efficientai -n efficientai \
  -f my-values.yaml \
  -f examples/gke/values-gke-high-concurrency.yaml \
  --wait

3. Expect worker-imports replicaCount 4, HPA max 8, concurrency 32 per pod (128 threads steady, up to 256 at max scale). Default worker continues draining celery,audio-metrics only.

[greptile-apps]

Greptile Summary

This PR adds Helm helpers to build the eai worker command dynamically for both the default worker and worker-imports, introduces pool, queues, concurrency, and spreadAcrossNodes knobs, and ships a GKE high-concurrency overlay that runs 4–8 worker-imports pods at 32 threads each (128–256 Celery threads).

• The command-building helpers correctly guard --queues and --pool behind truthiness checks, and the release-scoped topology spread in deployment.yaml is a clean improvement over the previous manual approach.
• NOTES.txt uses default \"celery,audio-metrics\" to display the worker's queue list, but the helper skips --queues entirely when worker.queues is empty — so the post-install message tells operators queue isolation is active when it isn't.
• The gcs_config_test.yaml path-syntax fix and the new worker_imports_deployment_test.yaml suite are both welcome additions.

Confidence Score: 3/5

Safe to merge once the NOTES.txt queue-drain message is corrected; the functional chart logic works, but post-install output currently misleads operators into thinking queue isolation is already enforced when it isn't by default.

The NOTES.txt block prints 'Default worker drains: celery,audio-metrics' even when worker.queues is blank, but the actual command helper emits no --queues flag in that case, meaning the worker drains every queue. An operator installing with default values would read the notes, believe the queues are split, and not set efficientai.worker.queues, leaving both workers competing on the imports/diarization/evaluations queues.

charts/efficientai/templates/NOTES.txt — the worker queue drain message needs a conditional instead of the default fallback to match what the command helper actually emits.

Important Files Changed

Filename	Overview
charts/efficientai/templates/NOTES.txt	New informational block uses default "celery,audio-metrics" for worker queues, outputting a factually wrong queue list when worker.queues is empty — the actual command has no --queues flag in that case
charts/efficientai/templates/_helpers.tpl	Adds two new command-building helpers for worker and worker-imports; default 8 $wi.concurrency silently coerces concurrency: 0 to 8, blocking Celery's CPU auto-detect
charts/efficientai/templates/worker-imports/deployment.yaml	Replaces inline command construction with the new helper; adds optional release-scoped topology spread via spreadAcrossNodes; logic is clean
charts/efficientai/templates/worker/deployment.yaml	Single-line change delegating command construction to the new helper; no issues
charts/efficientai/values.yaml	Adds worker.queues, worker.concurrency, workerImports.pool, and workerImports.spreadAcrossNodes; defaults are sane and workerImports.concurrency remains 8
examples/gke/values-gke-high-concurrency.yaml	New GKE overlay for 4–8 worker-imports pods at 32 threads; correctly enables spreadAcrossNodes and HPA; values look self-consistent
charts/efficientai/tests/worker_imports_deployment_test.yaml	New suite covering eai-worker command, custom pool/concurrency, spreadAcrossNodes topology spread, and custom command override; comprehensive
charts/efficientai/tests/worker_deployment_test.yaml	Tests updated to match new helper-based command; covers no-queues (disabled) and with-queues (enabled) paths
charts/efficientai/tests/gcs_config_test.yaml	Fixes pre-existing path syntax to data["config.yml"] — correct fix for the known helm-unittest assertion failure
charts/efficientai/README.md	Adds worker-only and updated worker-imports tables; documents minimum image version (>= 1.5.0) requirement

_{Reviews (2): Last reviewed commit: "Address PR review feedback and fix chart..." | Re-trigger Greptile}