Batch catalog reindex to avoid a single catalog-wide Celery chord

[feoh]

Description/Context

start_recreate_index and start_update_index build one enormous celery.group — a subtask per 100-item chunk (OPENSEARCH_INDEXING_CHUNK_SIZE, default 100) of every content file and learning resource in the catalog — then wrap it in a chain, which Celery executes as a single catalog-wide chord:

• mit-learn/learning_resources_search/tasks.py

  Lines 692 to 700 in 2c6caa6

  	.values_list("id", flat=True),
  	chunk_size=settings.OPENSEARCH_INDEXING_CHUNK_SIZE,
  	)
  	]
  	index_tasks = celery.group(index_tasks)
  	except: # noqa: E722
  	error = "start_recreate_index threw an error"
  	log.exception(error)

  (start_recreate_index: celery.group(index_tasks) then self.replace(celery.chain(index_tasks, finish_recreate_index.s(...))))

• mit-learn/learning_resources_search/tasks.py

  Lines 746 to 757 in 2c6caa6

  	if resource_type in indexes:
  	index_tasks = index_tasks + get_update_learning_resource_tasks(
  	resource_type
  	)
  	index_tasks = celery.group(index_tasks)
  	except: # noqa: E722
  	error = "start_update_index threw an error"
  	log.exception(error)
  	return [error]
  	return self.replace(celery.chain(index_tasks, finish_update_index.s()))

  (start_update_index: same pattern)

• mit-learn/main/settings.py

  Lines 584 to 588 in 2c6caa6

  	# Similar resources settings
  	MITOL_SIMILAR_RESOURCES_COUNT = get_int("MITOL_SIMILAR_RESOURCES_COUNT", 3)
  	OPEN_RESOURCES_MIN_DOC_FREQ = get_int("OPEN_RESOURCES_MIN_DOC_FREQ", 1)
  	OPEN_RESOURCES_MIN_TERM_FREQ = get_int("OPEN_RESOURCES_MIN_TERM_FREQ", 1)

  (chunk size default 100)

Celery's Redis result backend synchronizes a chord by having every subtask RPUSH its result into a per-group list and poll LLEN against the expected count. With thousands of subtasks over the full catalog, this produced a ~140x spike in Redis list commands (80k/hr → 11.4M/hr) and drove mitlearn-redis-production memory from ~60% to 99.98% in one hour on 2026-07-15. These reindexes are launched from the recreate_index / update_index management commands (manual / deploy-time), so the saturation recurs and pages on-call each time. Full data in the root-cause issue.

Note: finish_recreate_index(results, backing_indices) genuinely consumes the subtask results (merge_strings(results)), so the results cannot simply be ignored —

mit-learn/learning_resources_search/tasks.py

Lines 924 to 953 in 2c6caa6

	reject_on_worker_lost=True,
	autoretry_for=(RetryError, SystemExit),
	retry_backoff=True,
	rate_limit=settings.CELERY_SEARCH_RATE_LIMIT,
	)
	def finish_recreate_index(results, backing_indices):
	"""
	Swap reindex backing index with default backing index
	Args:
	results (list or bool): Results saying whether the error exists
	backing_indices (dict): The backing OpenSearch indices keyed by object type
	"""
	errors = merge_strings(results)
	if errors:
	try:
	api.delete_orphaned_indexes(
	list(backing_indices.keys()), delete_reindexing_tags=True
	)
	except RequestError as ex:
	raise RetryError(str(ex)) from ex
	msg = f"Errors occurred during recreate_index: {errors}"
	raise ReindexError(msg)
	log.info(
	"Done with temporary index. Pointing default aliases to newly created backing indexes..." # noqa: E501
	)
	for obj_type, backing_index in backing_indices.items():
	try:
	api.switch_indices(backing_index, obj_type)

Plan/Design

Bound the in-flight footprint of a reindex so a single run cannot saturate the shared Redis:

• Split the catalog-wide chord into sequential batches (e.g. chord-per-index or chord-per-N-chunks), so only one batch's results occupy Redis at a time.
• Consider driving the fan-out from a producer that throttles how many subtasks are in flight, rather than materializing the entire group up front.
• Combine with moving the result backend off the broker Redis (see companion issue) for defense in depth.

Preserve current behavior: error aggregation via merge_strings(results) and the final alias swap in finish_recreate_index must still run once all batches complete.