Summary
elasticapm.contrib.celery.set_celery_headers (connected to celery's before_task_publish signal by register_instrumentation) injects headers["elasticapm"]["trace_parent_string"] into every task published during an active APM transaction. When celery merges this custom header with its own built-in protocol headers (kwargsrepr, origin, etc.) into the AMQP wire format, the resulting header table is corrupted and RabbitMQ's broker cannot decode it — crashes the channel, then the whole connection (amqp.exceptions.InternalError 541 on the client).
Environment
- elastic-apm 6.25.0 (checked 6.26.0/6.26.1/6.26.2 changelogs — no related fix)
- celery 5.6.1, kombu 5.6.2, amqp (py-amqp) 5.3.1
- RabbitMQ broker, AMQPS/TLS, Python 3.13
Evidence
549 broker-side crashes over a 14h production window, 100% on tasks published from an active APM HTTP transaction (i.e. wherever a view calls .delay()), across 9+ different task types. Byte-exact reimplementation of RabbitMQ's parse_table confirms 549/549 fail at the identical byte — an invalid type tag right before the literal "kwargsrepr" header key, whose name-length prefix (should be 0x0A) is corrupted to 0x00. Only reproduces when the "elasticapm" header key from set_celery_headers is present.
Full technical writeup / sample bytes: see companion issue celery/celery#10412 (root mechanism looks like it's on celery's side, in how it merges custom headers= with its own protocol headers — but filing here too since set_celery_headers's use of a nested dict-under-a-custom-key is what triggers celery's merge path).
Ask
Is set_celery_headers's header shape (a dict value under a single custom top-level key) known to interact badly with celery's own header-stamping/merge? Workaround note: #1370 mentions a prior "Celery bug concerning its headers field" with a {"headers": {...}} double-nesting workaround — wondering if that's still needed / whether this is the same underlying issue resurfacing.
Happy to share the full 549-sample dataset and parse simulator.
Summary
elasticapm.contrib.celery.set_celery_headers(connected to celery'sbefore_task_publishsignal byregister_instrumentation) injectsheaders["elasticapm"]["trace_parent_string"]into every task published during an active APM transaction. When celery merges this custom header with its own built-in protocol headers (kwargsrepr, origin, etc.) into the AMQP wire format, the resulting header table is corrupted and RabbitMQ's broker cannot decode it — crashes the channel, then the whole connection (amqp.exceptions.InternalError 541 on the client).Environment
Evidence
549 broker-side crashes over a 14h production window, 100% on tasks published from an active APM HTTP transaction (i.e. wherever a view calls
.delay()), across 9+ different task types. Byte-exact reimplementation of RabbitMQ's parse_table confirms 549/549 fail at the identical byte — an invalid type tag right before the literal "kwargsrepr" header key, whose name-length prefix (should be 0x0A) is corrupted to 0x00. Only reproduces when the "elasticapm" header key fromset_celery_headersis present.Full technical writeup / sample bytes: see companion issue celery/celery#10412 (root mechanism looks like it's on celery's side, in how it merges custom
headers=with its own protocol headers — but filing here too sinceset_celery_headers's use of a nested dict-under-a-custom-key is what triggers celery's merge path).Ask
Is
set_celery_headers's header shape (a dict value under a single custom top-level key) known to interact badly with celery's own header-stamping/merge? Workaround note: #1370 mentions a prior "Celery bug concerning its headers field" with a{"headers": {...}}double-nesting workaround — wondering if that's still needed / whether this is the same underlying issue resurfacing.Happy to share the full 549-sample dataset and parse simulator.