Skip to content

feat(observability): add Pyroscope continuous profiling #46

Description

@hasansezertasan

Goal

Add Pyroscope continuous profiling to Croupier so we have CPU and wall-time flamegraphs for the receipt-printing subscriber.

Motivation

Croupier sits on the hot path between RabbitMQ and the in-store thermal printer. Latency tails (slow python-escpos socket writes, GC pauses, blocking calls inside the async handler) are not visible in Sentry — Sentry only sees raised exceptions. Continuous profiling lets us answer questions like:

  • Where is wall-time going during a printer outage retry storm?
  • Is aio-pika decoding dominating CPU when the queue backlog grows?
  • Are there hidden sync calls inside the async path?

Why Pyroscope specifically

  • Push-based (outbound HTTPS), so it works through NAT on branch-host deployments where pull-based observability is structurally incompatible.
  • Already supported by lite-bootstrap via a single pyroscope_endpoint field on FastStreamConfig, so the wiring is one extra field plus the corresponding Settings entry.
  • Auth-token support via pyroscope_additional_params covers the case where the collector lives outside the trusted edge network.

Proposed scope

  • pyproject.toml: add pyroscope to the lite-bootstrap[...] extras.
  • uv.lock: regenerate (pulls pyroscope-io plus transitive cffi / pycparser).
  • src/croupier/main.py: re-add the PyroscopeInstrument whitelist entry on _Bootstrapper.instruments_types.
  • Settings (src/croupier/main.py): add pyroscope_endpoint: HttpUrl | None = None, plus optional pyroscope_tags: dict[str, str] = {} and pyroscope_additional_params: dict[str, Any] = {}, all read from ~/.croupier.json.
  • create_app(): forward those fields to FastStreamConfig. Tag profiles with queue_name by default (same fleet-unique key already used for the Sentry tag) so multi-branch deployments stay separable in the Pyroscope UI.
  • README + CLAUDE.md: short opt-in section describing the config keys, edge-friendly push direction, and how to point the collector at a per-branch deployment.
  • Tests: cover the opt-in path (endpoint set → instrument registered) and the inert path (endpoint unset → instrument not registered, no outbound traffic on startup).

Open questions

  • Where does the Pyroscope collector live? In-cluster (local network), shared central instance, or per-branch?
  • Auth token storage: same ~/.croupier.json file as the Sentry DSN, or a separate secret?
  • Default sample_rate — 100 Hz is the lite-bootstrap default; verify this is acceptable on the lower-spec branch hosts before merging.
  • Profile retention budget — agree on Pyroscope tenancy / retention with whoever owns the collector before turning it on fleet-wide.

Out of scope

  • OpenTelemetry trace-profile linking (Croupier intentionally does not run OpenTelemetry; profiles will be standalone, no span correlation).
  • Cross-service profile aggregation (Croupier is a leaf service).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions