Skip to content

[Enhancement]: labeled filesystem metrics with latency, errors, saturation #591

Description

@tedxu

The filesystem metrics layer (FilesystemMetrics) exposes only flat
monotonic counters: read/write counts and bytes, per-op counts, a
single global failed count, and multipart created/finished. This is
enough to answer how much happened, but not whether the object store
is healthy, slow, or degrading.

Gaps for general-purpose monitoring:

  • No latency. No operation duration is measured, so p50/p95/p99 per
    op cannot be computed. This is the primary object-store SLI.
  • No error dimensionality. Failures collapse into one counter, so
    there is no per-op error rate and no cause (throttling, timeout,
    auth, network, not-found).
  • No distributions. read/write bytes are running sums, so per-request
    size distribution is unrecoverable.
  • No saturation. No point-in-time gauges (in-flight requests,
    connections) to reveal bottlenecks.
  • Retries are invisible.

Proposal: replace the flat counters with a labeled registry keyed by
operation type and status, adding per-op latency and payload-size
histograms, error classification, retry counts, and saturation
gauges. Export a fixed-size snapshot with sum, count, and static
bucket bounds over the C FFI so consumers can rebuild histograms.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions