[improve][pip] PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data - #24716
Open
codelipenghui wants to merge 2 commits into
Open
[improve][pip] PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data#24716codelipenghui wants to merge 2 commits into
codelipenghui wants to merge 2 commits into
Conversation
…verable Data This PIP proposes adding two broker-level metrics to provide visibility into non-recoverable data skipping when autoSkipNonRecoverableData is enabled: - pulsar_broker_non_recoverable_ledgers_skipped_total: Count of ledgers skipped - pulsar_broker_non_recoverable_entries_skipped_total: Count of entries skipped The metrics enable operators to: - Set up alerts when data loss occurs - Monitor data durability SLAs - Distinguish between systematic (ledger-level) vs localized (entry-level) issues - Use broker logs for detailed investigation of specific topics affected Implementation adds counters to BrokerOperabilityMetrics with integration points in ManagedLedgerImpl.skipNonRecoverableLedger() and ManagedCursorImpl.skipNonRecoverableEntries(). 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
github-actions
Bot
added
doc-not-needed
codelipenghui
added a commit
to codelipenghui/incubator-pulsar
that referenced
this pull request
Sep 10, 2025
This change implements GitHub PR apache#24716 to add operational visibility for non-recoverable data loss events in Apache Pulsar brokers. Key Changes: - Add NonRecoverableDataMetricsCallback interface in managed-ledger module - Integrate callback in ManagedLedgerImpl.skipNonRecoverableLedger() - Integrate callback in ManagedCursorImpl.skipNonRecoverableEntries() - Configure callback in BrokerService during managed ledger creation - Add two new metrics in BrokerOperabilityMetrics: - pulsar.broker.non.recoverable.ledgers.skipped.count - pulsar.broker.non.recoverable.entries.skipped.count - Support both Prometheus and OpenTelemetry metrics - Comprehensive test coverage including end-to-end integration tests The metrics are always available regardless of ManagedLedgerInterceptor configuration, providing reliable operational monitoring for data loss scenarios. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
9 tasks
lhotari
reviewed
Sep 10, 2025
|
|
||
| ## Out of Scope | ||
|
|
||
| - Topic/subscription-level metrics (would burden metrics system with high cardinality) |
Member
There was a problem hiding this comment.
How about in-memory topic & subscription stats?
Contributor
Author
There was a problem hiding this comment.
Yes, that makes sense.
Denovo1998
reviewed
Sep 16, 2025
| - **Alerting**: Get notified when data loss occurs | ||
| - **SLA Monitoring**: Track data durability metrics | ||
| - **Root Cause Analysis**: Compare metrics to understand if issues are systematic (ledger-level) or localized (entry-level) | ||
| - **Investigation**: Use metrics for alerting, then check broker logs for specific topic details |
Contributor
There was a problem hiding this comment.
Broker logs can pinpoint which specific topic is having an issue.
Is this log currently recorded in the code? If so, where is it located?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PIP proposes adding two broker-level metrics to provide essential visibility into non-recoverable data skipping when
autoSkipNonRecoverableDatais enabled.New Metrics
pulsar_broker_non_recoverable_ledgers_skipped_total- Count of entire ledgers skippedpulsar_broker_non_recoverable_entries_skipped_total- Count of individual entries skippedMotivation
Currently, there is no visibility when Pulsar skips non-recoverable data during disaster recovery scenarios. This creates operational blind spots where:
Implementation
BrokerOperabilityMetricsclassManagedLedgerImpl.skipNonRecoverableLedger()→ ledger metricManagedCursorImpl.skipNonRecoverableEntries()→ entry metricOperational Benefits
Design Rationale
Documentation
docdoc-requireddoc-not-neededdoc-complete