Skip to content

chore: add more logs and metrics - #2871

Draft
alok-ssv wants to merge 1 commit into
attack-sim-observability-cleanfrom
attack-sim-observability-clean.2
Draft

chore: add more logs and metrics#2871
alok-ssv wants to merge 1 commit into
attack-sim-observability-cleanfrom
attack-sim-observability-clean.2

Conversation

@alok-ssv

@alok-ssv alok-ssv commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Expand observability for pubsub validation, highlighted-peer tracing, connection handshakes, and stream operations so failures and latency can be attributed to the right stage, topic, and peer.

What Changed

  • Added validation-stage tagging to SSV validation errors so failures can be traced to the exact phase that failed.
  • Extended SSV validation observer events with:
    • validation stage
    • pubsub topic
    • payload size
  • Added pubsub validation metrics for:
    • messages received
    • messages validated by result
    • validation duration
    • handler errors after validation succeeds
  • Added highlighted-peer pubsub observability for:
    • reject events
    • drop events
  • Updated the pubsub tracer so batched DROP_RPC events emit one drop datapoint per real topic instead of collapsing into a synthetic multiple bucket.
  • Added p2p connection handshake observability with:
    • handshake outcome
    • handshake reason
    • handshake duration
  • Added stream error metrics with operation and reason labels.
  • Extended tests across validation, connections, streams, topics, and highlighted-peer tracing to cover the new metrics and event fields.

Why

The previous observability was too coarse to explain where failures happened or which traffic was affected. This change makes dashboards, alerts, and incident triage more accurate by distinguishing:

  • validation stage failures versus successful validation followed by handler failure
  • topic-specific pubsub behavior
  • real connection handshake outcomes versus ignored or duplicate connection paths
  • actual drop events versus normal RPC delivery paths
  • per-topic behavior for batched pubsub drops

Validation

  • go test ./network/topics -run TestPsTracerLogDropRPCRecordsEachTopic -count=1
  • go test ./network/topics

@alok-ssv
alok-ssv requested a review from julienharbulot June 3, 2026 09:40
@codecov

codecov Bot commented Jun 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.85714% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.1%. Comparing base (18f6006) to head (11b9f12).
⚠️ Report is 20 commits behind head on attack-sim-observability-clean.

Files with missing lines Patch % Lines
network/topics/tracer.go 18.1% 16 Missing and 2 partials ⚠️
message/validation/errors.go 52.1% 9 Missing and 2 partials ⚠️
network/peers/connections/conn_handler.go 80.0% 7 Missing ⚠️
network/streams/controller.go 56.2% 7 Missing ⚠️
network/topics/controller.go 44.4% 5 Missing ⚠️
message/validation/partial_validation.go 42.8% 4 Missing ⚠️
message/validation/validation.go 71.4% 4 Missing ⚠️
network/peers/peertrace/observer.go 84.0% 2 Missing and 2 partials ⚠️
message/validation/consensus_validation.go 66.6% 3 Missing ⚠️
message/validation/stage.go 86.6% 1 Missing and 1 partial ⚠️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 30 days if there are no updates. Please comment if you would like to keep it open.

@github-actions github-actions Bot added stale and removed stale labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant