Skip to content

Add System.Diagnostics.Metrics for per-connection Zebus monitoring via net10.0 multi-targeting#120

Open
Kuinox with Copilot wants to merge 8 commits into
masterfrom
copilot/add-metrics-for-zebus-connections
Open

Add System.Diagnostics.Metrics for per-connection Zebus monitoring via net10.0 multi-targeting#120
Kuinox with Copilot wants to merge 8 commits into
masterfrom
copilot/add-metrics-for-zebus-connections

Conversation

Copilot AI commented Apr 24, 2026

Copy link
Copy Markdown

Adds System.Diagnostics.Metrics instrumentation to monitor Zebus connection state and messaging, gated behind #if NET10_0_OR_GREATER with no new NuGet dependencies.

Metrics

All per-connection metrics are tagged with zebus.peer.id for individual peer filtering:

Metric Type Description
zebus.transport.messages.sent Counter (per-peer) Messages sent
zebus.transport.messages.received Counter (per-peer) Messages received
zebus.transport.bytes.sent Counter (per-peer) Bytes sent
zebus.transport.bytes.received Counter (per-peer) Bytes received
zebus.transport.messages.send_failures Counter (per-peer) Send failures
zebus.transport.peer_connections Counter (per-peer) Socket connects
zebus.transport.peer_disconnections Counter (per-peer) Socket disconnects
zebus.transport.peer_connection_failures Counter (per-peer) Connect failures
zebus.transport.connection.alive ObservableGauge (per-peer) 1=alive, 0=dead
zebus.transport.outbound_sockets UpDownCounter Active socket count
zebus.directory.known_peers UpDownCounter Known peer count
zebus.directory.peer_updates Counter Peer update events
// Meter name: "Abc.Zebus"
// Example: filter messages sent to a specific peer
// dotnet-counters monitor --counters "Abc.Zebus[zebus.peer.id=MyPeer.0]"

Metric class organization

Instruments are split into dedicated classes per service:

  • ZebusMetrics — shared Meter instance and PeerTag helper
  • TransportMetrics — transport-layer instruments (messages, bytes, connections, sockets)
  • DirectoryMetrics — directory-layer instruments (known peers, peer updates)

Multi-targeting

  • Abc.Zebus.csproj targets netstandard2.0;net10.0
  • Metrics code is entirely behind #if NET10_0_OR_GREATER; netstandard2.0 output is unchanged

.NET 10 compatibility fixes

Polyfill types that now exist in the BCL needed conditional compilation:

  • NullableAnnotations, IsExternalInit, SkipLocalsInitAttribute — wrapped in #if !NETCOREAPP
  • ExtendDictionary.GetValueOrDefault / ExtendIEnumerable.ToHashSet — excluded on NETCOREAPP to resolve ambiguities with BCL extension methods
  • ProtoBufConvertFormatterServices.GetUninitializedObjectRuntimeHelpers.GetUninitializedObject
  • MessageIdRNGCryptoServiceProviderRandomNumberGenerator.Fill
  • MessageProcessingException / DomainException — serialization constructors excluded on NETCOREAPP (SYSLIB0051)
  • ProtoBufferReader.TryReadGuid — fixed definite assignment of out parameter
  • BindingKey — replaced extension method call with explicit TryGetValue to avoid ambiguity

var parts = new string[routingMembers.Length];
for (var tokenIndex = 0; tokenIndex < routingMembers.Length; ++tokenIndex)
{
parts[tokenIndex] = fieldValues.GetValueOrDefault(routingMembers[tokenIndex].Member.Name, BindingKeyPart.StarToken);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copilot probably did that due to .NET double targeting.

@Kuinox

Kuinox commented Apr 28, 2026

Copy link
Copy Markdown

@copilot we may have multiple bus at the same time, having a active bus count metrics isn't very useful, but, we will want to track per bus instance.
You can do a static interlocked increment starting at 0 on the instance to get a stable id.

Copilot AI commented Apr 28, 2026

Copy link
Copy Markdown
Author

@copilot we may have multiple bus at the same time, having a active bus count metrics isn't very useful, but, we will want to track per bus instance.
You can do a static interlocked increment startin...

Done in e04af38. Bus now has a static _nextInstanceId incremented via Interlocked.Increment, and the zebus.bus.active UpDownCounter is tagged with zebus.bus.id per instance instead of being a global aggregate.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • REDACTED-bastard
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 35909 --endpoint 127.0.0.1:035909 --role client --parentprocessid 5536 --telemetryoptedin false (dns block)
  • non-existing-peer
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 35909 --endpoint 127.0.0.1:035909 --role client --parentprocessid 5536 --telemetryoptedin false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI requested a review from Kuinox April 28, 2026 18:08

@Kuinox Kuinox left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/Abc.Zebus/Core/Bus.cs Outdated

Started?.Invoke();
#if NET10_0_OR_GREATER
ZebusMetrics.BusActive.Add(1, ZebusMetrics.BusTag(_instanceId));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This metric isn't useful

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in e1e006b. The zebus.bus.active metric and related instance ID code have been removed from Bus.cs.


// Directory: peers
internal static readonly UpDownCounter<int> KnownPeerCount = Meter.CreateUpDownCounter<int>(
"zebus.directory.known_peers",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make a dedicated metric class for each services, this class should be only for the zebus client.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e1e006b. Metrics are now split into TransportMetrics (transport instruments), DirectoryMetrics (directory instruments), and ZebusMetrics (shared Meter and tag helpers only).

Comment thread src/Abc.Zebus/Transport/ZmqTransport.cs Outdated
{
outboundSocket.Send(bufferWriter.Buffer, bufferWriter.Position, transportMessage);
#if NET10_0_OR_GREATER
ZebusMetrics.MessagesSent.Add(1, ZebusMetrics.PeerTag(target.Id));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the metric also have the sender peer id ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — the received message metric includes the sender peer ID via transportMessage.Originator.SenderId:

TransportMetrics.MessagesReceived.Add(1, ZebusMetrics.PeerTag(transportMessage.Originator.SenderId));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm asking about the metric MessageSent.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, MessagesSent is tagged with the target peer ID: TransportMetrics.MessagesSent.Add(1, ZebusMetrics.PeerTag(target.Id)) where target is the destination Peer.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, MessagesSent includes the target peer ID: TransportMetrics.MessagesSent.Add(1, ZebusMetrics.PeerTag(target.Id))target.Id is the destination peer.

#if !NETCOREAPP
_cryptoServiceProvider.GetBytes(nodeId);
#else
RandomNumberGenerator.Fill(nodeId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RNGCryptoServiceProvider is obsolete on .NET Core (SYSLIB0023). RandomNumberGenerator.Fill is the recommended replacement and is available on all NETCOREAPP targets.

{
}

#if !NETCOREAPP

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The (SerializationInfo, StreamingContext) serialization constructor triggers SYSLIB0051 on .NET Core (binary serialization is obsolete). Excluding it on NETCOREAPP avoids the warning while keeping it for netstandard2.0 consumers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ltrzesniewski i suppose we want to keep this constructor ?

Copilot AI commented Apr 28, 2026

Copy link
Copy Markdown
Author

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • REDACTED-bastard
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 43095 --endpoint 127.0.0.1:043095 --role client --parentprocessid 5517 --telemetryoptedin false (dns block)
  • non-existing-peer
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 43095 --endpoint 127.0.0.1:043095 --role client --parentprocessid 5517 --telemetryoptedin false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI requested a review from Kuinox April 28, 2026 18:25

@Kuinox Kuinox left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a latency on the heartbeat, or other means to measure latency ? It could be cool to add a latency metric. @copilot

Comment thread src/Abc.Zebus/Monitoring/TransportMetrics.cs
Comment thread src/Abc.Zebus/Transport/ZmqTransport.cs Outdated
{
outboundSocket.Send(bufferWriter.Buffer, bufferWriter.Position, transportMessage);
#if NET10_0_OR_GREATER
ZebusMetrics.MessagesSent.Add(1, ZebusMetrics.PeerTag(target.Id));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm asking about the metric MessageSent.

…ndard, removing #ifdef on usings

Agent-Logs-Url: https://github.com/Abc-Arbitrage/Zebus/sessions/81a35d95-9277-416c-9f67-15fb83ff98cf

Co-authored-by: Kuinox <18743295+Kuinox@users.noreply.github.com>

Copilot AI commented Apr 28, 2026

Copy link
Copy Markdown
Author

Is there a latency on the heartbeat, or other means to measure latency ? It could be cool to add a latency metric. @copilot

The directory has a PingPeerCommand handler that tracks TimeSinceLastPing via a Stopwatch, but it only measures elapsed time since the last ping was received — not round-trip latency. There's no built-in RTT measurement in the current transport or directory code that could be exposed as a latency metric without adding new protocol-level instrumentation. Want me to add one, or is that out of scope for this PR?

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • REDACTED-bastard
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 41415 --endpoint 127.0.0.1:041415 --role client --parentprocessid 5105 --telemetryoptedin false (dns block)
  • non-existing-peer
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 41415 --endpoint 127.0.0.1:041415 --role client --parentprocessid 5105 --telemetryoptedin false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI requested a review from Kuinox April 28, 2026 18:51
@Kuinox

Kuinox commented Apr 30, 2026

Copy link
Copy Markdown

@copilot I also want a byte count per message metric.

Copilot AI commented Apr 30, 2026

Copy link
Copy Markdown
Author

@copilot I also want a byte count per message metric.

Added in 4dd924d. Two new counters: zebus.transport.bytes.sent and zebus.transport.bytes.received, both tagged with zebus.peer.id. Sent bytes use bufferWriter.Position, received bytes use bufferReader.Length.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • REDACTED-bastard
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 37177 --endpoint 127.0.0.1:037177 --role client --parentprocessid 4414 --telemetryoptedin false (dns block)
  • non-existing-peer
    • Triggering command: /usr/share/dotnet/dotnet /usr/share/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/Abc.Zebus.Tests.deps.json /home/REDACTED/work/Zebus/Zebus/src/Abc.Zebus.Tests/bin/Debug/net9.0/testhost.dll --port 37177 --endpoint 127.0.0.1:037177 --role client --parentprocessid 4414 --telemetryoptedin false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@Kuinox

Kuinox commented May 6, 2026

Copy link
Copy Markdown

Why do we need to create a gauge instance per transport ? @copilot

@Kuinox Kuinox left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copilot AI commented May 6, 2026

Copy link
Copy Markdown
Author

Why do we need to create a gauge instance per transport ? @copilot

The ObservableGauge needs a callback that captures per-instance state (_outboundSockets) to report each peer's alive/dead status. Unlike counters where you call .Add() imperatively, observable instruments pull current state on each scrape — so the callback must reference the specific transport's socket dictionary.

If you'd prefer, I can eliminate the per-instance gauge entirely since we already track socket state via the OutboundSocketCount UpDownCounter and PeerConnections/PeerDisconnections counters. Alternatively I can make it static by having transports register/unregister from a shared collection. What approach would you prefer?

@Kuinox Kuinox marked this pull request as ready for review May 6, 2026 17:15
Copilot finished work on behalf of Kuinox May 6, 2026 17:15
Copilot AI requested a review from Kuinox May 6, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants