Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RequestFlow.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
<File Path="benchmarks/README.md" />
<Project Path="benchmarks/RequestFlow.Benchmarks/RequestFlow.Benchmarks.csproj" />
<Project Path="benchmarks/RequestFlow.Benchmarks.Core/RequestFlow.Benchmarks.Core.csproj" />
<Project Path="benchmarks/RequestFlow.Benchmarks.DispatchR/RequestFlow.Benchmarks.DispatchR.csproj" />
<Project Path="benchmarks/RequestFlow.Benchmarks.Harness/RequestFlow.Benchmarks.Harness.csproj" />
<Project Path="benchmarks/RequestFlow.Benchmarks.MediatR/RequestFlow.Benchmarks.MediatR.csproj" />
<Project Path="benchmarks/RequestFlow.Benchmarks.RequestFlowFixtures/RequestFlow.Benchmarks.RequestFlowFixtures.csproj" />
Expand Down
25 changes: 20 additions & 5 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
# Benchmarks

Two BenchmarkDotNet suites, both on .NET 10:
Three BenchmarkDotNet suites, all on .NET 10:

| Suite | Cases | Measures |
| --- | --- | --- |
| `RequestFlow.Benchmarks.MediatR` | 224 | RequestFlow against MediatR 12.5.0 in one executable, with matching workloads and service lifetimes. See the [MediatR comparison](RequestFlow.Benchmarks.MediatR/README.md). |
| `RequestFlow.Benchmarks.DispatchR` | 260 | RequestFlow against DispatchR.Mediator 2.3.1 in one executable, with matching workloads and service lifetimes. See the [DispatchR comparison](RequestFlow.Benchmarks.DispatchR/README.md). |
| `RequestFlow.Benchmarks` | 34 | RequestFlow alone: Task vs ValueTask paths and stage chains. |

- A case is one benchmark method with one parameter set.
- Published results: [Benchmarks vs MediatR](../docs/benchmarks/mediatr/README.md), from the 2026-10-15 full run.
- RequestFlow is built from the checked-out source, so record the commit when sharing results.
- BenchmarkDotNet 0.15.8 and Microsoft DI 10.0.9. Package lock files pin the resolved dependencies.
- CI builds both suites but does not run them.
- CI builds all three suites but does not run them.

## Quick start

Expand Down Expand Up @@ -63,7 +64,7 @@ dotnet run --project benchmarks/RequestFlow.Benchmarks -c Release -f net10.0 --n
| `--filter` without `--smoke` or `--quick` | Runs `probe`, then the full profile: two launches, 15 to 50 warm-up and 60 to 100 measured iterations, 5% relative error target. |

- Without arguments, `dotnet run` applies the MediatR suite's first launch profile, **Verify comparison**, so it only runs `probe`.
- The RequestFlow suite has no launch profile. Without arguments it runs `probe`, then shows BenchmarkDotNet's selection prompt.
- The DispatchR and RequestFlow suites have no launch profile. Without arguments they run `probe`, then show BenchmarkDotNet's selection prompt.

## Filters

Expand All @@ -83,9 +84,22 @@ dotnet run --project benchmarks/RequestFlow.Benchmarks -c Release -f net10.0 --n
| Short circuit | `*MediatRShortCircuit*` | 18 |
| Direct-handler reference | `*MediatRTaskDirectHandler*` | 6 |

| DispatchR scenario | Filter | Cases |
| --- | --- | --- |
| Task requests | `*DispatchRTask*Scope*` | 36 |
| Void requests | `*DispatchRVoidRequest*` | 24 |
| ValueTask requests | `*DispatchRValueRetainedScope*` `*DispatchRValueScopePerOperation*` | 24 |
| ValueTask void requests | `*DispatchRValueVoidRequest*` | 24 |
| Streams | `*DispatchRStream*` | 72 |
| Sequential events | `*DispatchREventSequential*` | 16 |
| Catalog size | `*DispatchRCatalogSize*` | 18 |
| Mixed traffic | `*DispatchRMixedTraffic*` | 18 |
| Short circuit | `*DispatchRShortCircuit*` | 18 |
| Direct-handler references | `*DispatchR*DirectHandler*` | 10 |

## Reports

- `--artifacts <dir>` sets the output directory. Without it, reports go to `BenchmarkDotNet.Artifacts/<suite>/<tier>/<run>/` under the current directory, where `<suite>` is `mediatr` or `requestflow`.
- `--artifacts <dir>` sets the output directory. Without it, reports go to `BenchmarkDotNet.Artifacts/<suite>/<tier>/<run>/` under the current directory, where `<suite>` is `mediatr`, `dispatchr` or `requestflow`.
- Readable tables are the `results/*-report-github.md` files. `run-manifest.json` records the run status and host details.
- Git ignores `BenchmarkDotNet.Artifacts/` and `benchmarks/artifacts/`.

Expand Down Expand Up @@ -126,12 +140,13 @@ Use Visual Studio for exploration only. Publish from the command line.

## Projects

All five are in the root solution.
All six are in the root solution.

| Project | Contents |
| --- | --- |
| `RequestFlow.Benchmarks` | Stage-chain measurements, Task vs ValueTask dispatch, and probes |
| `RequestFlow.Benchmarks.MediatR` | MediatR fixtures, paired benchmarks, and probes. Does not depend on `RequestFlow.Benchmarks`. |
| `RequestFlow.Benchmarks.DispatchR` | DispatchR fixtures, paired benchmarks, and probes. Does not depend on `RequestFlow.Benchmarks`. |
| `RequestFlow.Benchmarks.Harness` | BenchmarkDotNet configuration, runner, run manifest, and provenance records. Knows nothing about workloads. |
| `RequestFlow.Benchmarks.Core` | Probe engine, trace names, provider hosts, shared workloads, and the short-circuit dependency graph |
| `RequestFlow.Benchmarks.RequestFlowFixtures` | RequestFlow handlers, stages, registration, and message catalogs |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ public static T[] Params<T>(Type benchmarkType, string member)
/// A <see cref="ScopeLifetime.RetainedScope"/> class must retain one scope across calls.
/// A <see cref="ScopeLifetime.DirectHandler"/> class is exempt from scope checks.
/// A <c>Completion</c> parameter fixes whether the returned awaitable is complete when the call returns.
/// Every message field of such a class must carry that <c>Completion</c>.
/// </remarks>
public static BenchmarkProbeResult Run(
Assembly assembly,
Expand Down Expand Up @@ -230,6 +231,8 @@ private static BenchmarkObservation ObserveMethod(
.Select(message => (Message: message, OriginalTrace: message.Trace))
.ToArray();

VerifyMessageCompletion(type, method, parameters, [.. messages.Select(message => message.Message)]);

try
{
foreach (var message in messages)
Expand Down Expand Up @@ -274,6 +277,24 @@ private static BenchmarkObservation ObserveMethod(
}
}

private static void VerifyMessageCompletion(
Type type, MethodInfo method, IReadOnlyDictionary<string, object?> parameters, WorkloadMessage[] messages)
{
if (!parameters.TryGetValue("Completion", out object? value) || value is not CompletionKind completion)
return;

if (messages.Length == 0)
throw new InvalidOperationException(
$"'{type.Name}.{method.Name}' has a Completion parameter but no message field to check it against.");

foreach (WorkloadMessage message in messages)
{
if (message.Completion != completion)
throw new InvalidOperationException(
$"'{type.Name}.{method.Name}' sends {message.GetType().Name} with Completion={message.Completion}, not {completion}.");
}
}

private static void VerifyCompletionShape(BenchmarkObservation observation, bool? completedOnReturn)
{
bool? synchronous = observation.Parameters.TryGetValue("Completion", out object? completion) && completion is CompletionKind kind
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
using BenchmarkDotNet.Attributes;
using RequestFlow.Benchmarks.Core;
using RequestFlow.Benchmarks.RequestFlowFixtures;
using RequestFlow.Benchmarks.RequestFlowFixtures.Catalog;

namespace RequestFlow.Benchmarks.DispatchR;

[BenchmarkScope(ScopeLifetime.RetainedScope)]
public class DispatchRCatalogSizeBenchmarks
{
[Params(4, 16, 64)]
public int CatalogSize { get; set; }

[Params(CompletionKind.Cached, CompletionKind.Synchronous, CompletionKind.Suspending)]
public CompletionKind Completion { get; set; }

private RetainedScopeHost<global::RequestFlow.IRequestDispatcher> _requestFlow = null!;
private RetainedScopeHost<global::DispatchR.IMediator> _dispatchR = null!;

private RuntimeTaskRequest<RuntimeTag1> _requestFlow1 = null!;
private DispatchRCatalogRequest<RuntimeTag1> _dispatchR1 = null!;

[GlobalSetup]
public void Setup()
{
_requestFlow = new(RequestFlowSetup.Build(RequestFlowSetup.CatalogServices(CatalogSize)));
_dispatchR = new(DispatchRSetup.Build(DispatchRSetup.CatalogServices(CatalogSize)));

_requestFlow1 = new RuntimeTaskRequest<RuntimeTag1>(Completion);
_dispatchR1 = new DispatchRCatalogRequest<RuntimeTag1>(Completion);

RequestFlow().GetAwaiter().GetResult();
DispatchR().GetAwaiter().GetResult();
}

[Benchmark(Baseline = true)]
public Task<int> RequestFlow()
=> _requestFlow.Service.SendAsync(_requestFlow1, CancellationToken.None);

[Benchmark]
public Task<int> DispatchR()
=> _dispatchR.Service.Send(_dispatchR1, CancellationToken.None);

[GlobalCleanup]
public void Cleanup()
{
_requestFlow.Dispose();
_dispatchR.Dispose();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
using BenchmarkDotNet.Attributes;
using RequestFlow.Benchmarks.Core;
using RequestFlow.Benchmarks.RequestFlowFixtures;

namespace RequestFlow.Benchmarks.DispatchR;

[BenchmarkScope(ScopeLifetime.RetainedScope)]
public class DispatchREventSequentialRetainedScopeBenchmarks
{
[Params(CompletionKind.Synchronous, CompletionKind.Suspending)]
public CompletionKind Completion { get; set; }

[Params(1, 5)]
public int HandlerCount { get; set; }

private RetainedScopeHost<global::RequestFlow.IEventPublisher> _requestFlow = null!;
private RetainedScopeHost<global::DispatchR.IMediator> _dispatchR = null!;

private EventMessage _requestFlowRequest = null!;
private DispatchREvent _dispatchRRequest = null!;

[GlobalSetup]
public void Setup()
{
_requestFlow = new(RequestFlowSetup.Build(RequestFlowSetup.EventServices(HandlerCount, parallel: false)));
_dispatchR = new(DispatchRSetup.Build(DispatchRSetup.EventServices(HandlerCount)));

_requestFlowRequest = new EventMessage(Completion);
_dispatchRRequest = new DispatchREvent(Completion);

RequestFlow().GetAwaiter().GetResult();
DispatchR().GetAwaiter().GetResult();
}

[Benchmark(Baseline = true)]
public Task RequestFlow()
=> _requestFlow.Service.PublishAsync(_requestFlowRequest, CancellationToken.None);

[Benchmark]
public ValueTask DispatchR()
=> _dispatchR.Service.Publish(_dispatchRRequest, CancellationToken.None);

[GlobalCleanup]
public void Cleanup()
{
_requestFlow.Dispose();
_dispatchR.Dispose();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
using BenchmarkDotNet.Attributes;
using Microsoft.Extensions.DependencyInjection;
using RequestFlow.Benchmarks.Core;
using RequestFlow.Benchmarks.RequestFlowFixtures;

namespace RequestFlow.Benchmarks.DispatchR;

[BenchmarkScope(ScopeLifetime.ScopePerOperation)]
public class DispatchREventSequentialScopePerOperationBenchmarks
{
[Params(CompletionKind.Synchronous, CompletionKind.Suspending)]
public CompletionKind Completion { get; set; }

[Params(1, 5)]
public int HandlerCount { get; set; }

private ScopePerOperationHost _requestFlow = null!;
private ScopePerOperationHost _dispatchR = null!;

private EventMessage _requestFlowRequest = null!;
private DispatchREvent _dispatchRRequest = null!;

[GlobalSetup]
public void Setup()
{
_requestFlow = new(RequestFlowSetup.Build(RequestFlowSetup.EventServices(HandlerCount, parallel: false)));
_dispatchR = new(DispatchRSetup.Build(DispatchRSetup.EventServices(HandlerCount)));

_requestFlowRequest = new EventMessage(Completion);
_dispatchRRequest = new DispatchREvent(Completion);

RequestFlow().GetAwaiter().GetResult();
DispatchR().GetAwaiter().GetResult();
}

[Benchmark(Baseline = true)]
public async Task RequestFlow()
{
using IServiceScope scope = _requestFlow.Provider.CreateScope();

var publisher = scope.ServiceProvider
.GetRequiredService<global::RequestFlow.IEventPublisher>();

await publisher
.PublishAsync(_requestFlowRequest, CancellationToken.None).ConfigureAwait(false);
}

[Benchmark]
public async ValueTask DispatchR()
{
using IServiceScope scope = _dispatchR.Provider.CreateScope();

var publisher = scope.ServiceProvider
.GetRequiredService<global::DispatchR.IMediator>();

await publisher
.Publish(_dispatchRRequest, CancellationToken.None).ConfigureAwait(false);
}

[GlobalCleanup]
public void Cleanup()
{
_requestFlow.Dispose();
_dispatchR.Dispose();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
using System.Reflection;
using BenchmarkDotNet.Attributes;
using RequestFlow.Benchmarks.Core;
using RequestFlow.Benchmarks.RequestFlowFixtures;
using RequestFlow.Benchmarks.RequestFlowFixtures.Catalog;

namespace RequestFlow.Benchmarks.DispatchR;

[BenchmarkScope(ScopeLifetime.RetainedScope)]
public class DispatchRMixedTrafficBenchmarks
{
private static readonly MethodInfo BindDispatchRSendDefinition =
typeof(DispatchRMixedTrafficBenchmarks).GetMethod(nameof(BindDispatchRSend), BindingFlags.NonPublic | BindingFlags.Static)!;

[Params(4, 16, 64)]
public int CatalogSize { get; set; }

[Params(CompletionKind.Cached, CompletionKind.Synchronous, CompletionKind.Suspending)]
public CompletionKind Completion { get; set; }

private RetainedScopeHost<global::RequestFlow.IRequestDispatcher> _requestFlow = null!;
private RetainedScopeHost<global::DispatchR.IMediator> _dispatchR = null!;

private global::RequestFlow.IRequest<int>[] _requestFlowRequests = null!;
private WorkloadMessage[] _dispatchRRequests = null!;

private Func<global::RequestFlow.IRequestDispatcher, Task<int>>[] _requestFlowSends = null!;
private Func<global::DispatchR.IMediator, Task<int>>[] _dispatchRSends = null!;

[GlobalSetup]
public void Setup()
{
_requestFlow = new(RequestFlowSetup.Build(RequestFlowSetup.CatalogServices(CatalogSize)));
_dispatchR = new(DispatchRSetup.Build(DispatchRSetup.CatalogServices(CatalogSize)));

IReadOnlyList<Type> order = RuntimeCatalog.MixedTrafficOrder(CatalogSize);

_requestFlowRequests = CreateRequests<global::RequestFlow.IRequest<int>>(typeof(RuntimeTaskRequest<>), order);
_dispatchRRequests = CreateRequests<WorkloadMessage>(typeof(DispatchRCatalogRequest<>), order);

_requestFlowSends = [.. _requestFlowRequests.Select(RequestFlowSend)];
_dispatchRSends = [.. _dispatchRRequests.Select(DispatchRSend)];

RequestFlow().GetAwaiter().GetResult();
DispatchR().GetAwaiter().GetResult();
}

[Benchmark(Baseline = true, OperationsPerInvoke = RuntimeCatalog.MixedTrafficBatch)]
public async Task<int> RequestFlow()
{
int sum = 0;

foreach (Func<global::RequestFlow.IRequestDispatcher, Task<int>> send in _requestFlowSends)
sum += await send(_requestFlow.Service).ConfigureAwait(false);

return sum;
}

[Benchmark(OperationsPerInvoke = RuntimeCatalog.MixedTrafficBatch)]
public async Task<int> DispatchR()
{
int sum = 0;

foreach (Func<global::DispatchR.IMediator, Task<int>> send in _dispatchRSends)
sum += await send(_dispatchR.Service).ConfigureAwait(false);

return sum;
}

[GlobalCleanup]
public void Cleanup()
{
_requestFlow.Dispose();
_dispatchR.Dispose();
}

private TRequest[] CreateRequests<TRequest>(Type requestDefinition, IReadOnlyList<Type> order)
=> [.. order.Select(tag => (TRequest)Activator.CreateInstance(requestDefinition.MakeGenericType(tag), Completion)!)];

private static Func<global::RequestFlow.IRequestDispatcher, Task<int>> RequestFlowSend(global::RequestFlow.IRequest<int> request)
=> dispatcher => dispatcher.SendAsync(request, CancellationToken.None);

// DispatchR resolves the handler from the compile-time request type, so each send is bound to its tag here.
private static Func<global::DispatchR.IMediator, Task<int>> DispatchRSend(WorkloadMessage request)
=> (Func<global::DispatchR.IMediator, Task<int>>)BindDispatchRSendDefinition
.MakeGenericMethod(request.GetType().GetGenericArguments()[0])
.Invoke(null, [request])!;

private static Func<global::DispatchR.IMediator, Task<int>> BindDispatchRSend<TTag>(DispatchRCatalogRequest<TTag> request)
=> mediator => mediator.Send(request, CancellationToken.None);
}
Loading