Motivation:
Metrics are currently defined in DecisionFocusedLearningAlgorithms.jl rather than in DecisionFocusedLearningBenchmarks.jl.
This separation feels unintuitive: when implementing a new benchmark, one typically also implements a set of reference policies for that benchmark. The metrics used to evaluate these policies are naturally benchmark-specific and therefore seem to belong alongside the benchmark itself.
Proposed change:
Move benchmark-specific metrics to DecisionFocusedLearningBenchmarks.jl. This would allow each benchmark to expose reference policies and the metrics used to evaluate them.
As a result, anyone implementing a new learning algorithm in DecisionFocusedLearningAlgorithms.jl could directly benchmark it against the existing policies and metrics provided by the benchmark package, without having to add benchmark-specific code to the algorithms package.
Motivation:
Metrics are currently defined in DecisionFocusedLearningAlgorithms.jl rather than in DecisionFocusedLearningBenchmarks.jl.
This separation feels unintuitive: when implementing a new benchmark, one typically also implements a set of reference policies for that benchmark. The metrics used to evaluate these policies are naturally benchmark-specific and therefore seem to belong alongside the benchmark itself.
Proposed change:
Move benchmark-specific metrics to DecisionFocusedLearningBenchmarks.jl. This would allow each benchmark to expose reference policies and the metrics used to evaluate them.
As a result, anyone implementing a new learning algorithm in DecisionFocusedLearningAlgorithms.jl could directly benchmark it against the existing policies and metrics provided by the benchmark package, without having to add benchmark-specific code to the algorithms package.