Skip to content

Update BARouter submission - #180

Open
zulk2002 wants to merge 1 commit into
RouteWorks:mainfrom
zulk2002:resubmit-barouter
Open

Update BARouter submission#180
zulk2002 wants to merge 1 commit into
RouteWorks:mainfrom
zulk2002:resubmit-barouter

Conversation

@zulk2002

Copy link
Copy Markdown
Contributor

This PR updates the BARouter submission artifacts for RouterArena leaderboard evaluation.
BARouter is a budget-adaptive, cost-aware router. For this submission, the router selects among:

google/gemini-3.1-flash-lite
deepseek/deepseek-v4-flash
deepseek/deepseek-v4-pro

The submitted artifacts contain the final routing decisions and generated outputs. No local predictor artifacts or training files are included.

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: BARouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7375
Accuracy 75.33%
Total Cost $2.444208
Avg Cost per Query $0.000291
Avg Cost per 1K Queries $0.2910
Number of Queries 8400
Abnormal Entries 25
Robustness Score 0.8310

⚠️ 25 of 8400 queries (0.3%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.7966
Opt.Cost (Cost Efficiency) 0.7207
Opt.Acc (Accuracy vs Optimal) 0.9245

Evaluation completed by RouterArena automated workflow

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: BARouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7379
Accuracy 75.72%
Total Cost $3.058621
Avg Cost per Query $0.000364
Avg Cost per 1K Queries $0.3641
Number of Queries 8400
Abnormal Entries 21
Robustness Score 0.6881

⚠️ 21 of 8400 queries (0.3%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.6441
Opt.Cost (Cost Efficiency) 0.6713
Opt.Acc (Accuracy vs Optimal) 0.9384

Evaluation completed by RouterArena automated workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant