fix: enforce cash constraint in challenge replay scoring (hidden leverage inflates challenge leaderboard) - #262
Open
haidrrrry wants to merge 1 commit into
Open
Conversation
Live trading rejects buys and shorts whose value exceeds available cash, but score_agent_trades replayed challenge trades with no cash check, so replay cash could go negative silently. The only guard, max_position_pct, is per-symbol: spreading buys across N symbols keeps each position at or under 100% of equity while total exposure reaches roughly N times the challenge capital. A participant trading a larger live account inside a smaller-capital challenge (or choosing a small client-supplied starting_cash at join time, which was entirely unvalidated) therefore gained hidden leverage and an inflated return_pct on the challenge leaderboard. Example: 10k challenge capital, ten 10k buys in ten symbols -> cash -90k, equity 10k, every position exactly 100% -> passed all checks and scored with ~10x leverage. Fixes: - challenge_scoring.py: disqualify (insufficient_challenge_cash:SYMBOL) when a buy or short pushes replay cash below zero, mirroring the live "Insufficient cash" rule; shorts escrow price*quantity in the live model so they are bounded the same way - challenges.py: reject non-finite or non-positive starting_cash on join Adds regression tests: overspend across symbols and over-short are disqualified (previously scored normally); exact full deployment and partial deployment remain unaffected. Full server suite passes (93 tests, 2 skipped).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
score_agent_trades(challenge replay scoring) appliesbuyandshorttrades without any cash check, so replay cash can go negative silently. Live trading rejects exactly this (Insufficient cash. Required: ...inroutes_signals.py), so the replay is inconsistent with the platform's own no-leverage rule.The only guard,
max_position_pct, is per-symbol. Spreading buys across N symbols keeps every position at/under 100% of equity while total exposure reaches ~N× the challenge capital:Because challenge trades are mirrored from live signals, a participant trading a larger live account inside a smaller-capital challenge gains hidden leverage and an inflated
return_pcton the challenge leaderboard. On top of that,starting_cashat join time is client-supplied and entirely unvalidated (ChallengeJoinRequest.starting_cash), including negative, zero, NaN, or Infinity.Fix
challenge_scoring.py: disqualify withinsufficient_challenge_cash:SYMBOLwhen abuyorshortpushes replay cash below zero (tolerance-1e-9, matching the file's existing epsilons). Shorts escrowprice * quantityin the live model, so they are cash-bounded exactly like buys. Disqualification matches how the scorer already treats other rule breaches (sell_exceeds_challenge_long,max_position_pct_exceeded).challenges.py: reject non-finite or non-positivestarting_cashon join.Tests
tests/test_challenge_scoring_leverage.py(pure-function tests againstscore_agent_trades):disqualified_reason=None)insufficient_challenge_cash:BBBFull server suite: 93 tests pass (2 skipped), no regressions.