Skip to content

fix: enforce cash constraint in challenge replay scoring (hidden leverage inflates challenge leaderboard) - #262

Open
haidrrrry wants to merge 1 commit into
HKUDS:mainfrom
haidrrrry:fix/challenge-scoring-leverage
Open

fix: enforce cash constraint in challenge replay scoring (hidden leverage inflates challenge leaderboard)#262
haidrrrry wants to merge 1 commit into
HKUDS:mainfrom
haidrrrry:fix/challenge-scoring-leverage

Conversation

@haidrrrry

Copy link
Copy Markdown
Contributor

Problem

score_agent_trades (challenge replay scoring) applies buy and short trades without any cash check, so replay cash can go negative silently. Live trading rejects exactly this (Insufficient cash. Required: ... in routes_signals.py), so the replay is inconsistent with the platform's own no-leverage rule.

The only guard, max_position_pct, is per-symbol. Spreading buys across N symbols keeps every position at/under 100% of equity while total exposure reaches ~N× the challenge capital:

10k challenge capital → ten 10k buys in ten symbols → cash −90k, equity 10k, each position exactly 100% → passes all checks, scores with ~10× leverage.

Because challenge trades are mirrored from live signals, a participant trading a larger live account inside a smaller-capital challenge gains hidden leverage and an inflated return_pct on the challenge leaderboard. On top of that, starting_cash at join time is client-supplied and entirely unvalidated (ChallengeJoinRequest.starting_cash), including negative, zero, NaN, or Infinity.

Fix

  • challenge_scoring.py: disqualify with insufficient_challenge_cash:SYMBOL when a buy or short pushes replay cash below zero (tolerance -1e-9, matching the file's existing epsilons). Shorts escrow price * quantity in the live model, so they are cash-bounded exactly like buys. Disqualification matches how the scorer already treats other rule breaches (sell_exceeds_challenge_long, max_position_pct_exceeded).
  • challenges.py: reject non-finite or non-positive starting_cash on join.

Tests

tests/test_challenge_scoring_leverage.py (pure-function tests against score_agent_trades):

Case Before fix After fix
Two 6k buys on 10k capital (60% per symbol, cash −2k) scored normally (disqualified_reason=None) disqualified insufficient_challenge_cash:BBB
Two 6k shorts on 10k capital scored normally disqualified
Exact full deployment (buy 10k, sell at +10%) 10% return unchanged, 10% return
Partial deployment (buy 6k, sell at +20%) 12% return unchanged, 12% return

Full server suite: 93 tests pass (2 skipped), no regressions.

Live trading rejects buys and shorts whose value exceeds available cash,
but score_agent_trades replayed challenge trades with no cash check, so
replay cash could go negative silently.

The only guard, max_position_pct, is per-symbol: spreading buys across
N symbols keeps each position at or under 100% of equity while total
exposure reaches roughly N times the challenge capital. A participant
trading a larger live account inside a smaller-capital challenge (or
choosing a small client-supplied starting_cash at join time, which was
entirely unvalidated) therefore gained hidden leverage and an inflated
return_pct on the challenge leaderboard.

Example: 10k challenge capital, ten 10k buys in ten symbols -> cash
-90k, equity 10k, every position exactly 100% -> passed all checks and
scored with ~10x leverage.

Fixes:
- challenge_scoring.py: disqualify (insufficient_challenge_cash:SYMBOL)
  when a buy or short pushes replay cash below zero, mirroring the live
  "Insufficient cash" rule; shorts escrow price*quantity in the live
  model so they are bounded the same way
- challenges.py: reject non-finite or non-positive starting_cash on join

Adds regression tests: overspend across symbols and over-short are
disqualified (previously scored normally); exact full deployment and
partial deployment remain unaffected. Full server suite passes
(93 tests, 2 skipped).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant