Reliability rework: leave 'New' faster, sharper reliable/unreliable split, four-stop create-game slider

[JorenC]

Goal

Rework the reliability tier system so committed players leave "New" much faster and the Reliable/Unreliable split actually differentiates players. Today "New" means a player hasn't completed ~10 games — a year-plus of real-world play — and a single 10% NMR line lumps the improving majority in with serial abandoners. Move to a four-tier system (unreliable < new < neutral < reliable) judged on order-phases rather than completed games, and surface it as a four-stop create-game slider.

Definition of done

• get_player_stats returns one of four tiers: unreliable, new, neutral, reliable.
• Tier is derived from a volume gate counted in rated order-phases (phases where the player had possible orders), and that count includes resolved phases from in-progress games, not only completed games — so a player crosses out of new within a game or two of decisions rather than after ~10 completed games.
• The NMR rate that drives the tier is smoothed toward a neutral baseline, so a small sample can't whipsaw a player to an extreme on one missed order.
• Civil disorder is applied as a clamp on top of the rate — a recent CD caps a player below reliable, repeated CDs force unreliable — rather than being mixed into the per-phase rate.
• Game.min_reliability supports four values corresponding to the slider stops Open / New / Neutral / Reliable; tier_allows_min_reliability, the join permission, and the eligible_only game-list filter all enforce them by tier rank, and an ineligible join is rejected.
• The create-game "Player Reliability" control is a four-stop stepped slider, capped at the creator's own tier (stops above are greyed out and unselectable) and defaulting to the highest stop available to them. This subsumes and closes Create game allows selecting min reliability higher than the current user's own tier #850.
• The numeric thresholds (the phase gate, the smoothing constants, the reliable/unreliable cutoffs, and the CD window) are calibrated against the real player distribution to a sane tier mix, not shipped at placeholder values.

Context

Current logic lives in service/user_profile/utils.py: the gate counts GameStatus.COMPLETED games and ignores in-progress ones, and a single 0.1 line separates reliable from unreliable against a population NMR rate around 40.9% — which is why only ~7.89% of players read as Reliable (see #581). The create-game control and its tier-cap behaviour are specified in #850, which this absorbs. Backend tier derivation needs to land before the four-stop slider can be wired, and calibration depends on the tier logic existing.

[JorenC]

@johnpooch what do you think of the above? If 90% is new, only 500 players have finished a game - and we're waiting for 10 games, it's a bit irrelevant - if all players are the same status.

This adds a "neutral" state, and moves people way faster our of new. It also has a protector to move people quickly into a certain category - I think it's much much better.

[JorenC]

Comes out of the North Star reliability discussion #581 — this RFC is the concrete tier-system slice of that thread.

[johnpooch]

Pulled fresh production numbers to sharpen the figures in the proposal. Confirming
the core problem: "New" is gated on 10 completed games (RELIABILITY_GAME_WINDOW
in user_profile/utils.py), not orders/phases.

Among the 1,589 users who have ever joined a game:

Metric	Value
Finished ≥1 game	547 (34%)
Finished ≥2 games	210 (13%)
Finished ≥10 games (can leave "New")	26 (1.6%)
Currently "New"	98.4%
Reliable	9 (0.57% of all members)
Unreliable (None)	17
Population per-phase NMR rate	36.6%

So the "~90% in New" estimate is actually conservative — it's 98.4%, and only
26 players in the whole database have ever been rated at all. Of those, 9 read
as Reliable and 17 as Unreliable.

(Couldn't reconcile #581's "7.89% Reliable" with current code, which gives 0.57% of
all members / 1.6% of finishers — that older figure likely predates the 10-game gate
or used a different population. The 40.9% vs my 36.6% population NMR is close enough
to be the same metric on newer data.)

[johnpooch]

Worth seriously considering switching the rating unit from completed games to
rated order-phases, for two reasons.

1. It moves engaged players out of "New" far faster. A game can resolve in a
handful of phases or run for dozens, so "10 games" is a wildly inconsistent bar.
Same nominal threshold of 10, games vs. phases:

Threshold	Users qualifying
≥10 completed games	26
≥10 rated phases	600
≥20 rated phases	382
≥30 rated phases	250

~23× more players cross the bar at the same headline number — and phases count
in-progress games too, so a committed player isn't invisible until their first
game finally ends.

2. More denominator = less small-sample noise in the NMR rate, which is exactly
what the "smooth toward a neutral baseline" idea is compensating for.

The calibration caveat — the distribution of rated phases per user (users who have
any phase activity):

Rated phases	Users
1–2	30
3–5	70
6–10	102
11–20	212
21–40	191
40+	179

There's a large one-and-done tail (median rated phases ≈ 1 across all members), so
the "New" exit threshold has to sit above it. A phase count somewhere in the 6–20
band looks like the natural place to draw the first tier boundary, but the exact
number should be picked off this distribution rather than by analogy to the old
10-games rule.

[JorenC]

So I think we're in agreement here?

Is there anything you want to discuss further?

I think it's very good to add a 4th band.
Naming is incorrect (unreliable/neutral is negative. It should be

• no badge
• new
• commited
• very committed

We should not use reliable - it implies their character/trustworthiness.

Alternatives could be

• engaged
• active