ftrio-conformance

A single, language-agnostic set of test vectors that every FtrIO port runs against, so that any behavioural drift between ports shows up as a failing test rather than as a bug someone files two years later.

The fixtures are the contract. They are pure JSON, owned by this one repository. Every port (the .NET origin, Python, Rust, and the BASIC and bare-metal ports to come) adapts itself to the fixtures, never the other way around. When the fixtures and a port disagree, either the port has a bug or the contract genuinely changed; both are things you want surfaced loudly.

Why it exists

FtrIO is ported to several languages. Each port has its own tests, but nothing guaranteed that a users:alice,bob targeting rule, an ab:50 bucket, or the offline default meant the same thing in every port. This suite is that guarantee: one set of vectors, run natively by each port's own test framework, gating each port's CI.

What is pinned

Only deterministic behaviour that must match across languages:

• the six strategies given a user context (boolean; percentage at the 0%/100% boundaries; blue-green; user targeting; attribute rules with every operator; A/B with a user);
• override precedence and resolution order;
• the offline default (no config resolves on);
• config parsing, value coercion, and case-insensitive key matching;
• the canonical error conditions;
• the deterministic SHA-256 A/B bucketing function.

Genuinely non-deterministic paths (percentage rollout and A/B with no user, which roll a random number) are pinned only at their deterministic boundaries. See SEMANTICS.md for the full contract and docs/BEHAVIOUR_CATALOGUE.md for how it was mined from the shipped ports.

The two kinds of case

Resolution case (fixtures/resolution/, one file per area): a config, an optional context, a toggle key, and an expected outcome that is either a boolean result or a named error. Exercises the whole resolution path.

{
  "id": "override_true_beats_false_value",
  "config": {
    "Toggles": { "NewCheckout": false },
    "TogglesOverrides": { "NewCheckout": { "alice": true } }
  },
  "context": { "userId": "alice", "attributes": {} },
  "toggleKey": "NewCheckout",
  "expect": { "result": true }
}

A/B bucket case (fixtures/bucket/ab-vectors.json): a user id, toggle key, and salt, with an expected integer bucket (0 to 99). Exercises the SHA-256 bucketing function directly, where cross-language determinism is most fragile.

{ "id": "ab_alice_unsalted", "userId": "alice", "toggleKey": "TestingABTest", "salt": "", "expect": { "bucket": 84 } }

Both kinds have a JSON Schema under schema/; every fixture validates against its schema.

Layout

ftrio-conformance/
├── schema/            JSON Schemas for the two case kinds
├── fixtures/
│   ├── resolution/    resolution cases, one file per area
│   └── bucket/        the pinned A/B bucket vectors
├── reference/         reference validators (Rust, Python): prove the fixtures + worked examples
├── driver/            the cross-port matrix driver (Phase 6)
├── docs/              the behaviour catalogue, adoption guide, and port-fix patches
├── SEMANTICS.md       the human-readable contract
├── README.md
└── CHANGELOG.md

How a port adopts it

1. Add this repository as a git submodule at a stable path (for example tests/conformance), pinned to a released tag (not a moving branch) so the contract can never silently change under a port.
2. Write one native conformance test (the port's own adapter) in the port's own framework: glob the fixture JSON, iterate every case, feed config + context + toggleKey through the port's real resolver, and assert the expected result or error; and iterate bucket/ab-vectors.json through the port's bucket function. This adapter lives in the port's own repository, not here.
3. Add it to CI so drift fails the build.

A new port passing this suite is the definition of "faithful". Full instructions: docs/ADOPTING.md.

The fixtures stand alone. They are the contract; the JSON in fixtures/ plus SEMANTICS.md is everything you need to write an adapter. You do not have to run, or even look at, anything in reference/ to do so. That keeps the repository honestly language-neutral: the reference validators are conveniences, not a dependency.

Versioning

The suite is semver-tagged, and each port pins a version. A change to the contract is a version bump that ports adopt on purpose, so the suite can never silently break every port at once. Every contract change is recorded in CHANGELOG.md.

The reference validators (optional)

reference/ holds two reference validators, in Rust and Python. They serve two purposes, neither of which the fixtures depend on:

1. They prove the fixtures against known-good implementations. Running every case through a real, shipped port is how we know the fixtures are internally consistent and not merely self-agreeing. This is why there are two, in different languages, rather than one.
2. They are the worked examples that other ports copy when writing their own adapters. They show the "glob the fixtures, feed each case through the resolver, compare" loop end to end.

They are deliberately not the ports' own tests: each port's real conformance test (its adapter) lives in that port's repository and gates that port's CI. The validators here are a starting point to copy, and a check that the contract itself is sound. You can clone this suite, read the JSON, and write an adapter without ever running either one.

Rust (drives the real Rust port):

cd reference/rust && cargo run

Python (drives the real Python port; needs ftrio importable and pip install jsonschema):

python reference/python/run.py

Each prints a per-file pass/fail summary and exits non-zero on any failure.