[roncodes]

Incident

Fleetbase Cloud sandbox issues: test mode shows live data, and live can be served cached test data.

Root cause

Auth::setSandboxSession() switches database.default and fleetbase.connection.db to sandbox for console requests with Access-Console-Sandbox: true and for flb_test_ keys. Models then read the separate fleetbase_sandbox database.

ApiModelCache built its keys without any environment or connection part:

• query: {api_query}:{table}:company_{uuid}:{versions}:{md5(params, path, session scope)}
• model: {api_model}:{table}:{versions}:{id}{withHash}
• relationship: {api_relation}:{table}:{versions}:{id}:{relation}

The session scope (api_credential, storefront_store, storefront_network) separates public API keys, but console (Sanctum) requests carry no credential. A live and a sandbox console request for the same company, path and query string got the same key. Model and relationship keys only use the id or uuid. Integer ids collide between the two databases, and synced uuids (users, companies) exist in both.

Reproduction

Local stack, console token:

1. GET /int/v1/orders?limit=1&_probe=N returned live orders with X-Cache-Status: MISS.
2. The same request with Access-Console-Sandbox: true returned the same live orders with the same X-Cache-Key and X-Cache-Status: HIT.
3. The sandbox has 0 orders. A fresh query string in sandbox correctly returns [].

Fix

• Every key ApiModelCache generates (query, model, relationship) now carries the connection its model reads, right after the Redis hash tag. The connection comes from the new ApiModelCache::connectionName(): $model->getConnectionName() ?? config('database.default').
  • query: {api_query}:{connection}:{table}:...
  • model: {api_model}:{connection}:{table}:...
  • relationship: {api_relation}:{connection}:{table}:...
• Fleetbase models resolve fleetbase.connection.db at call time, and models without a connection fall back to database.default. setSandboxSession() switches both, so sandbox keys read sandbox and live keys read the live connection (e.g. mysql). A model pinned to an explicit connection keeps one key in both modes, because it reads the same database in both.
• invalidateQueryCache() builds its key through generateQueryCacheKey(), so it is covered too. The HasApiModelCache / HasApiModelBehavior traits only call these generators and needed no change.
• Live keys change format, so this is not backward compatible on purpose. Old entries go unread and expire (see Deploy note).

Version keys stay shared across connections (deliberate)

api_query_version:* / api_model_version:* are still keyed by table (and company), not by connection:

• A list key also carries the versions of every table its result embeds ($apiCacheDependencies, request with/expand/count). Other models write those tables, possibly on another connection (e.g. a storefront model under a core list). dependencyTables() only knows table names, not which connection the writer uses. If versions were per connection, a list would read the version under its own connection. A write on the dependency's connection would bump a different version, and that list would serve stale data until its TTL ran out. That is a correctness regression.
• Sharing only over-invalidates: a sandbox write moves live keys to new versions and the reverse, which costs a cache miss and never serves wrong data. Sandbox write volume is small next to live.
• The invalidation tags (api_cache, api_model:{table}, company:{uuid}) also stay shared for the same reason. On Redis they only drop tag sets, and the versions do the real invalidation (see the class docblock).

Other caches checked

None of these needed a change in this PR:

• ValidateETag middleware: compares the client's If-None-Match with the ETag of the response this request produced. Nothing is stored server side.
• AttachCacheHeaders: only reports ApiModelCache's status and key. It now shows the connection in X-Cache-Key.
• Spatie ResponseCache: core-api registers no CacheResponse middleware. It only calls ResponseCache::clear() (HttpCacheObserver, ClearCacheAfterDelete), so nothing is served from it.
• AuthController::session (session_validation_{token}), bootstrap (auth_bootstrap_{user}_{token}), getUserOrganizations (user_organizations_v2_{user}) and UserCacheService (user:current:{user}:{company}:{updated_at}): these cache identity and organization data from the users/companies tables, which are synced into the sandbox. The keys have no connection part. If those tables ever drift between the two databases, these could show the other side's copy for up to 5 to 30 minutes. They don't hold tenant operational data, so they are noted here, not changed. Scoping them would also need every Cache::forget call site to clear both connections.
• LookupController blog posts, Setting system settings, ApiRateLimits, Telemetry, SMS ledger/policy, OAuth/Apple key caches: global or credential-keyed, with no environment-specific data.

Tests

php -d memory_limit=512M vendor/bin/pest (PHP 8.4.0): 2185 passed (13691 assertions), 0 failed (47 deprecated and 8 warnings come from vendor packages and are pre-existing).
php-cs-fixer fix -v --dry-run (= composer test:lint): 0 of 874 files to fix. composer test:date-drift: passed.

New tests in tests/Unit/Support/ApiModelCacheTest.php. They switch to sandbox through the real Auth::setSandboxSession() with an Access-Console-Sandbox: true request:

• the same request under live vs sandbox produces different query, model and relationship keys, both for a plain Eloquent model (database.default) and a Fleetbase\Models\Model (fleetbase.connection.db). A connection-pinned model keeps one key.
• a list cached under live is a MISS under sandbox (it returns [], not the live orders), and each side then HITs its own entry. This is the incident reproduction.
• model and relationship lookups cached under live are not served to sandbox, and the reverse.
• a sandbox write bumps the shared versions and moves live keys to a new version (documents the over-invalidation choice).

Existing expected key strings were updated to the new {hash}:{connection}:{table} format. The first three new tests fail against the old ApiModelCache.

Deploy note

Existing cache entries keep their old keys and are no longer read. They expire within their TTL (LIST_TTL 300s, MODEL_TTL 3600s, RELATIONSHIP_TTL 1800s). To purge them right away, run php artisan cache:clear on Cloud after deploy; deploy.sh already runs it.

🤖 Generated with Claude Code