Backfill WAS_RUNNING events for running apps, tasks, and service instances

Seed a synthetic WAS_RUNNING usage event for every currently-running app
process, a TASK_WAS_RUNNING event for every currently-running task, and a
WAS_RUNNING event for every existing service instance, so billing consumers
can bootstrap a complete baseline even after the original
STARTED/TASK_STARTED/CREATED events have been pruned.

The backfill is a batched, idempotent VCAP::WasRunningBackfill helper invoked
from thin no_transaction migrations (mirrors the bigint-migration pattern):
each batch keysets over the started processes / running tasks / service
instances by id and runs in its own READ COMMITTED transaction, so no
statement risks the migration timeout and MySQL's INSERT..SELECT takes no
shared next-key locks on the scanned source rows while the API serves
traffic. The app backfill scopes the package/droplet aggregates to each
batch's apps (index-backed) to keep package_state fidelity without scanning
the whole tables, and COALESCEs nullable legacy process/app/task columns so a
single NULL row cannot abort a deploy.

Because the API stays live during migrations, a batch can race a concurrent
stop/delete and insert a baseline row that no later ending event would ever
prune; a post-seed sweep removes WAS_RUNNING/TASK_WAS_RUNNING rows whose
resource is no longer running/present.

A skip_was_running_backfill config flag lets operators opt out (checked by the
migrations, not the helper, since the migrations are recorded as applied
either way); 'rake db:was_running_backfill' re-runs the seeding later for
operators who skipped. Rollback deletes are batched too.

Document the WAS_RUNNING/TASK_WAS_RUNNING states and their created_at
(migration-time) semantics on the V3 resources, and list the new states in
the legacy V2 usage-event docs because V2 reads the same event rows.

Author: joyvuu-dave

db/migrations/20260601120100_seed_was_running_app_usage_events.rb

@@ -0,0 +1,18 @@
+require 'database/was_running_backfill'
+
+Sequel.migration do
+  no_transaction # backfill manages its own per-batch transactions
+
+  up do
+    logger = Steno.logger('cc.backfill.was_running')
+    if VCAP::WasRunningBackfill.skip?
+      VCAP::WasRunningBackfill.log_skip(logger, 'app')
+    else
+      VCAP::WasRunningBackfill.seed_app_usage_events(self, logger)
+    end
+  end
+
+  down do
+    VCAP::WasRunningBackfill.delete_app_usage_events(self)
+  end
+end

db/migrations/20260601120200_seed_was_running_service_usage_events.rb

@@ -0,0 +1,18 @@
+require 'database/was_running_backfill'
+
+Sequel.migration do
+  no_transaction # backfill manages its own per-batch transactions
+
+  up do
+    logger = Steno.logger('cc.backfill.was_running')
+    if VCAP::WasRunningBackfill.skip?
+      VCAP::WasRunningBackfill.log_skip(logger, 'service')
+    else
+      VCAP::WasRunningBackfill.seed_service_usage_events(self, logger)
+    end
+  end
+
+  down do
+    VCAP::WasRunningBackfill.delete_service_usage_events(self)
+  end
+end

db/migrations/20260601120300_seed_was_running_task_usage_events.rb

@@ -0,0 +1,18 @@
+require 'database/was_running_backfill'
+
+Sequel.migration do
+  no_transaction # backfill manages its own per-batch transactions
+
+  up do
+    logger = Steno.logger('cc.backfill.was_running')
+    if VCAP::WasRunningBackfill.skip?
+      VCAP::WasRunningBackfill.log_skip(logger, 'task')
+    else
+      VCAP::WasRunningBackfill.seed_task_usage_events(self, logger)
+    end
+  end
+
+  down do
+    VCAP::WasRunningBackfill.delete_task_usage_events(self)
+  end
+end

docs/v2/app_usage_events/list_all_app_usage_events.html

@@ -631,9 +631,11 @@ <h4>Body</h4>
                 <ul class="valid_values">
                     <li>STARTED</li>
                     <li>STOPPED</li>
+                    <li>WAS_RUNNING</li>
                     <li>BUILDPACK_SET</li>
                     <li>TASK_STARTED</li>
                     <li>TASK_STOPPED</li>
+                    <li>TASK_WAS_RUNNING</li>
                 </ul>
               </td>
               <td>

docs/v2/service_usage_events/list_service_usage_events.html

@@ -290,6 +290,7 @@ <h4>Body</h4>
                     <li>CREATED</li>
                     <li>DELETED</li>
                     <li>UPDATED</li>
+                    <li>WAS_RUNNING</li>
                 </ul>
               </td>
               <td>

docs/v3/source/includes/resources/app_usage_events/_object.md.erb

@@ -30,3 +30,14 @@ Name | Type | Description
 **instance_count.current** | _integer_ or `null` | Current instance count of the app that this event pertains to, if applicable
 **instance_count.previous** | _integer_ or `null` | Previous instance count of the app that this event pertains to, if applicable
 **links** | [_links object_](#links) | Links to related resources
+
+#### WAS_RUNNING and TASK_WAS_RUNNING events
+
+`WAS_RUNNING` and `TASK_WAS_RUNNING` are synthetic values for `state.current` recorded once per running process (`WAS_RUNNING`) and once per running task (`TASK_WAS_RUNNING`) by a one-time data migration when the keep-running cleanup feature was introduced. They mark every process and task that was already running at the time of the upgrade so that billing consumers can bootstrap from a complete baseline even if the original `STARTED`/`TASK_STARTED` events have been pruned.
+
+**Consumer interpretation** (read `WAS_RUNNING`/`STARTED` as `TASK_WAS_RUNNING`/`TASK_STARTED` for task events, which are keyed by `task.guid`):
+
+* If you have not previously recorded a `STARTED` event for this resource, treat `WAS_RUNNING` as equivalent to `STARTED`.
+* If you have already recorded `STARTED` (or an earlier `WAS_RUNNING`) for this resource, treat as a redundant baseline confirmation and ignore.
+* `created_at` reflects when the backfill migration ran, **not** when the app or task actually started. Treat `WAS_RUNNING` as a baseline marker that the resource was already running as of that timestamp, not as the true start of the running interval.
+* `state.previous` on a `WAS_RUNNING` event is always `null`. Subsequent real events for the same resource will continue to report their actual prior process state in `state.previous` (typically `STARTED`). If you perform chain validation, treat `WAS_RUNNING` as equivalent to `STARTED` for the purpose of validating the next event's `state.previous`.

docs/v3/source/includes/resources/service_usage_events/_object.md.erb

@@ -26,3 +26,13 @@ Name | Type | Description
 **service_broker.guid** | _string_ or `null` | Unique identifier of the service broker that this event pertains to, if applicable
 **service_broker.name** | _string_ or `null` | Name of the service broker that this event pertains to, if applicable
 **links** | [_links object_](#links) | Links to related resources
+
+#### WAS_RUNNING events
+
+`WAS_RUNNING` is a synthetic value for `state` recorded once per existing service instance by a one-time data migration when the keep-running cleanup feature was introduced. It marks every service instance that existed at the time of the upgrade so that billing consumers can bootstrap from a complete baseline of service instances even if the original `CREATED` events have been pruned.
+
+**Consumer interpretation:**
+
+* If you have not previously recorded a `CREATED` event for this service instance, treat `WAS_RUNNING` as equivalent to `CREATED`.
+* If you have already recorded `CREATED` (or an earlier `WAS_RUNNING`) for this instance, treat as a redundant baseline confirmation and ignore.
+* `created_at` reflects when the backfill migration ran, **not** when the service instance was created. Treat `WAS_RUNNING` as a baseline marker that the instance already existed as of that timestamp.

lib/cloud_controller/config_schemas/api_schema.rb

@@ -109,6 +109,7 @@ class ApiSchema < VCAP::Config
           optional(:migration_psql_concurrent_statement_timeout_in_seconds) => Integer,
           optional(:migration_psql_worker_memory_kb) => Integer,
           optional(:skip_bigint_id_migration) => bool,
+          optional(:skip_was_running_backfill) => bool,
           db: {
             optional(:database) => Hash, # db connection hash for sequel
             max_connections: Integer, # max connections in the connection pool

lib/cloud_controller/config_schemas/migrate_schema.rb

@@ -10,6 +10,7 @@ class MigrateSchema < VCAP::Config
           optional(:migration_psql_concurrent_statement_timeout_in_seconds) => Integer,
           optional(:migration_psql_worker_memory_kb) => Integer,
           optional(:skip_bigint_id_migration) => bool,
+          optional(:skip_was_running_backfill) => bool,
 
           db: {
             optional(:database) => Hash, # db connection hash for sequel