Skip to content

Reduce storage by changing cumulative_probability and event_value from float8 to float4 #2331

Description

@Ahmad-Wahid

Context

The TimedBelief table (see e62ac5f519d7_create_table_for_timed_beliefs.py) defines both cumulative_probability and event_value as sa.Float(), which SQLAlchemy/Postgres maps to float8 (double precision, 8 bytes per value).

This is our largest and fastest-growing table, so column width has a direct effect on disk/storage costs and I/O.

Proposal

Change cumulative_probability and event_value from float8 (double precision) to float4 (single precision), via a new Alembic migration (ALTER COLUMN ... TYPE real).

Rough estimate: this could reduce the size of the timed_beliefs table by roughly 15-20%, since these are two of the core per-row numeric columns.

Things to check before implementing

  • Precision loss: float4 has ~7 significant decimal digits vs ~15-17 for float8. Need to confirm this is acceptable for:
    • event_value (sensor readings/schedules/forecasts - magnitudes and required precision vary by sensor/unit)
    • cumulative_probability (bounded in [0, 1], where lower precision may be more acceptable)
  • Migration cost/downtime: ALTER COLUMN TYPE on a large table rewrites it; needs a migration strategy (e.g. batched backfill, or accept the lock/downtime) given this is likely the largest table in most deployments.
  • Any downstream code that assumes float64 precision (e.g. numpy/pandas dtype expectations when reading beliefs into a BeliefsDataFrame, or serialization/rounding logic in the API).
  • Whether this should be an opt-in migration or bundled with other maintenance (partitioning, etc.).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions