Investigate right-truncation indexing across parameter estimates, cfa-stf-data, and PyRenew

[damonbayer]

Describe the improvement that needs to be made

Investigate and document the end-to-end indexing convention for the NSSP right-truncation PMF as it moves through:

1. the parameter-estimation process (currently referred to as cfa-parameter-estimates, with estimates published in datacat.public.stf.param_estimates),
2. cfa-stf-data,
3. PyRenew, and
4. this repository's PyRenew pipeline.

There is evidence of inconsistent zero- versus one-based indexing:

• PyRenew documents reporting_delay_pmf[i] as the probability of delay i. It defines right_truncation_offset = 0 to mean that the data was pulled on the same day as the last observation, so only delay-0 reports have arrived. Under that contract, the calendar calculation is (data_pull_date - last_observation_date).days.
• In cfa-stf-data, exclude_tail_auto labels the first cumulative PMF entry with offset_days = 1, labels the latest included observation with offset_days = 1, and joins on that value. This operationally maps the first PMF/CDF entry to one offset_day, but it does not establish whether the source PMF is intentionally supported on days 1, 2, ... or whether this consumer is itself shifted.
• This repository passes the PMF returned by get_nnh_right_truncation_pmf to PyRenew without reindexing, but currently calculates right_truncation_offset as (report_date - overall_last_training_date).days - 1. The comment says the first PMF entry corresponds to report_date - 1; that assumption needs to be verified against the producer.

The investigation should trace one concrete PMF element and one concrete observation date through every layer. For example, for a report/data-pull date of 2026-09-09 and an overall last observation date of 2026-09-05, determine whether PyRenew should receive an offset of 3 or 4, what event PMF element 0 represents, and which CDF value is applied to each observation date.

Questions to resolve:

• What random variable is estimated upstream: reporting delay in completed calendar days, number of additional reporting opportunities, or another quantity?
• Does the serialized PMF include mass for delay 0? If not, is zero-day mass impossible, omitted, conditioned away, or represented elsewhere?
• Are the PMF bins ordered as delays 0, 1, 2, ... or 1, 2, 3, ...?
• Does cfa-stf-data preserve the PMF unchanged when reading datacat.public.stf.param_estimates?
• Is the + 1 mapping in exclude_tail_auto intentional and correct?
• Does PyRenew require a zero-delay element to be prepended or another adapter transformation before receiving this PMF?
• Should the routine-forecasting offset be the raw date difference, or is the existing - 1 a deliberate adapter for the upstream PMF?
• Confirm that the offset should continue to use the overall last training date across all included data sources, including runs containing both NSSP and NHSN data.

Provide links to references to methods or data sources

• PyRenew right-truncation tutorial at the reviewed commit
• PyRenew right-truncation implementation at the reviewed commit
• cfa-stf-data PMF-to-offset mapping
• cfa-stf-data version used here
• ForecastRun.right_truncation_offset
• cfa-stf-data PMF retrieval in this repository
• right_truncation_offset serialization for model fitting

As part of this issue, identify and link the authoritative source/version in the parameter-estimation system that creates the right_truncation rows published to datacat.public.stf.param_estimates.

Describe the changes expected to the model's outputs

This investigation may result in no output change if the existing shift is confirmed. If the conventions are mismatched, correcting them will shift the proportion-reported CDF applied to recent NSSP observations by one day. That can change the likelihood contribution and inferred incidence near the right edge, with downstream changes to nowcasts and forecasts.

Before changing production behavior, compare the current and corrected mappings and quantify their effect on representative locations and diseases.

Suggest new tests that will need to be implemented

• A producer-level test or fixture that states the event represented by PMF elements 0 and 1.
• A cfa-stf-data test using a non-symmetric synthetic PMF, such as [0.2, 0.3, 0.5], that asserts the exact CDF assigned to the latest date and preceding dates.
• A PyRenew unit test demonstrating offsets 0, 1, and greater than 1 with the same non-symmetric PMF.
• An end-to-end contract test that follows a dated synthetic observation and PMF from the parameter representation through cfa-stf-data into PyRenew.
• Routine-forecasting tests for same-day, previous-day, and multi-day gaps.
• A routine-forecasting test with multiple data sources showing that the calculation uses the overall last training date.

Definition of done

• The support and interpretation of every PMF index are documented at the producer and consumer boundaries.
• A date-to-index table for at least one concrete example agrees across all four layers.
• The responsible repositories either adopt one shared convention or perform explicit, documented conversion at their boundaries.
• Tests fail under a one-day shift and pass only for the agreed mapping.
• Any production change includes an impact comparison for recent nowcasts/forecasts.