Skip to content

[FIX] Pre-compute schema before Ray Dataset transform to prevent empty schema error#834

Open
K2alyan wants to merge 2 commits into
Nixtla:mainfrom
K2alyan:fix/ray-exogenous-schema
Open

[FIX] Pre-compute schema before Ray Dataset transform to prevent empty schema error#834
K2alyan wants to merge 2 commits into
Nixtla:mainfrom
K2alyan:fix/ray-exogenous-schema

Conversation

@K2alyan

@K2alyan K2alyan commented Jul 4, 2026

Copy link
Copy Markdown

Problem

forecast() with X_df on a Ray distributed Dataset crashed with:

triad.collections.schema.SchemaError: Schema can't be empty

The crash happened in _distributed_forecast when X_df is not None:

df = fa.transform(df, format_df, schema="*,_in_sample:bool")
X_df = fa.transform(
    X_df,
    format_X_df,
    schema=fa.get_schema(df),           # ← Bug: empty schema on post-transform Ray Dataset
    params={..., "df_cols": fa.get_column_names(df)},
)

fa.get_schema() called on a Ray Dataset after fa.transform can return an empty schema in some Fugue/Ray version combinations, because Ray Datasets do not always propagate Arrow schema metadata through Fugue's transform pipeline the same way Spark/Dask do. Passing an empty schema to the next fa.transform call triggers SchemaError: Schema can't be empty.

Fix

Pre-compute the schema and column list before transforming df (when the original Ray Dataset has a reliable schema), then append _in_sample:bool explicitly. This is the same .copy() + .append() pattern already used in _get_schema() in this file.

_df_schema = fa.get_schema(df).copy()
_df_schema.append("_in_sample:bool")
_df_cols = fa.get_column_names(df) + ["_in_sample"]
df = fa.transform(df, format_df, schema="*,_in_sample:bool")
X_df = fa.transform(
    X_df,
    format_X_df,
    schema=_df_schema,
    params={"target_col": target_col, "df_cols": _df_cols},
)

Behavior is identical for Spark and Dask (where fa.get_schema works post-transform). For Ray it removes the crash.

Test plan

  • Removes @pytest.mark.xfail from test_forecast_x_dataframe in nixtla_tests/nixtla_client/test_ray.py
  • Run pytest nixtla_tests/nixtla_client/test_ray.py -m ray_run — all tests including the previously-failing test_forecast_x_dataframe should pass

K2alyan and others added 2 commits July 3, 2026 18:28
…y schema error

When forecasting with exogenous variables on a Ray distributed Dataset,
calling fa.get_schema() on the post-transform Dataset could return an
empty schema, causing triad.collections.schema.SchemaError: Schema
can't be empty. Fix by capturing the schema and column names from the
original Dataset before the transform and appending _in_sample:bool
explicitly, matching the pattern used in _get_schema().

Removes the xfail mark from test_forecast_x_dataframe in test_ray.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants