"Evaluate" method

[AFilipaVEL]

Hi... again 🤣

I wonder if method evaluate from utilsforecast.evaluation works in the distributed environment?

Here's my code:

models = [AutoETS()] 

fcst = StatsForecast(models=models,
                     freq='M',
                     n_jobs=1)

pred_sup = fcst.forecast(df = d_train, 
                         h=1,
                         fitted = True, 
                         level=[90])

from utilsforecast.evaluation import evaluate
from utilsforecast.losses import rmse

evaluate(pred_sup,  # output from the model
         metrics=[rmse],  # metric
         models=[AutoETS], # my fitted model 
         train_df= d_train)  # trainning spark dataframe

I get the error:

AttributeError: 'NoneType' object has no attribute 'col'
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File <command-1734731997705467>, line 1
----> 1 evaluate(pred_sup,
      2          metrics=[rmse],
      3          models=[AutoETS],
      4          train_df= d_train)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/evaluation.py:182, in evaluate(df, metrics, models, train_df, level, id_col, time_col, target_col)
    180         results_per_metric.append(result)
    181 else:
--> 182     result = metric(**kwargs)
    183     result = ufp.assign_columns(result, "metric", metric_name)
    184     results_per_metric.append(result)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/losses.py:137, in rmse(df, models, id_col, target_col)
    120 @_base_docstring
    121 def rmse(
    122     df: DataFrame,
   (...)
    125     target_col: str = "y",
    126 ) -> Union[float, np.ndarray]:
    127     """Root Mean Squared Error (RMSE)
    128 
    129     RMSE measures the relative prediction
   (...)
    135     series is possible only if they share a common scale.
    136     RMSE has a direct connection to the L2 norm."""
--> 137     res = mse(df, models, id_col, target_col)
    138     if isinstance(res, pd.DataFrame):
    139         res[models] = res[models].pow(0.5)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/losses.py:116, in mse(df, models, id_col, target_col)
    113     def gen_expr(model):
    114         return pl.col(target_col).sub(pl.col(model)).pow(2).alias(model)
--> 116     res = _pl_agg_expr(df, models, id_col, gen_expr)
    117 return res

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/losses.py:51, in _pl_agg_expr(df, models, id_col, gen_expr)
     45 def _pl_agg_expr(
     46     df: pl_DataFrame,
     47     models: Union[List[str], List[Tuple[str, str]]],
     48     id_col: str,
     49     gen_expr: Callable[[Union[str, Tuple[str, str]]], "pl.Expr"],
     50 ) -> pl_DataFrame:
---> 51     exprs = [gen_expr(model) for model in models]
     52     df = df.select([id_col, *exprs])
     53     return ufp.group_by(df, id_col, maintain_order=True).mean()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/losses.py:51, in <listcomp>(.0)
     45 def _pl_agg_expr(
     46     df: pl_DataFrame,
     47     models: Union[List[str], List[Tuple[str, str]]],
     48     id_col: str,
     49     gen_expr: Callable[[Union[str, Tuple[str, str]]], "pl.Expr"],
     50 ) -> pl_DataFrame:
---> 51     exprs = [gen_expr(model) for model in models]
     52     df = df.select([id_col, *exprs])
     53     return ufp.group_by(df, id_col, maintain_order=True).mean()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-05cfa563-7603-4e3a-a431-5ec30789c6f1/lib/python3.10/site-packages/utilsforecast/losses.py:114, in mse.<locals>.gen_expr(model)
    113 def gen_expr(model):
--> 114     return pl.col(target_col).sub(pl.col(model)).pow(2).alias(model)

AttributeError: 'NoneType' object has no attribute 'col'

Thanks again!

[jmoralez]

Hey. It doesn't, but I think we can add support for them. Can you please open an issue in https://github.com/Nixtla/utilsforecast/issues/new?

[AFilipaVEL]

I will, thank you!

[AFilipaVEL]

Hi there!

Any updates on this?

Thanks!

[jmoralez]

This works since 0.2.5