IndexRange and reindexByRange

[tnielens]

Hello here,

I'm experimenting a bit with a generic IndexRange[S, T] and a Stepper[S, T]. The end goal is introducing a new operation on the series and frames along this type signature:

def reindexByRange[S](from: T, to: T, step: S)(implicit s: Stepper[S, T]): Series[T] = ???

An instance of Stepper would be provided for java.time or joda types, for example Stepper[Duration, Instant] or Stepper[Duration, LocalDateTime] etc.
This is an attempt at providing equivalent capabilities to resample from pandas.

The work is in an early stage. But I'm open to early feedback on this and see if there is interest in adding this to saddle.

[pityka]

I am okay with this but I don't have much experience on time series. So I will ask you to put enough unit tests.
Also, I am not sure whether saddle should carry on the joda time dependency, probably the first thing would be to get rid of that and migrate to java.time.

[tnielens]

java.time and and joda typeclass Stepper could be provided independently and be imported with

import org.saddle.index.Stepper
import org.saddle.jtime._
Stepper[Duration, LocalDateTime]

[tnielens]

One motivation for providing time range features is aligning time series data for further processing. One example would be processing ts data sampled roughly secondly but nevertheless not aligning. One wants to align/resample/reindex each series to sum them.

import java.time.LocalTime
import org.saddle._

val s1 = Series(LocalTime("00:00:00.112") -> 1.2,  LocalTime("00:00:01.122") -> 1.5,  LocalTime("00:00:02.117") -> 1.3, ...)
val s2 = Series(LocalTime("00:00:00.113") -> 0.9,  LocalTime("00:00:01.125") -> 1.2,  LocalTime("00:00:02.116") -> 1.1, ...)

val sum = ???

In saddle, what is the idiomatic way of resampling both series and adding them?

Here is an option in python with pandas

import pandas as pd
range = pd.date_range(pd.Timestamp("00:00:00"), pd.Timestamp("00:00:02"))
s1 = pd.Series({pd.Timestamp("00:00:00.112"): -> 1.2,  pd.Timestamp("00:00:01.122"): 1.5,  pd.Timestamp("00:00:02.117"): 1.3}, ...)
s2 = pd.Series(pd.Timestamp("00:00:00.113") -> 0.9,  pd.Timestamp("00:00:01.125"): 1.2,  pd.Timestamp("00:00:02.116"): 1.1}, ...)


val sum = (
  pd.concat([s1, s2], axis='columns') # concat returns a DataFrame
    .resample("1s")
    .ffill() # forward fill, fill NA values with present ones forward
    .sum(axis='columns'))
    
# or 
range = pd.date_range(pd.Timestamp("00:00:00"), pd.Timestamp("00:00:02"), freq="1s")
sum = (
  pd.concat([s1, s2], axis='columns')
    .reindex(range, method='ffill') # similar to .resample().ffill()
    .sum(axis='columns'))

I see these tracks for extending saddle in this area:

• More ways for filling missing data, forward/backward filling is quite central, but more sophisticated options could be discussed such as linear interpolation.
• How to create and manipulate ranges, and more specifically time ranges.
• Adding missing data filling in reindexing. In pandas, .reindex(.., method="...") lets you specify what how to deal with keys not present in the receiver series/frame.

Let me know if I'm missing on saddle features that cover parts of the discussion here.

[pityka]

Yes I am not aware that saddle has such. I don't really now what is in the saddle-time module because I just separated it from the rest, put it into the build and never looked back on it (I mentioned it in the docs that it is unmaintained).

If you need this I will merge it into saddle-core with the joda type classes being in saddle-time. I also welcome any interpolation and filling routines. I am happy if we extend the functionality of saddle. Only, the changes should come with a test and do not break the current binary compatibility.

Regarding the reindex, in saddle you can do reindex + fillNA. But yes fillNA could be improved by e.g. taking the index value.