Firstly, I'm sure it wasn't intentional. I reopen this issue and just want to discuss this question clearly. Maybe I'm wrong, but after I read the code, especially encode() detailed, I still cannot understand the following words:
For example, we assume t is the first timestamp in the test set. For the first sample in the test set, the original input is [t-padding, t]. However, your input is [t, t], which feeds only one timestamp as input, resulting in poor performance and biased distribution.
First, the whole series is fed into encode() in the eval_forecasting(). Does encode() know where the test part starts? In fact, there is no related variable t in the eval_forecasting() to indicate encode() the series is the training set or test set. I print the sliced representation of train/valid/test parts and I confirm that the encoded train/valid/test sets are overlapped. (I hope you can observe it and then reply to me.) When you encode the whole series to representation through the sliding window and sliding_length is 1. If encode() does not know where the test set starts, then the last sample of validation would cover the first test sample. But, I think this bug does not belong to encode().
I use Electricity as an example to explain this bug. I performed the univariate forecasting task and collected the shape of some variables. The shape of the original series(variable data in the eval_forecasting()) is (26304, 1). The shape of all_repr which is the encoded series by encode() is (26304, 128). The shape of the representation of train/valid/test sets are (15782, 128), (5261, 128), (5261, 128). If there are no overlapped samples among the train/valid/test sets, the sum of their samples should not equal the total samples.
Of course, if you skip the head-overlapped samples, you can avoid this leakage. The generate_pred_samples() is used to discard header samples, this problem may be solved by setting reasonable parameters. But, for the valid/test sets, the parameter of the drop is set as ZERO. That means that the valid set and test set would continue to the previous section.
for pred_len in pred_lens:
train_features, train_labels = generate_pred_samples(train_repr, train_data, pred_len, drop=padding)
valid_features, valid_labels = generate_pred_samples(valid_repr, valid_data, pred_len)
test_features, test_labels = generate_pred_samples(test_repr, test_data, pred_len)
The default value of drop in the generate_pred_samples() is 0. Is the [t-padding, t] you mentioned? Are you forget to set the drop?
At last, I hope you can provide detailed evidence to prove you avoid this leakage, rather than simply asking me to read the code because I still think the bug exists after reading it again.
Firstly, I'm sure it wasn't intentional. I reopen this issue and just want to discuss this question clearly. Maybe I'm wrong, but after I read the code, especially encode() detailed, I still cannot understand the following words:
First, the whole series is fed into encode() in the eval_forecasting(). Does encode() know where the test part starts? In fact, there is no related variable t in the eval_forecasting() to indicate encode() the series is the training set or test set. I print the sliced representation of train/valid/test parts and I confirm that the encoded train/valid/test sets are overlapped. (I hope you can observe it and then reply to me.) When you encode the whole series to representation through the sliding window and sliding_length is 1. If encode() does not know where the test set starts, then the last sample of validation would cover the first test sample. But, I think this bug does not belong to encode().
I use Electricity as an example to explain this bug. I performed the univariate forecasting task and collected the shape of some variables. The shape of the original series(variable data in the eval_forecasting()) is (26304, 1). The shape of all_repr which is the encoded series by encode() is (26304, 128). The shape of the representation of train/valid/test sets are (15782, 128), (5261, 128), (5261, 128). If there are no overlapped samples among the train/valid/test sets, the sum of their samples should not equal the total samples.
Of course, if you skip the head-overlapped samples, you can avoid this leakage. The generate_pred_samples() is used to discard header samples, this problem may be solved by setting reasonable parameters. But, for the valid/test sets, the parameter of the drop is set as ZERO. That means that the valid set and test set would continue to the previous section.
The default value of drop in the generate_pred_samples() is 0. Is the [t-padding, t] you mentioned? Are you forget to set the drop?
At last, I hope you can provide detailed evidence to prove you avoid this leakage, rather than simply asking me to read the code because I still think the bug exists after reading it again.