Auto-PyTorch
Auto-PyTorch copied to clipboard
automl
Extremely long time-series data leads causes evaluation failure
NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com
Your issue may already be reported! Also, please search on the issue tracker before creating one.
-
I'm submitting a ...
- [x] bug report
Issue Description
- When Issue Happens
- Steps To Reproduce
- Download Covid Death Daily Dataset from https://zenodo.org/record/4656009/files/covid_deaths_dataset.zip?download=1 as mentioned in the paper. This dataset
- Convert corresponding .tsf file into the same pandas data frame format as in https://github.com/automl/Auto-PyTorch/blob/master/examples/20_basics/example_time_series_forecasting.py.
- To prove that it is the length of the time series that matters, the first 30 time series data points are picked to form a shorter time series sequence, which did not cause any error.
Expected Behavior
Evaluate the performance normally of the initial designs on Covid Death Daily Dataset.
Current Behavior
An error "ValueError: Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0)" occurs.
Possible Solution
Maybe there's a default maximum length of time-series sequence I didn't notice.
Your Code
# will be added soon.
Error message
[INFO] [2022-08-07 11:07:57,681:Client-TAE] Starting to evaluate configuration 1
[DEBUG] [2022-08-07 11:07:57,685:Client-TAE] Search space updates for 2: <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7ff3b2664f10>
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Restricting your function to 499 seconds wall time.
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Allowing a grace period of 0 seconds.
[DEBUG] [2022-08-07 11:07:57,685:Client-pynisher] Function called with argument: (), {'queue': <multiprocessing.queues.Queue object at 0x7ff3b2301bb0>, 'config': Configuration(values={
'data_loader:backcast': False,
'data_loader:batch_size': 32,
'data_loader:num_batches_per_epoch': 50,
'data_loader:sample_strategy': 'SeqUniform',
'data_loader:transform_time_features': False,
'data_loader:window_size': 2,
'loss:DistributionLoss:aggregation': 'median',
'loss:DistributionLoss:dist_cls': 'studentT',
'loss:DistributionLoss:forecast_strategy': 'sample',
'loss:DistributionLoss:num_samples': 100,
'loss:__choice__': 'DistributionLoss',
'lr_scheduler:ReduceLROnPlateau:factor': 0.5,
'lr_scheduler:ReduceLROnPlateau:mode': 'max',
'lr_scheduler:ReduceLROnPlateau:patience': 10,
'lr_scheduler:__choice__': 'ReduceLROnPlateau',
'network_backbone:__choice__': 'flat_encoder',
'network_backbone:flat_encoder:MLPDecoder:has_local_layer': True,
'network_backbone:flat_encoder:MLPDecoder:num_layers': 0,
'network_backbone:flat_encoder:MLPDecoder:units_local_layer': 40,
'network_backbone:flat_encoder:MLPEncoder:activation': 'relu',
'network_backbone:flat_encoder:MLPEncoder:normalization': 'NoNorm',
'network_backbone:flat_encoder:MLPEncoder:num_groups': 1,
'network_backbone:flat_encoder:MLPEncoder:num_units_1': 40,
'network_backbone:flat_encoder:MLPEncoder:use_dropout': False,
'network_backbone:flat_encoder:__choice__': 'MLPEncoder',
'network_embedding:__choice__': 'NoEmbedding',
'network_init:XavierInit:bias_strategy': 'Normal',
'network_init:__choice__': 'XavierInit',
'optimizer:AdamOptimizer:beta1': 0.9,
'optimizer:AdamOptimizer:beta2': 0.999,
'optimizer:AdamOptimizer:lr': 0.001,
'optimizer:AdamOptimizer:weight_decay': 1e-08,
'optimizer:__choice__': 'AdamOptimizer',
'target_scaler:scaling_mode': 'mean_abs',
'trainer:__choice__': 'ForecastingStandardTrainer',
})
, 'backend': <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7ff3b24c1610>, 'metric': mean_MASE_forecasting, 'seed': 1, 'num_run': 2, 'output_y_hat_optimization': True, 'include': {}, 'exclude': {}, 'disable_file_output': [], 'instance': '{"task_id": "1fcff3ff-15fe-11ed-9173-1dfac5c47e62"}', 'init_params': {'instance': '{"task_id": "1fcff3ff-15fe-11ed-9173-1dfac5c47e62"}'}, 'budget': 5.555555555555555, 'budget_type': 'epochs', 'pipeline_config': {'device': 'cuda', 'budget_type': 'epochs', 'epochs': 50, 'runtime': 3600, 'torch_num_threads': 1, 'early_stopping': 20, 'use_tensorboard_logger': False, 'metrics_during_training': True, 'optimize_metric': 'mean_MASE_forecasting'}, 'logger_port': 61103, 'all_supported_metrics': True, 'search_space_updates': <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7ff3b2664f10>}
[DEBUG] [2022-08-07 11:08:00,319:Client-pynisher] Redirecting output of the function to files. Access them via the stdout and stderr attributes of the wrapped function.
[DEBUG] [2022-08-07 11:08:00,320:Client-pynisher] call function
[DEBUG] [2022-08-07 11:08:00,674:Client-TimeSeriesForecastingTrainEvaluator(1)] Fit dictionary in Abstract evaluator: dataset_properties: {'input_shape': (94, 0), 'categorical_features': [], 'numerical_columns': [], 'feature_shapes': {}, 'n_prediction_steps': 3, 'numerical_features': [], 'output_shape': [3, 1], 'feature_names': (), 'categories': [], 'freq': '1Y', 'sequence_lengths_train': array([94]), 'is_small_preprocess': True, 'task_type': 'time_series_forecasting', 'time_feature_names': ('time_feature_Constant',), 'known_future_features': (), 'categorical_columns': [], 'static_features': (), 'output_type': 'continuous', 'issparse': False, 'sp': 1, 'time_feature_transform': [Constant()], 'uni_variant': True, 'static_features_shape': 0, 'future_feature_shapes': (3, 0), 'targets_have_missing_values': False, 'encoder_can_be_auto_regressive': True, 'features_have_missing_values': False}
additional_metrics: ['mean_MASE_forecasting', 'mean_MASE_forecasting', 'median_MASE_forecasting', 'mean_MAE_forecasting', 'median_MAE_forecasting', 'mean_MAPE_forecasting', 'median_MAPE_forecasting', 'mean_MSE_forecasting', 'median_MSE_forecasting']
X_train: None
y_train: 0
0 1.000000e-26
0 1.000000e-26
0 1.000000e-26
0 1.000000e-26
0 1.000000e-26
.. ...
0 4.200000e+01
0 4.300000e+01
0 4.700000e+01
0 5.000000e+01
0 5.700000e+01
[97 rows x 1 columns]
X_test: None
y_test: [[58.]
[60.]
[64.]]
backend: <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7f2c5a4bbca0>
logger_port: 61103
optimize_metric: mean_MASE_forecasting
device: cuda
budget_type: epochs
epochs: 5.555555555555555
torch_num_threads: 1
early_stopping: 20
use_tensorboard_logger: False
metrics_during_training: True
[DEBUG] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f2c5a4d00a0>
[DEBUG] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f2c5a4d00a0>
[INFO] [2022-08-07 11:08:00,675:Client-TimeSeriesForecastingTrainEvaluator(1)] Starting fit 0
[DEBUG] [2022-08-07 11:08:01,991:Client-TimeSeriesForecastingTrainEvaluator(1)] /home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_initializer/XavierInit.py:27: UserWarning:nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
[DEBUG] [2022-08-07 11:08:03,129:Client-pynisher] function returned properly: (None, 0)
[DEBUG] [2022-08-07 11:08:03,129:Client-pynisher] return value: (None, 0)
[DEBUG] [2022-08-07 11:08:03,700:Client-TAE] Finish function evaluation 2.
Status: StatusType.CRASHED, Cost: 2147483647.0, Runtime: 5.444417238235474,
Additional information:
traceback: Traceback (most recent call last):
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/tae.py", line 61, in fit_predict_try_except_decorator
ta(queue=queue, **kwargs)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/time_series_forecasting_train_evaluator.py", line 558, in forecasting_eval_train_function
evaluator.fit_predict_and_loss()
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/time_series_forecasting_train_evaluator.py", line 176, in fit_predict_and_loss
y_train_pred, y_opt_pred, y_valid_pred, y_test_pred = self._fit_and_predict(pipeline, split_id,
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/train_evaluator.py", line 364, in _fit_and_predict
fit_and_suppress_warnings(self.logger, pipeline, X, y)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/evaluation/abstract_evaluator.py", line 338, in fit_and_suppress_warnings
pipeline.fit(X, y)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/base_pipeline.py", line 158, in fit
self.fit_estimator(X, y, **fit_params)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/base_pipeline.py", line 177, in fit_estimator
self._final_estimator.fit(X, y, **fit_params)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 211, in fit
self._fit(
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/__init__.py", line 310, in _fit
train_loss, train_metrics = self.choice.train_epoch(
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 106, in train_epoch
loss, outputs = self.train_step(data, targets)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/training/trainer/forecasting_trainer/forecasting_base_trainer.py", line 206, in train_step
outputs = self.model(past_targets=past_target,
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network/forecasting_architecture.py", line 602, in forward
output = self.head(decoder_output)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_head/forecasting_network_head/distribution.py", line 100, in forward
return self.dist_cls(*self.domain_map(*params_unbounded))
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/studentT.py", line 50, in __init__
self._chi2 = Chi2(self.df)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/chi2.py", line 22, in __init__
super(Chi2, self).__init__(0.5 * df, 0.5, validate_args=validate_args)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/gamma.py", line 52, in __init__
super(Gamma, self).__init__(batch_shape, validate_args=validate_args)
File "/home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/torch/distributions/distribution.py", line 55, in __init__
raise ValueError(
ValueError: Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values:
tensor([[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]],
[[nan],
[nan],
[nan]]], device='cuda:0', grad_fn=<MulBackward0>)
error: ValueError("Expected parameter df (Tensor of shape (32, 3, 1)) of distribution Chi2() to satisfy the constraint GreaterThan(lower_bound=0.0), but found invalid values:\ntensor([[[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]],\n\n [[nan],\n [nan],\n [nan]]], device='cuda:0', grad_fn=<MulBackward0>)")
Further information are recorded in following log file: nan-error.log
Your Local environment
- Operating System, version Ubuntu 20.04
- Python, version Python 3.8
- Outputs of
pip freezeorconda listpytorch=1.12+cu116
Supplementary Information
Covid Death dataset is uni-variate dataset, to give more evidence about what's really happening behind this error, the following
are the print-out dataframe of y_train and y_test. (In order to form a complete control group following the format of example code, I further modify the data frequency from '1D' to '1Y' to be identical with Longley dataset just for debug purpose.)
nan-error-y-info.txt
Hi, does this NAN errors happens for all the configurations or just this single configuration? Sometimes this is simply the result of an unsuited hyperparameter configuration that result in an unstable training.
Hi! I've just done several experiments, it shows that among the 9 initial configurations, the first 6 are crashed and the last 3 survived. The result is stable over repetitive runs. I don't know if this is normal.
The full log file and run history json file are as below: AutoPyTorch_1c0e9c2e-1719-11ed-8c29-312016f08e28_1.log runhistory.json.txt
The problem is a bit complicated; this dataset only contains integral values. Therefore, it is considered a 'multiclass' dataset and all its 0 values are transformed into a very small value. Thereby, the target values are not properly scaled.
As a temporary solution, you could let this judgment always return False: https://github.com/automl/Auto-PyTorch/blob/master/autoPyTorch/data/tabular_target_validator.py#L184. Alternatively, you could set the argument normalize_yas True within api.search
Hi!
The error is gone after setting normalize_y as True, but as I'm trying to reproduce the time-series forecasting experiment result in the paper, and current implementation of auto-pytorch is not capable of time-series classification, I think it might not be the case, maybe there are connections I lose track of.
After several experiment, I found that the MASE loss value in log file sometimes appears to be negative, which is really confusing. The following log file records several such training sessions.
latest.log
a snapshot of one of the training sessions in the log file above.
[INFO] [2022-08-09 22:23:55,570:Client-TAE] Starting to evaluate configuration 1
[DEBUG] [2022-08-09 22:23:55,573:Client-TAE] Search space updates for 2: <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f4ab691ebe0>
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Restricting your function to 499 seconds wall time.
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Allowing a grace period of 0 seconds.
[DEBUG] [2022-08-09 22:23:55,573:Client-pynisher] Function called with argument: (), {'queue': <multiprocessing.queues.Queue object at 0x7f4ab6878e20>, 'config': Configuration(values={
'data_loader:backcast': False,
'data_loader:batch_size': 32,
'data_loader:num_batches_per_epoch': 50,
'data_loader:sample_strategy': 'SeqUniform',
'data_loader:transform_time_features': False,
'data_loader:window_size': 9,
'loss:DistributionLoss:aggregation': 'median',
'loss:DistributionLoss:dist_cls': 'studentT',
'loss:DistributionLoss:forecast_strategy': 'sample',
'loss:DistributionLoss:num_samples': 100,
'loss:__choice__': 'DistributionLoss',
'lr_scheduler:ReduceLROnPlateau:factor': 0.5,
'lr_scheduler:ReduceLROnPlateau:mode': 'max',
'lr_scheduler:ReduceLROnPlateau:patience': 10,
'lr_scheduler:__choice__': 'ReduceLROnPlateau',
'network_backbone:__choice__': 'flat_encoder',
'network_backbone:flat_encoder:MLPDecoder:has_local_layer': True,
'network_backbone:flat_encoder:MLPDecoder:num_layers': 0,
'network_backbone:flat_encoder:MLPDecoder:units_local_layer': 40,
'network_backbone:flat_encoder:MLPEncoder:activation': 'relu',
'network_backbone:flat_encoder:MLPEncoder:normalization': 'NoNorm',
'network_backbone:flat_encoder:MLPEncoder:num_groups': 1,
'network_backbone:flat_encoder:MLPEncoder:num_units_1': 40,
'network_backbone:flat_encoder:MLPEncoder:use_dropout': False,
'network_backbone:flat_encoder:__choice__': 'MLPEncoder',
'network_embedding:__choice__': 'NoEmbedding',
'network_init:XavierInit:bias_strategy': 'Normal',
'network_init:__choice__': 'XavierInit',
'optimizer:AdamOptimizer:beta1': 0.9,
'optimizer:AdamOptimizer:beta2': 0.999,
'optimizer:AdamOptimizer:lr': 0.001,
'optimizer:AdamOptimizer:weight_decay': 1e-08,
'optimizer:__choice__': 'AdamOptimizer',
'target_scaler:scaling_mode': 'mean_abs',
'trainer:__choice__': 'ForecastingStandardTrainer',
})
, 'backend': <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7f4ab6766850>, 'metric': mean_MASE_forecasting, 'seed': 1, 'num_run': 2, 'output_y_hat_optimization': True, 'include': {}, 'exclude': {}, 'disable_file_output': [], 'instance': '{"task_id": "e2d24e1c-17ee-11ed-9a73-2f4d5d9d434e"}', 'init_params': {'instance': '{"task_id": "e2d24e1c-17ee-11ed-9a73-2f4d5d9d434e"}'}, 'budget': 5.555555555555555, 'budget_type': 'epochs', 'pipeline_config': {'device': 'cuda', 'budget_type': 'epochs', 'epochs': 50, 'runtime': 3600, 'torch_num_threads': 1, 'early_stopping': 20, 'use_tensorboard_logger': False, 'metrics_during_training': True, 'optimize_metric': 'mean_MASE_forecasting'}, 'logger_port': 47969, 'all_supported_metrics': True, 'search_space_updates': <autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7f4ab691ebe0>}
[DEBUG] [2022-08-09 22:23:58,146:Client-pynisher] Redirecting output of the function to files. Access them via the stdout and stderr attributes of the wrapped function.
[DEBUG] [2022-08-09 22:23:58,147:Client-pynisher] call function
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Fit dictionary in Abstract evaluator: dataset_properties: {'categorical_features': [], 'static_features': (), 'task_type': 'time_series_forecasting', 'numerical_features': [], 'categories': [], 'is_small_preprocess': True, 'known_future_features': (), 'categorical_columns': [], 'numerical_columns': [], 'freq': '1D', 'n_prediction_steps': 30, 'sequence_lengths_train': array([152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152, 152,
152, 152, 152, 152, 152, 152]), 'feature_names': (), 'input_shape': (152, 0), 'output_shape': [30, 1], 'time_feature_names': ('time_feature_DayOfWeek', 'time_feature_DayOfMonth', 'time_feature_DayOfYear'), 'feature_shapes': {}, 'output_type': 'continuous', 'issparse': False, 'sp': 7, 'time_feature_transform': [DayOfWeek(), DayOfMonth(), DayOfYear()], 'uni_variant': True, 'static_features_shape': 0, 'future_feature_shapes': (30, 0), 'targets_have_missing_values': False, 'encoder_can_be_auto_regressive': True, 'features_have_missing_values': False}
additional_metrics: ['mean_MASE_forecasting', 'mean_MASE_forecasting', 'median_MASE_forecasting', 'mean_MAE_forecasting', 'median_MAE_forecasting', 'mean_MAPE_forecasting', 'median_MAPE_forecasting', 'mean_MSE_forecasting', 'median_MSE_forecasting']
X_train: None
y_train: 0
0 -0.678603
0 -0.678603
0 -0.678603
0 -0.678603
0 -0.678603
.. ...
265 3.787003
265 3.974447
265 3.974447
265 4.161892
265 4.161892
[48412 rows x 1 columns]
X_test: None
y_test: [[ 2.84623863]
[ 2.90844171]
[ 2.94991043]
...
[25.71804243]
[27.40504549]
[27.59249028]]
backend: <autoPyTorch.automl_common.common.utils.backend.Backend object at 0x7fe367e3fc70>
logger_port: 47969
optimize_metric: mean_MASE_forecasting
device: cuda
budget_type: epochs
epochs: 5.555555555555555
torch_num_threads: 1
early_stopping: 20
use_tensorboard_logger: False
metrics_during_training: True
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7fe367e54070>
[DEBUG] [2022-08-09 22:23:58,504:Client-TimeSeriesForecastingTrainEvaluator(1)] Search space updates :<autoPyTorch.utils.hyperparameter_search_space_update.HyperparameterSearchSpaceUpdates object at 0x7fe367e54070>
[INFO] [2022-08-09 22:23:58,509:Client-TimeSeriesForecastingTrainEvaluator(1)] Starting fit 0
[DEBUG] [2022-08-09 22:23:59,786:Client-TimeSeriesForecastingTrainEvaluator(1)] /home/robby/miniconda3/envs/auto-pytorch/lib/python3.8/site-packages/autoPyTorch/pipeline/components/setup/network_initializer/XavierInit.py:27: UserWarning:nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
[DEBUG] [2022-08-09 22:24:06,842:Client-2_1660055039.7867444]
========================================
Epoch 1
========================================
end_time: 1660055046.841151
start_time: 1660055039.7924404
test_loss: None
test_metrics:
train_loss: 0.5615030335262418
train_metrics:
val_loss: 0.7551549077033997
val_metrics:
mean_MAE_forecasting: 0.3713443875312805
mean_MAPE_forecasting: 76028412166144.0
mean_MASE_forecasting: 6.3204656801767305
mean_MSE_forecasting: 0.5660822987556458
median_MAE_forecasting: 0.12749794125556946
median_MAPE_forecasting: 0.14768588542938232
median_MASE_forecasting: 0.9045267487515163
median_MSE_forecasting: 0.021999072283506393
========================================
[DEBUG] [2022-08-09 22:24:13,077:Client-2_1660055039.7867444]
========================================
Epoch 2
========================================
end_time: 1660055053.075863
start_time: 1660055046.84346
test_loss: None
test_metrics:
train_loss: -0.21076665975153447
train_metrics:
val_loss: 0.37946489453315735
val_metrics:
mean_MAE_forecasting: 0.306547611951828
mean_MAPE_forecasting: 34252896862208.0
mean_MASE_forecasting: 5.483508303192071
mean_MSE_forecasting: 0.47867074608802795
median_MAE_forecasting: 0.06652390211820602
median_MAPE_forecasting: 0.1471085548400879
median_MASE_forecasting: 0.5327428760948727
median_MSE_forecasting: 0.006378884892910719
========================================
[DEBUG] [2022-08-09 22:24:19,345:Client-2_1660055039.7867444]
========================================
Epoch 3
========================================
end_time: 1660055059.3446581
start_time: 1660055053.0779555
test_loss: None
test_metrics:
train_loss: -0.4158153299242258
train_metrics:
val_loss: 0.11834084987640381
val_metrics:
mean_MAE_forecasting: 0.27670568227767944
mean_MAPE_forecasting: 21054133108736.0
mean_MASE_forecasting: 5.087560038876242
mean_MSE_forecasting: 0.42972803115844727
median_MAE_forecasting: 0.04717379808425903
median_MAPE_forecasting: 0.1136779934167862
median_MASE_forecasting: 0.4350006861389294
median_MSE_forecasting: 0.0034421852324157953
========================================
[DEBUG] [2022-08-09 22:24:25,618:Client-2_1660055039.7867444]
========================================
Epoch 4
========================================
end_time: 1660055065.6175072
start_time: 1660055059.3467505
test_loss: None
test_metrics:
train_loss: -0.5240315041341819
train_metrics:
val_loss: 0.09902344644069672
val_metrics:
mean_MAE_forecasting: 0.27489030361175537
mean_MAPE_forecasting: 18999616208896.0
mean_MASE_forecasting: 5.04518199855496
mean_MSE_forecasting: 0.42932283878326416
median_MAE_forecasting: 0.050067659467458725
median_MAPE_forecasting: 0.11365534365177155
median_MASE_forecasting: 0.442323123342876
median_MSE_forecasting: 0.0035072178579866886
========================================
[DEBUG] [2022-08-09 22:24:32,030:Client-2_1660055039.7867444]
========================================
Epoch 5
========================================
end_time: 1660055072.0294464
start_time: 1660055065.6196034
test_loss: None
test_metrics:
train_loss: -0.5758035796135664
train_metrics:
val_loss: 0.018720028921961784
val_metrics:
mean_MAE_forecasting: 0.2657284140586853
mean_MAPE_forecasting: 14370088157184.0
mean_MASE_forecasting: 4.9683659710981765
mean_MSE_forecasting: 0.42335325479507446
median_MAE_forecasting: 0.03455820679664612
median_MAPE_forecasting: 0.11295084655284882
median_MASE_forecasting: 0.3162790758898065
median_MSE_forecasting: 0.0017664149636402726
========================================
[DEBUG] [2022-08-09 22:24:32,030:Client-2_1660055039.7867444] Early stopped model 2 on epoch 5
[INFO] [2022-08-09 22:24:32,032:Client-2_1660055039.7867444] Finished training with
========================================
Epoch 5
========================================
end_time: 1660055072.0294464
start_time: 1660055065.6196034
test_loss: None
test_metrics:
train_loss: -0.5758035796135664
train_metrics:
val_loss: 0.018720028921961784
val_metrics:
mean_MAE_forecasting: 0.2657284140586853
mean_MAPE_forecasting: 14370088157184.0
mean_MASE_forecasting: 4.9683659710981765
mean_MSE_forecasting: 0.42335325479507446
median_MAE_forecasting: 0.03455820679664612
median_MAPE_forecasting: 0.11295084655284882
median_MASE_forecasting: 0.3162790758898065
median_MSE_forecasting: 0.0017664149636402726
========================================
if you want to reproduce the results from the paper, you could use the code from this branch: https://github.com/dengdifan/Auto-PyTorch/tree/ecml22_apt_ts (I did some modifications to make it compatible with the current workflow of AutoPyTorch).
The metric 'MASE' is only applied to evaluate the trained model as a loss that will be returned to SMAC. Depending on the head type, networks could be trained with different losses (for instance, this implementation is trained with negative log-likelihood loss (DistributionLoss)). Normally I would recommend not evaluating validation metrics at each iteration, as it might be too expensive for some auto-regressive networks to do inference at every iteration (for the same reason, the val_loss in the log file does not really correspond to the validation loss)
I am closing this issue due to inactivity. Feel free to reopen if the issue persists.
