NVlabs
TP accuracy
Hello everyone. I have an issue with the TP acc which is extremely low. does anyone know what can be the problem? below you can find the output after an inference and my input data info:
input h5 shape: tp(120, 721, 1440).
inference: 2023-06-25 11:24:06,620 - root - INFO - Timestep 0 of 20. TP RMS Error: 0.0, ACC: 1.0 2023-06-25 11:24:09,065 - root - INFO - Timestep 1 of 20. TP RMS Error: 0.0016368781216442585, ACC: 0.35278499126434326 2023-06-25 11:24:09,388 - root - INFO - Timestep 2 of 20. TP RMS Error: 0.00156010827049613, ACC: 0.3554094731807709 2023-06-25 11:24:09,688 - root - INFO - Timestep 3 of 20. TP RMS Error: 0.0015420995187014341, ACC: 0.3688367009162903 2023-06-25 11:24:09,988 - root - INFO - Timestep 4 of 20. TP RMS Error: 0.0014832873130217195, ACC: 0.3889610171318054 2023-06-25 11:24:10,289 - root - INFO - Timestep 5 of 20. TP RMS Error: 0.0014608813216909766, ACC: 0.36930468678474426 2023-06-25 11:24:10,589 - root - INFO - Timestep 6 of 20. TP RMS Error: 0.0015008833725005388, ACC: 0.32038554549217224 2023-06-25 11:24:10,889 - root - INFO - Timestep 7 of 20. TP RMS Error: 0.0013966681435704231, ACC: 0.36792299151420593 2023-06-25 11:24:11,189 - root - INFO - Timestep 8 of 20. TP RMS Error: 0.001374703599140048, ACC: 0.3730277121067047 2023-06-25 11:24:11,490 - root - INFO - Timestep 9 of 20. TP RMS Error: 0.0013704199809581041, ACC: 0.33114054799079895 2023-06-25 11:24:11,790 - root - INFO - Timestep 10 of 20. TP RMS Error: 0.0014247711515054107, ACC: 0.2770615518093109 2023-06-25 11:24:12,090 - root - INFO - Timestep 11 of 20. TP RMS Error: 0.001329558901488781, ACC: 0.3315066695213318 2023-06-25 11:24:12,390 - root - INFO - Timestep 12 of 20. TP RMS Error: 0.0012841359712183475, ACC: 0.35082289576530457 2023-06-25 11:24:12,690 - root - INFO - Timestep 13 of 20. TP RMS Error: 0.0012776957591995597, ACC: 0.32422271370887756 2023-06-25 11:24:12,990 - root - INFO - Timestep 14 of 20. TP RMS Error: 0.001365577569231391, ACC: 0.2353413850069046 2023-06-25 11:24:13,290 - root - INFO - Timestep 15 of 20. TP RMS Error: 0.001326375175267458, ACC: 0.28706997632980347 2023-06-25 11:24:13,590 - root - INFO - Timestep 16 of 20. TP RMS Error: 0.0013120684307068586, ACC: 0.3122824430465698 2023-06-25 11:24:13,890 - root - INFO - Timestep 17 of 20. TP RMS Error: 0.001352619961835444, ACC: 0.270219087600708 2023-06-25 11:24:14,190 - root - INFO - Timestep 18 of 20. TP RMS Error: 0.0014553270302712917, ACC: 0.1826096624135971 2023-06-25 11:24:14,489 - root - INFO - Timestep 19 of 20. TP RMS Error: 0.0014066090807318687, ACC: 0.23088878393173218
Hello again guys. Today I downloaded the pre-processed data (which is uploaded to the globus) for 2018 for example and sliced it to just 10-time steps. I performed the precipitation inference for this small dataset and the results were very disappointing. The acc was around 0.03 and I dont know where do i made a mistake! I will paste here both results. The first output was performed with preprocessed dataset provided here in GitHub and globus and the second one is the output performed on my custom dataset.
2023-06-27 10:49:01,288 - root - INFO - --------------- Versions --------------- 2023-06-27 10:49:01,300 - root - INFO - git branch: b'* master' 2023-06-27 10:49:01,307 - root - INFO - git hash: b'4cac4678bcf003e2692f3c6528ccb4aa62204f4d' 2023-06-27 10:49:01,307 - root - INFO - Torch: 2.0.1+cu118 2023-06-27 10:49:01,307 - root - INFO - ---------------------------------------- 2023-06-27 10:49:01,307 - root - INFO - ------------------ Configuration ------------------ 2023-06-27 10:49:01,307 - root - INFO - Configuration file: /content/drive/Shareddrives/Final_FCN/FourCastNet/config/AFNO.yaml 2023-06-27 10:49:01,307 - root - INFO - Configuration name: precip 2023-06-27 10:49:01,308 - root - INFO - in_channels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] 2023-06-27 10:49:01,308 - root - INFO - out_channels [0] 2023-06-27 10:49:01,308 - root - INFO - nettype afno 2023-06-27 10:49:01,308 - root - INFO - nettype_wind afno 2023-06-27 10:49:01,308 - root - INFO - log_to_wandb True 2023-06-27 10:49:01,308 - root - INFO - lr 0.00025 2023-06-27 10:49:01,308 - root - INFO - batch_size 64 2023-06-27 10:49:01,308 - root - INFO - max_epochs 25 2023-06-27 10:49:01,308 - root - INFO - precip /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy 2023-06-27 10:49:01,308 - root - INFO - time_means_path_tp /content/drive/Shareddrives/Final_FCN/stats_v0_globus/time_mean_tp/time_means.npy 2023-06-27 10:49:01,308 - root - INFO - model_wind_path /content/drive/Shareddrives/Final_FCN/backbone.ckpt 2023-06-27 10:49:01,308 - root - INFO - precip_eps 1e-05 2023-06-27 10:49:01,308 - root - INFO - scheduler CosineAnnealingLR 2023-06-27 10:49:01,308 - root - INFO - orography False 2023-06-27 10:49:01,309 - root - INFO - orography_path None 2023-06-27 10:49:01,309 - root - INFO - exp_dir /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output 2023-06-27 10:49:01,309 - root - INFO - train_data_path /pscratch/sd/s/shas1693/data/era5/train 2023-06-27 10:49:01,309 - root - INFO - valid_data_path /pscratch/sd/s/shas1693/data/era5/test 2023-06-27 10:49:01,309 - root - INFO - inf_data_path /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone 2023-06-27 10:49:01,309 - root - INFO - time_means_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/time_means.npy 2023-06-27 10:49:01,309 - root - INFO - global_means_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/global_means.npy 2023-06-27 10:49:01,309 - root - INFO - global_stds_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/global_stds.npy 2023-06-27 10:49:01,309 - root - INFO - loss l2 2023-06-27 10:49:01,309 - root - INFO - num_data_workers 4 2023-06-27 10:49:01,309 - root - INFO - dt 1 2023-06-27 10:49:01,309 - root - INFO - n_history 0 2023-06-27 10:49:01,309 - root - INFO - prediction_type iterative 2023-06-27 10:49:01,310 - root - INFO - prediction_length 3 2023-06-27 10:49:01,310 - root - INFO - n_initial_conditions 5 2023-06-27 10:49:01,310 - root - INFO - ics_type default 2023-06-27 10:49:01,310 - root - INFO - save_raw_forecasts True 2023-06-27 10:49:01,310 - root - INFO - save_channel False 2023-06-27 10:49:01,310 - root - INFO - masked_acc False 2023-06-27 10:49:01,310 - root - INFO - maskpath None 2023-06-27 10:49:01,310 - root - INFO - perturb False 2023-06-27 10:49:01,310 - root - INFO - add_grid False 2023-06-27 10:49:01,310 - root - INFO - N_grid_channels 0 2023-06-27 10:49:01,310 - root - INFO - gridtype sinusoidal 2023-06-27 10:49:01,310 - root - INFO - roll False 2023-06-27 10:49:01,310 - root - INFO - num_blocks 8 2023-06-27 10:49:01,310 - root - INFO - patch_size 8 2023-06-27 10:49:01,311 - root - INFO - width 56 2023-06-27 10:49:01,311 - root - INFO - modes 32 2023-06-27 10:49:01,311 - root - INFO - target default 2023-06-27 10:49:01,311 - root - INFO - normalization zscore 2023-06-27 10:49:01,311 - root - INFO - log_to_screen True 2023-06-27 10:49:01,311 - root - INFO - save_checkpoint True 2023-06-27 10:49:01,311 - root - INFO - enable_nhwc False 2023-06-27 10:49:01,311 - root - INFO - optimizer_type FusedAdam 2023-06-27 10:49:01,311 - root - INFO - crop_size_x None 2023-06-27 10:49:01,311 - root - INFO - crop_size_y None 2023-06-27 10:49:01,311 - root - INFO - two_step_training False 2023-06-27 10:49:01,311 - root - INFO - plot_animations False 2023-06-27 10:49:01,312 - root - INFO - add_noise False 2023-06-27 10:49:01,312 - root - INFO - noise_std 0 2023-06-27 10:49:01,312 - root - INFO - world_size 1 2023-06-27 10:49:01,312 - root - INFO - global_batch_size 64 2023-06-27 10:49:01,312 - root - INFO - experiment_dir /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output 2023-06-27 10:49:01,312 - root - INFO - best_checkpoint_path /content/drive/Shareddrives/Final_FCN/precip.ckpt 2023-06-27 10:49:01,312 - root - INFO - resuming False 2023-06-27 10:49:01,312 - root - INFO - local_rank 0 2023-06-27 10:49:01,312 - root - INFO - --------------------------------------------------- 2023-06-27 10:49:01,312 - root - INFO - Inference for 1 initial conditions 2023-06-27 10:49:01,315 - root - INFO - Getting file stats from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone/2011_11.h5 2023-06-27 10:49:01,315 - root - INFO - Number of samples per year: 120 2023-06-27 10:49:01,315 - root - INFO - Found data at path /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone. Number of examples: 120. Image Shape: 720 x 1440 x 20 2023-06-27 10:49:01,316 - root - INFO - Delta t: 6 hours 2023-06-27 10:49:01,316 - root - INFO - Including 0 hours of past history in training at a frequency of 6 hours 2023-06-27 10:49:01,316 - root - INFO - Loading trained model checkpoint from /content/drive/Shareddrives/Final_FCN/precip.ckpt 2023-06-27 10:49:05,853 - root - INFO - Loading validation data 2023-06-27 10:49:05,853 - root - INFO - Validation data from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone/2011_11.h5 2023-06-27 10:49:05,855 - root - INFO - Loading validation precip data 2023-06-27 10:49:05,855 - root - INFO - Validation data from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/2018.h5 2023-06-27 10:49:06,239 - root - INFO - Initial condition 1 of 1 2023-06-27 10:49:07,653 - root - INFO - Begin autoregressive inference 2023-06-27 10:49:09,671 - root - INFO - Timestep 0 of 3. TP RMS Error: 0.0, ACC: 1.0 2023-06-27 10:49:12,681 - root - INFO - Timestep 1 of 3. TP RMS Error: 0.0030234134756028652, ACC: 0.031174656003713608 2023-06-27 10:49:14,035 - root - INFO - Timestep 2 of 3. TP RMS Error: 0.002889520488679409, ACC: 0.03148724511265755 2023-06-27 10:49:14,041 - root - INFO - Saving files at /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output/autoregressive_predictions_tp.h5
2023-06-27 10:49:52,721 - root - INFO - --------------- Versions --------------- 2023-06-27 10:49:52,733 - root - INFO - git branch: b'* master' 2023-06-27 10:49:52,741 - root - INFO - git hash: b'4cac4678bcf003e2692f3c6528ccb4aa62204f4d' 2023-06-27 10:49:52,741 - root - INFO - Torch: 2.0.1+cu118 2023-06-27 10:49:52,741 - root - INFO - ---------------------------------------- 2023-06-27 10:49:52,742 - root - INFO - ------------------ Configuration ------------------ 2023-06-27 10:49:52,742 - root - INFO - Configuration file: /content/drive/Shareddrives/Final_FCN/FourCastNet/config/AFNO.yaml 2023-06-27 10:49:52,742 - root - INFO - Configuration name: precip 2023-06-27 10:49:52,742 - root - INFO - in_channels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] 2023-06-27 10:49:52,742 - root - INFO - out_channels [0] 2023-06-27 10:49:52,742 - root - INFO - nettype afno 2023-06-27 10:49:52,742 - root - INFO - nettype_wind afno 2023-06-27 10:49:52,742 - root - INFO - log_to_wandb True 2023-06-27 10:49:52,742 - root - INFO - lr 0.00025 2023-06-27 10:49:52,742 - root - INFO - batch_size 64 2023-06-27 10:49:52,743 - root - INFO - max_epochs 25 2023-06-27 10:49:52,743 - root - INFO - precip /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy 2023-06-27 10:49:52,743 - root - INFO - time_means_path_tp /content/drive/Shareddrives/Final_FCN/stats_v0_globus/time_mean_tp/time_means.npy 2023-06-27 10:49:52,743 - root - INFO - model_wind_path /content/drive/Shareddrives/Final_FCN/backbone.ckpt 2023-06-27 10:49:52,743 - root - INFO - precip_eps 1e-05 2023-06-27 10:49:52,743 - root - INFO - scheduler CosineAnnealingLR 2023-06-27 10:49:52,743 - root - INFO - orography False 2023-06-27 10:49:52,743 - root - INFO - orography_path None 2023-06-27 10:49:52,743 - root - INFO - exp_dir /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output 2023-06-27 10:49:52,743 - root - INFO - train_data_path /pscratch/sd/s/shas1693/data/era5/train 2023-06-27 10:49:52,743 - root - INFO - valid_data_path /pscratch/sd/s/shas1693/data/era5/test 2023-06-27 10:49:52,743 - root - INFO - inf_data_path /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone 2023-06-27 10:49:52,744 - root - INFO - time_means_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/time_means.npy 2023-06-27 10:49:52,744 - root - INFO - global_means_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/global_means.npy 2023-06-27 10:49:52,744 - root - INFO - global_stds_path /content/drive/Shareddrives/Final_FCN/stats_v0_globus/global_stds.npy 2023-06-27 10:49:52,744 - root - INFO - loss l2 2023-06-27 10:49:52,744 - root - INFO - num_data_workers 4 2023-06-27 10:49:52,744 - root - INFO - dt 1 2023-06-27 10:49:52,744 - root - INFO - n_history 0 2023-06-27 10:49:52,744 - root - INFO - prediction_type iterative 2023-06-27 10:49:52,744 - root - INFO - prediction_length 3 2023-06-27 10:49:52,744 - root - INFO - n_initial_conditions 5 2023-06-27 10:49:52,745 - root - INFO - ics_type default 2023-06-27 10:49:52,745 - root - INFO - save_raw_forecasts True 2023-06-27 10:49:52,745 - root - INFO - save_channel False 2023-06-27 10:49:52,745 - root - INFO - masked_acc False 2023-06-27 10:49:52,745 - root - INFO - maskpath None 2023-06-27 10:49:52,745 - root - INFO - perturb False 2023-06-27 10:49:52,745 - root - INFO - add_grid False 2023-06-27 10:49:52,745 - root - INFO - N_grid_channels 0 2023-06-27 10:49:52,745 - root - INFO - gridtype sinusoidal 2023-06-27 10:49:52,745 - root - INFO - roll False 2023-06-27 10:49:52,745 - root - INFO - num_blocks 8 2023-06-27 10:49:52,746 - root - INFO - patch_size 8 2023-06-27 10:49:52,746 - root - INFO - width 56 2023-06-27 10:49:52,746 - root - INFO - modes 32 2023-06-27 10:49:52,746 - root - INFO - target default 2023-06-27 10:49:52,746 - root - INFO - normalization zscore 2023-06-27 10:49:52,746 - root - INFO - log_to_screen True 2023-06-27 10:49:52,746 - root - INFO - save_checkpoint True 2023-06-27 10:49:52,746 - root - INFO - enable_nhwc False 2023-06-27 10:49:52,746 - root - INFO - optimizer_type FusedAdam 2023-06-27 10:49:52,746 - root - INFO - crop_size_x None 2023-06-27 10:49:52,746 - root - INFO - crop_size_y None 2023-06-27 10:49:52,746 - root - INFO - two_step_training False 2023-06-27 10:49:52,747 - root - INFO - plot_animations False 2023-06-27 10:49:52,747 - root - INFO - add_noise False 2023-06-27 10:49:52,747 - root - INFO - noise_std 0 2023-06-27 10:49:52,747 - root - INFO - world_size 1 2023-06-27 10:49:52,747 - root - INFO - global_batch_size 64 2023-06-27 10:49:52,747 - root - INFO - experiment_dir /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output 2023-06-27 10:49:52,747 - root - INFO - best_checkpoint_path /content/drive/Shareddrives/Final_FCN/precip.ckpt 2023-06-27 10:49:52,747 - root - INFO - resuming False 2023-06-27 10:49:52,747 - root - INFO - local_rank 0 2023-06-27 10:49:52,747 - root - INFO - --------------------------------------------------- 2023-06-27 10:49:52,747 - root - INFO - Inference for 1 initial conditions 2023-06-27 10:49:52,766 - root - INFO - Getting file stats from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone/2011_11.h5 2023-06-27 10:49:52,767 - root - INFO - Number of samples per year: 120 2023-06-27 10:49:52,767 - root - INFO - Found data at path /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone. Number of examples: 120. Image Shape: 720 x 1440 x 20 2023-06-27 10:49:52,767 - root - INFO - Delta t: 6 hours 2023-06-27 10:49:52,767 - root - INFO - Including 0 hours of past history in training at a frequency of 6 hours 2023-06-27 10:49:52,768 - root - INFO - Loading trained model checkpoint from /content/drive/Shareddrives/Final_FCN/precip.ckpt 2023-06-27 10:49:57,272 - root - INFO - Loading validation data 2023-06-27 10:49:57,272 - root - INFO - Validation data from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/backbone/2011_11.h5 2023-06-27 10:49:57,275 - root - INFO - Loading validation precip data 2023-06-27 10:49:57,275 - root - INFO - Validation data from /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/2011_11_tp.h5 2023-06-27 10:49:57,694 - root - INFO - Initial condition 1 of 1 2023-06-27 10:49:58,841 - root - INFO - Begin autoregressive inference 2023-06-27 10:50:00,844 - root - INFO - Timestep 0 of 3. TP RMS Error: 0.0, ACC: 1.0 2023-06-27 10:50:03,975 - root - INFO - Timestep 1 of 3. TP RMS Error: 0.0016368868527933955, ACC: 0.35275188088417053 2023-06-27 10:50:05,338 - root - INFO - Timestep 2 of 3. TP RMS Error: 0.001560143311508, ACC: 0.3554510176181793 2023-06-27 10:50:05,345 - root - INFO - Saving files at /content/drive/Shareddrives/Final_FCN/validation_tp_increase_accuracy/output/autoregressive_predictions_tp.h5