forecasting in real time with gencast_mini_demo

[Michael-Sharma]

For what period are gencast_mini_demo forecasts made? is it possible to get real-time forecasts using gencast_mini_demo?

[Muhammad1umer-tech]

I also have a concern about real time prediction.

I am using (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download to download the ERA5 dataset by providing coordinates. However, some variables, such as year progress (cos, sin) and others, are missing. Even after hardcoding them and doing some transformation, I am still encountering errors.

Like: -ValueError: Longitude coords don't match those used by spherical_harmonic.SphericalHarmonicBasis. -Lat is no sorted -grid2mesh_gnn/~_networks_builder/encoder_nodes_grid_nodes_mlp/~/linear_0/w' with retrieved shape (267, 512) does not match shape=[303, 512] dtype=dtype('float32')

I am using the GenCast1deg model

I can provide the code, if someone wants to review it.

Thanks

[alvarosg]

Hi!

We recently started publishing, both realtime and historical forecasts for more up to date versions of GraphCast and GenCast under the name WeatherNext Graph and WeatherNext Gen.

You should be able to request access via this form.

Hope this will make things easier!

[Michael-Sharma]

@Muhammad1umer-tech, thank you very much! I would be very grateful if you could provide the code.

[gtamkin]

@Muhammad1umer-tech , I too would be grateful if you could provide the code. I'm looking for a recipe to prepare current data for gencast_mini_demo forecasts. In other words, how do I create sample data like the ones that are used in the sample notebooks (e.g., https://storage.googleapis.com/dm_graphcast/gencast/dataset/source-era5_date-2019-03-29_res-1.0_levels-13_steps-01.nc). Any guidance would be greatly appreciated. Thank you!

[JacquesMeyerCognizant]

Hi Team !

I am also trying to work on a gencast 1.0 model by creating sample data like the ones that are used in the sample notebooks using the era5 ECMWF Copernicus API. I am able to launch the model with my personalized data but the prediction precision of the results are far from the ones when using the input samples provided. There is a very slight difference in the values of the input data would this explain the performance difference ?

Is it actually possible to use the data from the Copernicus API with 1 deg (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download) or has there been modifications made to the data on your side ?

Is the only solution to be able to launch to model on any historical date to use WeatherNext Gen ?

Thank you for you response.

[andrewlkd]

Hey,

Is it actually possible to use the data from the Copernicus API with 1 deg

Absolutely, this is where we got the data from as well!

There is a very slight difference in the values of the input data would this explain the performance difference ?

This will definitely cause issues. Although, without more information on the differences, I can't say much about this.

As an intermediate unblocker, you might wish to make use of ready zarrs from

https://weatherbench2.readthedocs.io/en/latest/data-guide.html#era5 or https://github.com/google-research/arco-era5?tab=readme-ov-file#analysis-ready-data

We produce our 1deg datasets by subsampling the 0.25deg data. I.e.

1deg_ds = 0p25deg_ds.isel(lat=slice(None, None, 4), lon=slice(None, None, 4))

Hope this helps,

Andrew

[JacquesMeyerCognizant]

Thank you for your quick response ! It helped a lot :)

[JacquesMeyerCognizant]

Hi ! I am trying to use the model to do real time prediction, with this data source https://github.com/google-research/arco-era5?tab=readme-ov-file#analysis-ready-data But if I understand this correctly the precipitation variable is missing. Is it still possible to launch the model without ? or if not, what data source do you use to do real time prediction ?

Thank you !

[andrewlkd]

Hey Jacques,

You might find the responses on https://github.com/google-deepmind/graphcast/issues/165 useful.

Best,

Andrew