satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Add arctic weather satelliter l1b reader

Open adybbroe opened this issue 2 years ago • 7 comments

This PR adds a reader for the ESA Arctic Weather Satellite (AWS) l1b format.

The AWS file format is netCDF and supposed to follow the EPS-SG MWS level-1b format. See https://www.eumetsat.int/media/44139

However, please be aware that there are some issues with the level-1b format of the test data we have acquired from ESA. It is yet unsure to what extent ESA will adapt to our feedback.

  • [ ] Closes #xxxx
  • [x] Tests added
  • [ ] Fully documented
  • [ ] Add your name to AUTHORS.md if not there already

adybbroe avatar Sep 05 '23 06:09 adybbroe

Here is a view of channel 1 remapped to a near-ided perspective projection over northern Europe:

aws_testdata_50 3GHz_20230707_1200_nsper_swe

adybbroe avatar Sep 05 '23 06:09 adybbroe

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (ee63577) 95.18% compared to head (47821d6) 95.30%. Report is 215 commits behind head on main.

Files Patch % Lines
satpy/readers/aws_l1b.py 98.73% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2565      +/-   ##
==========================================
+ Coverage   95.18%   95.30%   +0.12%     
==========================================
  Files         354      371      +17     
  Lines       51316    52226     +910     
==========================================
+ Hits        48846    49776     +930     
+ Misses       2470     2450      -20     
Flag Coverage Δ
behaviourtests 4.18% <0.00%> (-0.06%) :arrow_down:
unittests 95.92% <99.46%> (+0.11%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Sep 05 '23 06:09 codecov[bot]

Pull Request Test Coverage Report for Build 7047743504

Warning: This coverage report may be inaccurate.

We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report. To ensure accuracy in future PRs, please see these guidelines. A quick fix for this PR: rebase it; your next report should be accurate.

  • 185 of 186 (99.46%) changed or added relevant lines in 2 files are covered.
  • 214 unchanged lines in 32 files lost coverage.
  • Overall coverage increased (+0.1%) to 95.875%

Changes Missing Coverage Covered Lines Changed/Added Lines %
satpy/readers/aws_l1b.py 78 79 98.73%
<!-- Total: 185 186
Files with Coverage Reduction New Missed Lines %
satpy/modifiers/spectral.py 1 92.13%
satpy/readers/atms_sdr_hdf5.py 1 90.91%
satpy/readers/generic_image.py 1 97.67%
satpy/tests/modifier_tests/test_angles.py 1 99.5%
satpy/tests/modifier_tests/test_parallax.py 1 99.72%
satpy/tests/reader_tests/test_scmi.py 1 99.07%
satpy/tests/reader_tests/test_vii_base_nc.py 1 98.77%
satpy/tests/reader_tests/test_vii_l1b_nc.py 1 97.22%
satpy/modifiers/angles.py 2 99.23%
satpy/_scene_converters.py 2 91.67%
<!-- Total: 214
Totals Coverage Status
Change from base Build 6737851658: 0.1%
Covered Lines: 49902
Relevant Lines: 52049

💛 - Coveralls

coveralls avatar Nov 07 '23 09:11 coveralls

So, just a comment/observation. The AWS testdata file delivered August 2023 has the datasets structured a bit odd and different from most other satellite datasets (e.g. the EPS-SG MWS testdata which format it is supposed to follow).

Here a phrase communicated to ESA abouth this layout issue:

"The ordering of the dimensions is unexpected, e.g. for “aws_toa_brightness_temperature” we have (19, 145, 4991) which is (n_channels, n_fovs, n_scans). But in netCDF the first dimension is always the one that varies the most slowly. For MWS the order is reversed, i.e. (n_scans, n_fovs, n_channels)."

This ordering currently has the unfortunate consequence that the optimal radius of influence used for remapping cannot be determined. The method geocentric_resolution in the CoordinateDefinition class in the geometry.py module of Pyresample assumes the first dimension is the scanlines, but here it is the pixel number across scan instead, so the points found are the points close to nadir of the first scanline (instead of the outer most edge of the scan in the middle of the swath). In the testdata set delivered the first 10-20 scanlines or so are missing-data in the geolocation (for some strange reason).

I am preparing a PR for Pyresample on this.

Until now one has to provide a reasonable value (say 40000) for radius_of_influence parameter explicitly in the resample call on the scene object.

When doing this one may be able create an image like the below using code similar to this (I have here skipped the decoration/coast overlay part for simplification):

scn = Scene(filenames=FILENAMES, reader='aws_l1b_nc')
composite_name = 'mw183_humidity_surface'
scn.load([composite_name])

local = scn.resample(AREAID, radius_of_influence=40000)
local.show(composite_name)

aws_testdata_radsim_mw183_humidity_surface_20240115_1111_nsper_swe

adybbroe avatar Nov 17 '23 11:11 adybbroe

I somehow thought we had merged this long ago? But, it makes sense not to merge it now, as we will have new test data from ESA in about a month, where the issues with the format that we raised with the, should be solved. Then we can update the reader for the new format. So, we can wait another month or two perhaps, or what do you say @mraspaud and @pnuu ?

Sorry for not coming back to your comments/questions earlier!

adybbroe avatar Feb 21 '24 17:02 adybbroe

Any news on the new test dataset?

mraspaud avatar Apr 17 '24 13:04 mraspaud