pygmt Add GEBCO data source to load_earth

This pull request adds the GEBCO data source to load_earth_relief

Addresses #1786

Reminders

[ ] Run make format and make check to make sure the code follows the style guide.
[ ] Add tests for new features or tests that would have caught the bug that you're fixing.
[ ] Add new public functions/methods/classes to doc/api/index.rst.
[ ] Write detailed docstrings for all functions/methods.
[ ] If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
[ ] If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform specific operations. Supported slash commands are:

/format: automatically format and lint the code
/test-gmt-dev: run full tests on the latest GMT development version

Mar 15 '22 10:03 willschlitzer

I'm running into an error when my machine is trying to find the cached version of the GEBCO files. When I run both grid = load_earth_relief(data_source="gebco", region = [0, 5, 10, 12]) and grid = pygmt.grdcut(grid="@earth_gebco_01d", region = [0, 5, 10, 12]) I get the following:

GMTCLibError: Module 'grdcut' failed with status code 16:
grdcut [ERROR]: Libcurl Error: HTTP response code said error
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd not found

My guess is the issue is that the Earth GEBCO files are stored under .gmt/server/earth/earth_gebco rather than .gmt/cache. Is this a GMT error for how it finds the cached files?

Mar 15 '22 10:03 willschlitzer

What's the output if you run the following GMT command?

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd

Mar 15 '22 11:03 seisman

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd

Same issue. Does it work for you?

Here is the output:

grdcut [NOTICE]:   -> Download grid file [105K]: earth_gebco_01d_g.grd
grdcut [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/cache/earth_gebco_01d.grd ...
grdcut [ERROR]: Libcurl Error: HTTP response code said error
grdcut [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdcut [DEBUG]: Delete /tmp/earth_gebco_01d.grd.download
grdcut [DEBUG]: Replace file @earth_gebco_01d.grd with /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [INFORMATION]: Processing input grid
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd not found
[Session gmt (0)]: Error returned from GMT API: GMT_FILE_NOT_FOUND (16)

The file earth_gebco_01d_g.grd is in .gmt/server/earth/earth_gebco, so my guess is that GMT is looking in the wrong folder for the .grd file?

Mar 15 '22 12:03 willschlitzer

I've got the same GMT_FILE_NOT_FOUND error when using grdcut, but using gmt which -Ga @earth_gebco_01d_g works:

gmt [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/gmt_data_server.txt ...
gmt [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/gmt_hash_server.txt ...
gmtwhich [INFORMATION]: Writing Data Table to Standard Output stream
gmtwhich [NOTICE]: Remote data courtesy of GMT data server oceania [http://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: GEBCO Earth Relief at 1x1 arc degrees reduced by Gaussian Cartesian filtering (111.2 km fullwidth) [GEBCO Compilation Group, 2021 GEBCO 2021 Grid].
gmtwhich [NOTICE]:   -> Download grid file [105K]: earth_gebco_01d_g.grd
gmtwhich [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/server/earth/earth_gebco/earth_gebco_01d_g.grd ...
/home/username/.gmt/server/earth/earth_gebco/earth_gebco_01d_g.grd

Mar 15 '22 13:03 weiji14

@weiji14 Does the use of which request a download from the remote server regardless or if it's cached, or will it use the cached version of the file if available? I was able to download the .grd and .nc files to .gmt/server/earth/earth_gebco.

Mar 15 '22 15:03 willschlitzer

@weiji14 Does the use of which request a download from the remote server regardless or if it's cached, or will it use the cached version of the file if available? I was able to download the .grd and .nc files to .gmt/server/earth/earth_gebco.

It'll definitely take the local cached file. Looking at the verbose log though, which does seem to ping server network first to check for a new gmt_data_server.txt file and will pull an update if needed.

Mar 15 '22 17:03 weiji14

These issues could be related to https://github.com/GenericMappingTools/gmt/pull/6160. I would be curious if it works after deleting ~/.gmt/server/gmt_data_server.txt

Mar 15 '22 19:03 maxrjones

These issues could be related to GenericMappingTools/gmt#6160. I would be curious if it works after deleting ~/.gmt/server/gmt_data_server.txt

Just deleted both gmt_data_server.txt and gmt_data_server,tx.old; same result

Mar 16 '22 08:03 willschlitzer

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd fails for me with GMT 6.3.0 but works with the main branch, due to the fix applied in https://github.com/GenericMappingTools/gmt/pull/6160. We may want to wait until after the GMT 6.4.0 release to add support for earth_gebco and earth_geoid.

Mar 18 '22 18:03 maxrjones

How extensively should this be testing load_earth_relief with data_source="gebco"? I don't want to duplicate all of the tests for Earth relief using SRTM data, as that seems excessive, so I just added tests using 1-degree resolution with and without setting region.

Sep 13 '22 14:09 willschlitzer

How extensively should this be testing load_earth_relief with data_source="gebco"? I don't want to duplicate all of the tests for Earth relief using SRTM data, as that seems excessive, so I just added tests using 1-degree resolution with and without setting region.

Yes, I agree.

Sep 14 '22 05:09 seisman

Please note that we decided to support multiple GMT versions in issue https://github.com/GenericMappingTools/pygmt/issues/1991.

As GMT 6.3.0 is still supported, and earth_gebco is known to have issues with GMT 6.3.0, I think we need to either find a fix for GMT 6.3.0 on the PyGMT side, or raise an error when earth_gebco is used with GMT 6.3.0.

Sep 21 '22 16:09 seisman

Please note that we decided to support multiple GMT versions in issue #1991.

As GMT 6.3.0 is still supported, and earth_gebco is known to have issues with GMT 6.3.0, I think we need to either find a fix for GMT 6.3.0 on the PyGMT side, or raise an error when earth_gebco is used with GMT 6.3.0.

I don't think there's a way to fix this through PyGMT. We should raise an error for earth_gebco or earth_geoid (when added) is used with GMT <6.4.

Sep 21 '22 17:09 maxrjones

Please note that we decided to support multiple GMT versions in issue #1991. As GMT 6.3.0 is still supported, and earth_gebco is known to have issues with GMT 6.3.0, I think we need to either find a fix for GMT 6.3.0 on the PyGMT side, or raise an error when earth_gebco is used with GMT 6.3.0.

I don't think there's a way to fix this through PyGMT. We should raise an error for earth_gebco or earth_geoid (when added) is used with GMT <6.4.

From comment https://github.com/GenericMappingTools/pygmt/pull/1818#issuecomment-1067996849, it seems for GMT 6.3.0, earth_gebco_01d doesn't work but earth_gebco_01d_g works, so perhaps having a default registration is a good fix.

Sep 22 '22 15:09 seisman

xref: https://github.com/GenericMappingTools/pygmt/issues/1929

Sep 22 '22 15:09 seisman

I'm opting to just raise an error rather than try and make other data sources backwards compatible. Is there a good way to treat the GMT version like a number? The output for Session().get_default(name="API_VERSION") is a string, so my only fix so far has been:

    with Session() as lib:
        if lib.get_default(name="API_VERSION") != "6.4.0" and data_source != "igpp":
            raise GMTVersionError(
                f"The {data_source} option is not available for GMT"
                " versions before 6.4.0."
            )

This seems like it will cause headaches with future GMT releases. Ideally, I would like to be able to specify that a GMT version <6.4.0 would raise errors, but I'm not sure how.

Sep 22 '22 17:09 willschlitzer

I'm opting to just raise an error rather than try and make other data sources backwards compatible. Is there a good way to treat the GMT version like a number? The output for Session().get_default(name="API_VERSION") is a string, so my only fix so far has been:
    with Session() as lib:
        if lib.get_default(name="API_VERSION") != "6.4.0" and data_source != "igpp":
            raise GMTVersionError(
                f"The {data_source} option is not available for GMT"
                " versions before 6.4.0."
            )
This seems like it will cause headaches with future GMT releases. Ideally, I would like to be able to specify that a GMT version <6.4.0 would raise errors, but I'm not sure how.

Please try this:

from packaging.version import Version

with Session() as lib:
    if Version(lib.info["version"]) < Version("6.4.0") and data_source == "gebco":
        raise ...

Sep 23 '22 01:09 seisman

Need to add @earth_gebco_01d to the cache too at pygmt/helpers/testing.py right?

Oct 07 '22 02:10 weiji14

@GenericMappingTools/pygmt-maintainers Can someone else give this PR a final review before we merge it?

Oct 12 '22 05:10 seisman

Add GEBCO data source to load_earth_relief