pygeoapi unit tests fail to run in Docker

[justb4]

Description The pygeoapi Docker image has an entrypoint.sh command argument to run unit tests. The command is simply docker run geopython/pygeoapi:latest test (by default the arg is run to run the container).

This does not work anymore:

• pytest is not available in the container
• many tests fail

The unit tests will skip some Providers that require a running backend like Postgres or Elastic, but for example all OGR tests are executed. But as many (most?) of the pygeoapi deployments use Docker it is important to have tests working 100%. Also for the work on #801 (GH Workflows for Docker build/test/push)

Steps to Reproduce Steps to reproduce the behavior:

• docker run geopython/pygeoapi:latest test
• module pytest cannot be found
• add in entrypoint.sh under test arg: pip3 install -r requirements-dev.txt
• now pytest is installed but many tests fail: some Providers may need to be excluded (SensorThings, xarray, ...) and e.g. OGR WFS backends (Geosolutions, known issue) is not available, needs changed to other WFS.

Expected behavior A clear and concise description of what you expected to happen.

• expect tests to run with 100% success
• provide some means to indicate overall success/fail e.g. for #801 to proceed with Push Docker Image to GH and docker.io

Screenshots/Tracebacks Once tests are running, most succeed though some fail.

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 33 items

tests/test_api.py ................F................                      [100%]

=================================== FAILURES ===================================
_________________________ test_get_collection_coverage _________________________

config = {'logging': {'level': 'ERROR'}, 'metadata': {'contact': {'address': 'Mailing Address', 'city': 'City', 'country': 'Cou....}, ...}, 'server': {'bind': {'host': '0.0.0.0', 'port': 5000}, 'cors': True, 'encoding': 'utf-8', 'gzip': False, ...}}
api_ = <pygeoapi.api.API object at 0x7f79c0dee100>

.
.

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 17 items

tests/test_ogr_shapefile_provider.py .................                   [100%]

============================== 17 passed in 1.13s ==============================
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 10 items

tests/test_ogr_sqlite_provider.py ..........                             [100%]

============================== 10 passed in 0.44s ==============================
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 21 items

tests/test_ogr_wfs_provider.py ..F.......F..F..F....                     [100%]

=================================== FAILURES ===================================
______________________________ test_get_geosol_gs ______________________________

self = <OGRProvider> {'source_type': 'WFS', 'source': 'WFS:https://demo.geo-solutions.it/geoserver/wfs?', 'source_srs': 'EPSG...e_options': {'OGR_WFS_LOAD_MULTIPLE_LAYER_DEFN': 'NO'}, 'gdal_ogr_options': {'GDAL_CACHEMAX': '64', 'CPL_DEBUG': 'NO'}}
identifier = 'Unesco_point.123', kwargs = {}, result = None
.
.
------------------------------ Captured log call -------------------------------
ERROR    pygeoapi.provider.ogr:ogr.py:819 Error Number: 1, Type: Failure, Msg: HTTP error code : 503
ERROR    pygeoapi.provider.ogr:ogr.py:225 Ignore errors during the connection for Driver                     WFS
ERROR    pygeoapi.provider.ogr:ogr.py:232 Cannot open OGR Source: WFS:https://demo.geo-solutions.it/geoserver/wfs?
ERROR    pygeoapi.provider.ogr:ogr.py:285 Cannot open OGR Source: WFS:https://demo.geo-solutions.it/geoserver/wfs?
ERROR    pygeoapi.provider.ogr:ogr.py:819 Error Number: 1, Type: Failure, Msg: HTTP error code : 503
ERROR    pygeoapi.provider.ogr:ogr.py:225 Ignore errors during the connection for Driver                     WFS
ERROR    pygeoapi.provider.ogr:ogr.py:232 Cannot open OGR Source: WFS:https://demo.geo-solutions.it/geoserver/wfs?
ERROR    pygeoapi.provider.ogr:ogr.py:367 Cannot open OGR Source: WFS:https://demo.geo-solutions.it/geoserver/wfs?
.
.
============================== 3 passed in 1.38s ===============================
Skipping: tests/test_postgresql_provider.py
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 5 items

tests/test_rasterio_provider.py .....                                    [100%]

============================== 5 passed in 1.50s ===============================
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /pygeoapi, configfile: pytest.ini
plugins: env-0.6.2, cov-3.0.0
collected 3 items

tests/test_sensorthings_provider.py FFF                                  [100%]

.
.
pygeoapi/provider/xarray_.py:75: ProviderConnectionError

Environment

• in Docker using Ubuntu Focal
• Python version: think 3.8
• pygeoapi version: latest on feb 2, 2022

Additional context At least these changes are needed:

• have pytest available when tests are run, either permanently in Image or install when run in test mode
• make all tests working again:
  • OGR: change to a working WFS (or new WFS URL from Geosolutions)
  • exclude tests that need external provider (or use a public service): at least SensorThings and xarray (?)
• create an "exit code" success/fail, where 'success' is 100% tests succeed, 'fail' in any other case

[justb4]

From @francbartoli

Hi @justb4, thanks for looking into this issue. I was just wondering if it makes sense to definitively overcome this kind of 
integration tests through the use of VCR.py (via https://github.com/kiwicom/pytest-recording for example). I remember 
this happened several times with the WFP GeoNode too.

What do you think?

[justb4]

Just the other day in another project got the same 'VCR' suggestion. But wondering: the WFS is accessed by GDAL/OGR via their driver, will that work?

[justb4]

Quick look at the code, looks like they wrap some lower-level networking but within Python...

[francbartoli]

Yes right, they are wrapping the curl library so it won't work for OGR

[justb4]

GDAL has similar issues with WFS testing, they simply skip. Remember the opengeo WFS? Also the pygeoapi OGR-WFS tests deliberately target different WFS products (MapServer, GeoServer, deegree) as to make sure they all work.

[justb4]

So I choose the GeoSolutions' WFS as it uses GeoServer.

[KoalaGeo]

If you want a WFS that's not going anywhere - http://ogc.bgs.ac.uk/digmap625k_gsml_insp_gs/wfs? for our INSPIRE download service

[justb4]

OK, thanks @KoalaGeo but looks like it provides GML Application Schema FeatureTypes, as http://ogc.bgs.ac.uk/digmap625k_gsml_insp_gs/wfs?service=WFS&request=DescribeFeatureType&version=2.0.0 returns references to XSDs, not FT defs. That may complicate things. Forgot to mention: for the tests we need OGC Simple Features Point types from a GeoServer instance.

[justb4]

Closing, some stuff has been fixed, like the WFS, replaced by #1003.