RAiDER Orbit issues `RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token`

Describe the bug

It appears a new exception is being raised in the orbit download of raider. I am not sure if this is transient issue or something that is a error with EOF in how exceptions are being raised. In <1 hour, I saw several thousand jobs fail so am disabling processing.

 â•â”€[48;2;255;255;255m[38;2;136;33;27mâ–Œ[0m[38;2;12;53;118m[48;2;240;234;210mâ–Œ[38;2;0;0;0m[1mHerbie[0mâ”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•®
 â”‚ INFO: Created a default config file.                 â”‚
 â”‚ You may view/edit Herbie's configuration here:       â”‚
 â”‚ [38;2;255;153;0m     /home/raider/.config/herbie/config.toml      [0m   â”‚
 â•°â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•¯
Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/f8e84e33-4fe3-4aca-8d2d-b9cb302be80c/S1-GUNW-D-R-115-tops-20240223_20230204-141542-00124W_00040N-PP-48c1-v3_0_1.nc to S1-GUNW-D-R-115-tops-20240223_20230204-141542-00124W_00040N-PP-48c1-v3_0_1.nc
/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_azimuth_timing.py:242: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.
  t_ceil = ts_0.floor(f'{time_step_hours}H')
/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_azimuth_timing.py:243: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.
  t_floor = ts_1.ceil(f'{time_step_hours}H')
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2024-Feb-23 14:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2024-Feb-23 15:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2023-Feb-04 14:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2023-Feb-04 15:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/f8e84e33-4fe3-4aca-8d2d-b9cb302be80c/S1-GUNW-D-R-115-tops-20240223_20230204-141542-00124W_00040N-PP-48c1-v3_0_1.json to S1-GUNW-D-R-115-tops-20240223_20230204-141542-00124W_00040N-PP-48c1-v3_0_1.json
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 301, in get_access_token
    r.raise_for_status()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/__main__.py", line 44, in <module>
    main()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/__main__.py", line 40, in main
    process_entry_point.load()()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/raider.py", line 548, in calcDelaysGUNW
    path_cfg, wavelength = RAiDER.aria.prepFromGUNW.main(iargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 400, in main
    GUNWObj = GUNW(args.file, args.weather_model, args.output_directory)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 6, in __init__
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 165, in __post_init__
    self.OrbitFile  = self.get_orbit_file()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 280, in get_orbit_file
    path_orb = eof.download.download_eofs([dt], [sat], save_dir=orbit_dir)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/download.py", line 107, in download_eofs
    results = client.download_all(
              ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 175, in download_all
    return download_all(
           ^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 407, in download_all
    access_token = get_access_token(username, password)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 303, in get_access_token
    raise RuntimeError(f"Access token creation failed. Reason: {str(err)}")
RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token

To Reproduce Here are 10 job parameter dicts supplied to hyp3 all of which failed with this error:

[{'job_id': '551df0b4-66e2-4fff-ba6e-7fb339b54eec',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17883,
   'granules': ['S1A_IW_SLC__1SDV_20240211T141451_20240211T141518_052512_0659F8_A955',
    'S1A_IW_SLC__1SDV_20240211T141516_20240211T141543_052512_0659F8_BE86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230204T141446_20230204T141513_047087_05A618_A004',
    'S1A_IW_SLC__1SDV_20230204T141511_20230204T141538_047087_05A618_DFC0'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/551df0b4-66e2-4fff-ba6e-7fb339b54eec/551df0b4-66e2-4fff-ba6e-7fb339b54eec.log'],
  'expiration_time': '2024-09-30T00:00:00+00:00',
  'processing_times': [9026.85, 9.468],
  'credit_cost': 1},
 {'job_id': 'f8e84e33-4fe3-4aca-8d2d-b9cb302be80c',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17885,
   'granules': ['S1A_IW_SLC__1SDV_20240223T141516_20240223T141542_052687_065FEB_15BF',
    'S1A_IW_SLC__1SDV_20240223T141540_20240223T141608_052687_065FEB_E793'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230204T141511_20230204T141538_047087_05A618_DFC0',
    'S1A_IW_SLC__1SDV_20230204T141536_20230204T141604_047087_05A618_C23D'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/f8e84e33-4fe3-4aca-8d2d-b9cb302be80c/f8e84e33-4fe3-4aca-8d2d-b9cb302be80c.log'],
  'expiration_time': '2024-09-21T00:00:00+00:00',
  'processing_times': [8015.603, 9.257],
  'credit_cost': 1},
 {'job_id': '68a187f0-727a-4cf4-ae38-773b4a7aab6c',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17884,
   'granules': ['S1A_IW_SLC__1SDV_20230111T141511_20230111T141538_046737_059A5B_053F',
    'S1A_IW_SLC__1SDV_20230111T141536_20230111T141604_046737_059A5B_CFF5'],
   'secondary_granules': ['S1B_IW_SLC__1SDV_20211217T141418_20211217T141445_030066_039705_98D3',
    'S1B_IW_SLC__1SDV_20211217T141443_20211217T141510_030066_039705_D4FC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/68a187f0-727a-4cf4-ae38-773b4a7aab6c/68a187f0-727a-4cf4-ae38-773b4a7aab6c.log'],
  'expiration_time': '2024-09-30T00:00:00+00:00',
  'processing_times': [10328.91, 9.075],
  'credit_cost': 1},
 {'job_id': '7a43b041-4daa-4523-90c0-8e8553b0950f',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17878,
   'granules': ['S1A_IW_SLC__1SDV_20230228T141303_20230228T141333_047437_05B1EF_408B',
    'S1A_IW_SLC__1SDV_20230228T141331_20230228T141358_047437_05B1EF_9ABB'],
   'secondary_granules': ['S1B_IW_SLC__1SDV_20211217T141213_20211217T141240_030066_039705_C63B',
    'S1B_IW_SLC__1SDV_20211217T141238_20211217T141305_030066_039705_1226'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/7a43b041-4daa-4523-90c0-8e8553b0950f/7a43b041-4daa-4523-90c0-8e8553b0950f.log'],
  'expiration_time': '2024-09-30T00:00:00+00:00',
  'processing_times': [11710.868, 9.684],
  'credit_cost': 1},
 {'job_id': 'cd9d691c-88c2-4d85-a120-6681412ccecc',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17880,
   'granules': ['S1A_IW_SLC__1SDV_20230204T141357_20230204T141424_047087_05A618_DCBE',
    'S1A_IW_SLC__1SDV_20230204T141421_20230204T141448_047087_05A618_B5B0'],
   'secondary_granules': ['S1B_IW_SLC__1SDV_20211123T141304_20211123T141332_029716_038BFF_D7F1',
    'S1B_IW_SLC__1SDV_20211123T141329_20211123T141356_029716_038BFF_8694'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/cd9d691c-88c2-4d85-a120-6681412ccecc/cd9d691c-88c2-4d85-a120-6681412ccecc.log'],
  'expiration_time': '2024-09-20T00:00:00+00:00',
  'processing_times': [9197.827, 9.027],
  'credit_cost': 1},
 {'job_id': '8bcb7958-bdbf-49be-ba45-f25f133e65b0',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17885,
   'granules': ['S1A_IW_SLC__1SDV_20240211T141516_20240211T141543_052512_0659F8_BE86',
    'S1A_IW_SLC__1SDV_20240211T141540_20240211T141608_052512_0659F8_7291'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230204T141511_20230204T141538_047087_05A618_DFC0',
    'S1A_IW_SLC__1SDV_20230204T141536_20230204T141604_047087_05A618_C23D'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/8bcb7958-bdbf-49be-ba45-f25f133e65b0/8bcb7958-bdbf-49be-ba45-f25f133e65b0.log'],
  'expiration_time': '2024-09-30T00:00:00+00:00',
  'processing_times': [9317.596, 8.949],
  'credit_cost': 1},
 {'job_id': 'ac0a622c-95ef-4437-a6d0-6e6b89de96a3',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17880,
   'granules': ['S1A_IW_SLC__1SDV_20240223T141401_20240223T141428_052687_065FEB_AAD8',
    'S1A_IW_SLC__1SDV_20240223T141426_20240223T141453_052687_065FEB_4048'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230123T141357_20230123T141424_046912_05A043_B3A4',
    'S1A_IW_SLC__1SDV_20230123T141422_20230123T141449_046912_05A043_05EE'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/ac0a622c-95ef-4437-a6d0-6e6b89de96a3/ac0a622c-95ef-4437-a6d0-6e6b89de96a3.log'],
  'expiration_time': '2024-09-20T00:00:00+00:00',
  'processing_times': [8338.455, 8.998],
  'credit_cost': 1},
 {'job_id': 'a504c30b-43e9-484d-8407-0dd960cf67c0',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17884,
   'granules': ['S1A_IW_SLC__1SDV_20240211T141516_20240211T141543_052512_0659F8_BE86',
    'S1A_IW_SLC__1SDV_20240211T141540_20240211T141608_052512_0659F8_7291'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230123T141512_20230123T141538_046912_05A043_1170',
    'S1A_IW_SLC__1SDV_20230123T141536_20230123T141604_046912_05A043_315F'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a504c30b-43e9-484d-8407-0dd960cf67c0/a504c30b-43e9-484d-8407-0dd960cf67c0.log'],
  'expiration_time': '2024-09-20T00:00:00+00:00',
  'processing_times': [7505.512, 8.939],
  'credit_cost': 1},
 {'job_id': 'c4865955-60d8-4a04-8c7c-e7a258fcaa14',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17887,
   'granules': ['S1A_IW_SLC__1SDV_20240211T141606_20240211T141633_052512_0659F8_DF42',
    'S1A_IW_SLC__1SDV_20240211T141631_20240211T141658_052512_0659F8_77AC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230111T141602_20230111T141629_046737_059A5B_8B05',
    'S1A_IW_SLC__1SDV_20230111T141627_20230111T141654_046737_059A5B_12EB'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/c4865955-60d8-4a04-8c7c-e7a258fcaa14/c4865955-60d8-4a04-8c7c-e7a258fcaa14.log'],
  'expiration_time': '2024-09-20T00:00:00+00:00',
  'processing_times': [8014.97, 8.876],
  'credit_cost': 1},
 {'job_id': '52bff614-7787-4c59-8a5a-b421aaf2c7b2',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-03-20T01:00:43+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'conus_west_coast-track115',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 17886,
   'granules': ['S1A_IW_SLC__1SDV_20230204T141536_20230204T141604_047087_05A618_C23D',
    'S1A_IW_SLC__1SDV_20230204T141602_20230204T141629_047087_05A618_CCEB'],
   'secondary_granules': ['S1B_IW_SLC__1SDV_20211217T141443_20211217T141510_030066_039705_D4FC',
    'S1B_IW_SLC__1SDV_20211217T141508_20211217T141535_030066_039705_A371'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/52bff614-7787-4c59-8a5a-b421aaf2c7b2/52bff614-7787-4c59-8a5a-b421aaf2c7b2.log'],
  'expiration_time': '2024-09-30T00:00:00+00:00',
  'processing_times': [9702.935, 9.418],
  'credit_cost': 1}]

Apr 02 '24 16:04 cmarshak

I will tag @scottstanie and @asjohnston-asf in case they have additional input. Thank you.

Apr 02 '24 17:04 cmarshak

Some discussion of the rate-limiting and orbit fetching in https://github.com/dbekaert/RAiDER/issues/610 may be relevant.

Apr 02 '24 17:04 jhkennedy

Some discussion of the rate-limiting and orbit fetching in https://github.com/dbekaert/RAiDER/issues/610 may be relevant.

Is our use of ASF first for the topsApp step avoiding this issue?

Apr 02 '24 17:04 cmarshak

The downloads from copernicus for these granules seem to work for me with my account/credentials

$ touch S1A_IW_SLC__1SDV_20230204T141536_20230204T141604_047087_05A618_C23D
$ touch S1A_IW_SLC__1SDV_20230204T141602_20230204T141629_047087_05A618_CCEB
$ touch S1B_IW_SLC__1SDV_20211217T141443_20211217T141510_030066_039705_D4FC
$ touch S1B_IW_SLC__1SDV_20211217T141508_20211217T141535_030066_039705_A371

$ eof
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1A on 2023-02-04
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1A on 2023-02-04
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1B on 2021-12-17
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1B on 2021-12-17
[04/02 16:26:01] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:02] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:04] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:04] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:06] [INFO download.py] Attempting download from SciHub
[04/02 16:26:08] [INFO dataspace_client.py] Orbit file downloaded to S1A_OPER_AUX_POEORB_OPOD_20230224T080742_V20230203T225942_20230205T005942.EOF
[04/02 16:26:08] [INFO dataspace_client.py] Orbit file downloaded to S1B_OPER_AUX_POEORB_OPOD_20220106T111538_V20211216T225942_20211218T005942.EOF
[04/02 16:26:09] [INFO dataspace_client.py] Orbit file downloaded to S1A_OPER_AUX_POEORB_OPOD_20230224T080742_V20230203T225942_20230205T005942.EOF
[04/02 16:26:10] [INFO dataspace_client.py] Orbit file downloaded to S1B_OPER_AUX_POEORB_OPOD_20220106T111538_V20211216T225942_20211218T005942.EOF

Apr 02 '24 20:04 scottstanie

I suspect (but haven't tested) that we may be hitting the "100 active sessions" limit documented at https://documentation.dataspace.copernicus.eu/Quotas.html

I notice sentineleof generates an authentication token but doesn't release the token. They live for 10 minutes by default, so if we were to request more than 100 tokens in ten minutes I'd expect to see errors.

You can see a list of active tokens by logging into CDSE and navigating through My Account -> Security -> Device Activity.

hyp3-lib built a class to request CDSE tokens and automatically release them when they go out of scope. If we can confirm my hypothesis then I'd recommend implementing something similar in sentineleof: https://github.com/ASFHyP3/hyp3-lib/blob/develop/src/hyp3lib/get_orb.py#L24

Apr 03 '24 01:04 asjohnston-asf

We had throughput of 1000 jobs so this is a very likely the culprit. I can't see any active sessions when I log into the email we are using for our access account, but it was at 9 this morning when the tokens were generated.

Apr 03 '24 01:04 cmarshak

Confirmed. Ran a few copies of this script in parallel:

import datetime

import eof

eof.log._set_logger_handler()

date = datetime.datetime(2024, 1, 1, 0, 0, 0)

for ii in range(101):
    print(ii)
    eof.download.download_eofs([date], ['S1A'], cdse_user='***', cdse_password='***')

Initially the downloads work fine, but they all fail with an unauthorized error after the total number of requests get to around 100:

[04/02 22:10:26] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 22:10:27] [INFO download.py] Attempting download from SciHub
Traceback (most recent call last):
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/eof/dataspace_client.py", line 301, in get_access_token
    r.raise_for_status()
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/asjohnston/src/hyp3-operations/eof_test.py", line 11, in <module>
    eof.download.download_eofs([date], ['S1A'], cdse_user='***', cdse_password='***')
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/eof/download.py", line 107, in download_eofs
    results = client.download_all(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/eof/dataspace_client.py", line 175, in download_all
    return download_all(
           ^^^^^^^^^^^^^
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/eof/dataspace_client.py", line 407, in download_all
    access_token = get_access_token(username, password)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/asjohnston/mambaforge/envs/orbits/lib/python3.12/site-packages/eof/dataspace_client.py", line 303, in get_access_token
    raise RuntimeError(f"Access token creation failed. Reason: {str(err)}")
RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token

Apr 03 '24 02:04 asjohnston-asf

It's too bad they don't return HTTP 429 -- "too many requests" here, or similar, like they do in rate-limiting situations: https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Overview/RateLimiting.html

because it'd be a lot more clear what's happening.

Apr 03 '24 18:04 jhkennedy

Hey all, sounds like this might be better handled in sentineleof than in RAiDER. @scottstanie do you think you'll implement something like what @asjohnston-asf mentioned, or should we try to do a work-around?

Apr 04 '24 02:04 jlmaurer

I agree that sounds like a good thing to add to sentineleof and thanks for pointing to the nice similar example in hyp3; however, I don't think I'll get to it for some number of weeks since I'm on paternity leave through April

Apr 04 '24 14:04 scottstanie

@scottstanie if ASF is much more stable and not rate limited (@jhkennedy and @cmarshak might have experience), perhaps an easy fix that decreases amount of calls would be to swap ESA and ASF in order. Then only restituted orbits would rely on ESA.

Apr 08 '24 20:04 dbekaert

As of now, you can skip CDSE by using the --force-asf flag. That might be the quickest solution?

$ time eof -d 2024-04-08 -m S1A --force-asf
[04/10 13:54:24] [INFO asf_client.py] Downloading all filenames from ASF (may take awhile)
...

[04/10 13:54:29] [WARNING asf_client.py] The following dates were not found: [(datetime.datetime(2024, 4, 8, 23, 0), 'S1A')]
[04/10 13:54:29] [WARNING asf_client.py] Attempting to download the restituted orbits for these dates.
[04/10 13:54:29] [INFO asf_client.py] Downloading all filenames from ASF (may take awhile)
[04/10 13:54:30] [INFO asf_client.py] Downloading https://s1qc.asf.alaska.edu/aux_resorb/S1A_OPER_AUX_RESORB_OPOD_20240409T082501_V20240408T212629_20240409T004359.EOF
[04/10 13:54:32] [INFO asf_client.py] Saving to S1A_OPER_AUX_RESORB_OPOD_20240409T082501_V20240408T212629_20240409T004359.EOF
[04/10 13:54:32] [INFO download.py] Finished https://s1qc.asf.alaska.edu/aux_resorb/S1A_OPER_AUX_RESORB_OPOD_20240409T082501_V20240408T212629_20240409T004359.EOF, saved to S1A_OPER_AUX_RESORB_OPOD_20240409T082501_V20240408T212629_20240409T004359.EOF

real	0m10.767s
user	0m0.365s
sys	0m0.059s

I'd been using the Copernicus API as default for two reasons

The stability and latency of the ASF mirror are unknown to me. If they are ~as good as CDSE
The long delay for getting the entire RESORB file list

I now see that the RESORB list on ASF has been truncated to just 2024? So it only took a little longer to run for recent orbits (10 seconds for ASF, 4 seconds for CDSE), but I'd be interested in seeing what support the ASF mirror wants to have, or if they think it's a good idea to switch to ASF by default.

Apr 10 '24 18:04 scottstanie

@scottstanie on the HyP3-side, we also prefer ESA and fall back to ASF by default -- ASF's orbit "catalog" leaves a lot to be desired, and the stability isn't as high as I'd like due to it being an on-prem dataset.

If you like, I can open a PR into sentineleof to release the token as we do in HyP3.

In the future, and hopefully, near future, ASF may entirely redo our orbit offerings and move everything to S3 with a simple search and discovery API in front. So hopefully, we can just fix these issues for the community. I can't promise a timeline right now, though.

Apr 10 '24 18:04 jhkennedy

@bbuzz31 @dbekaert - I submitted 198 jobs yesterday afternoon (after the ASF outage in the morning) using the fix from #635. Unfortunately, 153/198 of the jobs failed with a similar error as before. It is unclear if ASF orbit availability was cut off exposing the same ESA issue. Before I resubmit these jobs, we should discuss briefly here - maybe @jhkennedy or @asjohnston-asf have some insights regarding ASF availability.

Here are a sample of 3 of such jobs:

[{'job_id': '971ce4e7-829f-49d6-8975-841f3b36c966',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2024-04-18T00:39:31+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-1_71_HRRR_0417',
  'job_parameters': {'compute_solid_earth_tide': True,
   'dense_offsets': False,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 11041,
   'goldstein_filter_power': 0.5,
   'granules': ['S1B_IW_SLC__1SDV_20211027T135132_20211027T135159_029322_037FE8_7018',
    'S1B_IW_SLC__1SDV_20211027T135157_20211027T135225_029322_037FE8_1991'],
   'output_resolution': 90,
   'secondary_granules': ['S1B_IW_SLC__1SDV_20211015T135132_20211015T135159_029147_037A79_1B14',
    'S1B_IW_SLC__1SDV_20211015T135157_20211015T135225_029147_037A79_3DFD'],
   'unfiltered_coherence': True,
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-nisar-jpl-contentbucket-vjuc0xp687bi.s3.us-west-2.amazonaws.com/971ce4e7-829f-49d6-8975-841f3b36c966/971ce4e7-829f-49d6-8975-841f3b36c966.log'],
  'expiration_time': '2024-05-03T00:00:00+00:00',
  'processing_times': [8799.232, 10.148],
  'credit_cost': 1},
 {'job_id': 'df8b8688-4e38-402b-bbb1-62559f9c9ffe',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2024-04-18T00:39:31+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-1_71_HRRR_0417',
  'job_parameters': {'compute_solid_earth_tide': True,
   'dense_offsets': False,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 11042,
   'goldstein_filter_power': 0.5,
   'granules': ['S1A_IW_SLC__1SDV_20210506T135238_20210506T135305_037768_047520_2C81',
    'S1A_IW_SLC__1SDV_20210506T135303_20210506T135330_037768_047520_DD1F'],
   'output_resolution': 90,
   'secondary_granules': ['S1B_IW_SLC__1SDV_20210430T135150_20210430T135216_026697_03305A_1D60',
    'S1B_IW_SLC__1SDV_20210430T135214_20210430T135241_026697_03305A_FE92'],
   'unfiltered_coherence': True,
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-nisar-jpl-contentbucket-vjuc0xp687bi.s3.us-west-2.amazonaws.com/df8b8688-4e38-402b-bbb1-62559f9c9ffe/df8b8688-4e38-402b-bbb1-62559f9c9ffe.log'],
  'expiration_time': '2024-05-03T00:00:00+00:00',
  'processing_times': [8978.686, 11.822],
  'credit_cost': 1},
 {'job_id': '385f29ab-e827-46e6-b5ed-1703e3698b7d',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2024-04-18T00:39:31+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-1_71_HRRR_0417',
  'job_parameters': {'compute_solid_earth_tide': True,
   'dense_offsets': False,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 11041,
   'goldstein_filter_power': 0.5,
   'granules': ['S1A_IW_SLC__1SDV_20220101T135218_20220101T135246_041268_04E7A6_68E0',
    'S1A_IW_SLC__1SDV_20220101T135244_20220101T135311_041268_04E7A6_AA40'],
   'output_resolution': 90,
   'secondary_granules': ['S1A_IW_SLC__1SDV_20211220T135219_20211220T135245_041093_04E1C6_848E',
    'S1A_IW_SLC__1SDV_20211220T135243_20211220T135311_041093_04E1C6_571D'],
   'unfiltered_coherence': True,
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-nisar-jpl-contentbucket-vjuc0xp687bi.s3.us-west-2.amazonaws.com/385f29ab-e827-46e6-b5ed-1703e3698b7d/385f29ab-e827-46e6-b5ed-1703e3698b7d.log'],
  'expiration_time': '2024-05-03T00:00:00+00:00',
  'processing_times': [8616.931, 12.623],
  'credit_cost': 1}]

All such jobs can be obtained:

import hyp3_sdk
hyp3_isce = hyp3_sdk.HyP3('https://hyp3-nisar-jpl.asf.alaska.edu')

failed_jobs = hyp3_isce.find_jobs(name='Los-Angeles-1_71_HRRR_0417', status_code='FAILED', user_id='cmarshak')
dicts_failed = [j.to_dict() for j in failed_jobs if len(j.to_dict()['processing_times']) == 2]
len(dicts_failed)

Apr 18 '24 16:04 cmarshak

For clarity, here is a log from one the jobs:

 â•â”€[48;2;255;255;255m[38;2;136;33;27mâ–Œ[0m[38;2;12;53;118m[48;2;240;234;210mâ–Œ[38;2;0;0;0m[1mHerbie[0mâ”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•®
 â”‚ INFO: Created a default config file.                 â”‚
 â”‚ You may view/edit Herbie's configuration here:       â”‚
 â”‚ [38;2;255;153;0m     /home/raider/.config/herbie/config.toml      [0m   â”‚
 â•°â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•¯
Downloading s3://hyp3-nisar-jpl-contentbucket-vjuc0xp687bi/385f29ab-e827-46e6-b5ed-1703e3698b7d/S1-GUNW-D-R-071-tops-20220101_20211220-135244-00119W_00034N-PP-98b6-v3_0_1.nc to S1-GUNW-D-R-071-tops-20220101_20211220-135244-00119W_00034N-PP-98b6-v3_0_1.nc
/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_azimuth_timing.py:242: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.
  t_ceil = ts_0.floor(f'{time_step_hours}H')
/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_azimuth_timing.py:243: FutureWarning: 'H' is deprecated and will be removed in a future version, please use 'h' instead.
  t_floor = ts_1.ceil(f'{time_step_hours}H')
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2022-Jan-01 14:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2022-Jan-01 13:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2021-Dec-20 14:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
âœ… Found â”Š model=hrrr â”Š [3mproduct=nat[0m â”Š [38;2;41;130;13m2021-Dec-20 13:00 UTC[92m F00[0m â”Š [38;2;255;153;0m[3mGRIB2 @ aws[0m â”Š [38;2;255;153;0m[3mIDX @ aws[0m
Downloading s3://hyp3-nisar-jpl-contentbucket-vjuc0xp687bi/385f29ab-e827-46e6-b5ed-1703e3698b7d/S1-GUNW-D-R-071-tops-20220101_20211220-135244-00119W_00034N-PP-98b6-v3_0_1.json to S1-GUNW-D-R-071-tops-20220101_20211220-135244-00119W_00034N-PP-98b6-v3_0_1.json
[31;21mERROR: Could not download orbit from ASF, trying ESA...[0m
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/asf_client.py", line 203, in _download_and_write
    response.raise_for_status()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=BO_n7nTIlMljdvU6kRRB3g&redirect_uri=https://auth.asf.alaska.edu/login&state=https://s1qc.asf.alaska.edu/aux_poeorb/S1A_OPER_AUX_POEORB_OPOD_20220121T121549_V20211231T225942_20220102T005942.EOF&app_type=401
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_orbits.py", line 74, in download_eofs
    orb_file = eof.download.download_eofs(dt, mission, save_dir=save_dir, force_asf=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/download.py", line 138, in download_eofs
    cur_filename = result.get()
                   ^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/opt/conda/envs/RAiDER/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/asf_client.py", line 213, in _download_and_write
    response.raise_for_status()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=BO_n7nTIlMljdvU6kRRB3g&redirect_uri=https://auth.asf.alaska.edu/login&state=https://s1qc.asf.alaska.edu/aux_poeorb/S1A_OPER_AUX_POEORB_OPOD_20220121T121549_V20211231T225942_20220102T005942.EOF
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 301, in get_access_token
    r.raise_for_status()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/__main__.py", line 44, in <module>
    main()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/__main__.py", line 40, in main
    process_entry_point.load()()
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/cli/raider.py", line 559, in calcDelaysGUNW
    path_cfg, wavelength = RAiDER.aria.prepFromGUNW.main(iargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 400, in main
    GUNWObj = GUNW(args.file, args.weather_model, args.output_directory)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 6, in __init__
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 165, in __post_init__
    self.OrbitFile  = self.get_orbit_file()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/aria/prepFromGUNW.py", line 280, in get_orbit_file
    path_orb = download_eofs([dt], [sat], str(orbit_dir))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/RAiDER/s1_orbits.py", line 79, in download_eofs
    orb_file = eof.download.download_eofs(dt, mission, save_dir=save_dir, force_asf=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/download.py", line 107, in download_eofs
    results = client.download_all(
              ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 175, in download_all
    return download_all(
           ^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 407, in download_all
    access_token = get_access_token(username, password)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/RAiDER/lib/python3.12/site-packages/eof/dataspace_client.py", line 303, in get_access_token
    raise RuntimeError(f"Access token creation failed. Reason: {str(err)}")
RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token

Apr 18 '24 16:04 cmarshak

I ran a small test locally and was able to get orbits without error. Can share a script if useful.

Apr 22 '24 18:04 bbuzz31

Yes - please share in this thread. I am assuming you are doing something similar to what @asjohnston-asf did above with 100 simultaneous requests.

Apr 22 '24 18:04 cmarshak

You may meed to use multiprocessing to better hit your rate limit as RAiDER takes 20 minutes to complete so the scaling issue might not be observed locally as easily as in a serial loop. Or use a really small test dataset?

Apr 22 '24 18:04 cmarshak

Resolved by #637.

I was able to successfully run 100+ jobs through the dev branch via hyp3 after the above PR was merged.

Apr 26 '24 21:04 cmarshak