Incompatibility with python3.10

[sssangha]

In the rangecoreg step, topsApp crashes with the following error message:

....
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/iscesys/C
omponent/Component.py", line 179, in __call__
    return getattr(self, self.__class__.__name__.lower())(*args)
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/mroipac/a
mpcor/Ampcor.py", line 335, in ampcor
    self.setState()
  File "/u/trappist-r0/ssangha/conda_installation/stable_mach14_2022/envs/ARIA-tools/lib/python3.10/site-packages/isce/components/mroipac/a
mpcor/Ampcor.py", line 484, in setState
    ampcor.setImageDataType1_Py(str(self.imageDataType1))
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

As documented in other cases (https://stackoverflow.com/questions/70705404/systemerror-py-ssize-t-clean-macro-must-be-defined-for-formats), this error results from isce not being compatible with python3.10.

However, I've confirmed this error is circumvented with using an earlier version (python 3.9.10). So until this compatibility issue is resolved, <python3.10 dependencies should be enforced.

[sssangha]

While I suggested a change to the readme in PR #459 , we need to also impose a hard constraint for now in the conda distribution, with which I experienced these issues

[rtburns-jpl]

I pushed a commit to https://github.com/isce-framework/isce2/pull/424 to try to fix this, can you try again on that branch and see if it works now?

[yunjunz]

I got the similar SystemError: PY_SSIZE_T_CLEAN as below while running dem.py using conda installed isce2 in python 3.10.

dem.py -a stitch -b 37 40 -119 -114 -r -s 1 -c
...
API open (WR): demLat_N37_N40_Lon_W119_W114.dem.wgs84
Traceback (most recent call last):
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/applications/dem.py", line 171, in <module>
    sys.exit(main())
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/applications/dem.py", line 141, in main
    demImg = ds.correct()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/DemStitcher.py", line 862, in correct
    return cg(image,conversionType) if image else cg(self._image,conversionType)
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 182, in __call__
    self.correct_geoid_i2_srtm()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 196, in correct_geoid_i2_srtm
    self.setState()
  File "/home/zyunjun/tools/miniconda3/envs/insar/lib/python3.10/site-packages/isce/components/contrib/demUtils/Correct_geoid_i2_srtm.py", line 239, in setState
    correct_geoid_i2_srtm.setGeoidFilename_Py(self.geoidFilename)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

I tried to add #define PY_SSIZE_T_CLEAN to correct_geoid_i2_srtmmodule.cpp#L30, then the following Segmentation fault error came out instead:

API open (R): ./demLat_N37_N40_Lon_W119_W114.dem
API close:  ./demLat_N37_N40_Lon_W119_W114.dem
Writing geotrans to VRT for ./demLat_N37_N40_Lon_W119_W114.dem
GDAL open (R): ./demLat_N37_N40_Lon_W119_W114.dem.vrt
API open (WR): demLat_N37_N40_Lon_W119_W114.dem.wgs84
Segmentation fault

[EJFielding]

I hit this problem with a Python 3.10 installed by Anaconda also. Is there a way to define the required PY_SSIZE_T_CLEAN globally in all of ISCE2 to make it compatible with Python 3.10? Or does it require other changes?

[rtburns-jpl]

If you're using cmake, you can try export CFLAGS=-DPY_SSIZE_T_CLEAN and export CXXFLAGS=-DPY_SSIZE_T_CLEAN before running cmake configure, that should set those macros globally for all C/C++ code. Is that sufficient to make it compatible?

[EJFielding]

I ran into this problem in the NISAR On-demand system where the environment setup notebook installs ISCE2 (and ISCE3) with Anaconda and a yml file that lists isce2 and all the dependencies. Anaconda defaults to Python 3.10 now.

[rtburns-jpl]

@yunjunz I think I figured out why you're getting a segfault here. Python3.10 changed the size type, so any time we use PyArg_ParseTuple with a 's#' argument we need to change the result var from an int to a Py_ssize_t. Can you check if defining PY_SSIZE_T_CLEAN along with this patch this fixes your dem.py workflow?

--- a/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
+++ b/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
@@ -172,12 +172,13 @@ PyObject * setConversionType_C(PyObject* self, PyObject* args)
 PyObject * setGeoidFilename_C(PyObject* self, PyObject* args)
 {
     char * varChar;
-    int  var;
+    Py_ssize_t var;
     if(!PyArg_ParseTuple(args, "s#", &varChar ,&var))
     {
         return NULL;
     }
-    setGeoidFilename_f(varChar,&var);
+    int ivar = var;
+    setGeoidFilename_f(varChar, &ivar);
     return Py_BuildValue("i", 0);
 }

I'll try to fix the other occurrences of this in isce2 to hopefully get us working with python 3.10.

[yunjunz]

Thank you for fixing @rtburns-jpl. I was trying to test it, but there is a problem while downloading SRTM from the USGS website, so I could not confirm if the fix works or not.

[yuankailiu]

Thank you for fixing @rtburns-jpl. I was trying to test it, but there is a problem while downloading SRTM from the USGS website, so I could not confirm if the fix works or not.

Yeh, @yunjunz, right now, it shows an URL error for me

curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N4
1W125.SRTMGL1.hgt.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   299    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 503
2022-07-21 12:48:57,859 - isce.contrib.demUtils.DemStitcher - WARNING - There was a problem in retrieving the file  http://e4ftl01.cr.
usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N41W125.SRTMGL1.hgt.zip. Exception

[rtburns-jpl]

That's unfortunate. I thought this was a temporary outage but maybe their download url format has changed? Should we open a separate issue for that?

[EJFielding]

If they changed the SRTM download URLs, then I would recommend a new issue. I was going to open one earlier this week but then it worked for me again at least once. I did not find any announcements about changes but they tend to change it about every two years.

[AhmetMA]

If they changed the SRTM download URLs, then I would recommend a new issue. I was going to open one earlier this week but then it worked for me again at least once. I did not find any announcements about changes but they tend to change it about every two years.

A recently updated thread regarding the connectivity issues: https://forum.earthdata.nasa.gov/viewtopic.php?t=2262#p11891

[sisi-ali]

I have had this error for about a month now. Has anyone encountered this error before and solved it

[EJFielding]

I have had this error for about a month now. Has anyone encountered this error before and solved it

Hi sisi-ali,

Yes, we are all getting similar errors. It seems the Land Processes DAAC servers are overloaded for a long time. It works occasionally.

[sisi-ali]

thank for reply me But I can't download even though I'm always trying. You have no solution for this problem. I really need this problem to be solved so that I can move forward with my work

[yunjunz]

@yunjunz I think I figured out why you're getting a segfault here. Python3.10 changed the size type, so any time we use PyArg_ParseTuple with a 's#' argument we need to change the result var from an int to a Py_ssize_t. Can you check if defining PY_SSIZE_T_CLEAN along with this patch this fixes your dem.py workflow?

--- a/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
+++ b/contrib/demUtils/correct_geoid_i2_srtm/bindings/correct_geoid_i2_srtmmodule.cpp
@@ -172,12 +172,13 @@ PyObject * setConversionType_C(PyObject* self, PyObject* args)
 PyObject * setGeoidFilename_C(PyObject* self, PyObject* args)
 {
     char * varChar;
-    int  var;
+    Py_ssize_t var;
     if(!PyArg_ParseTuple(args, "s#", &varChar ,&var))
     {
         return NULL;
     }
-    setGeoidFilename_f(varChar,&var);
+    int ivar = var;
+    setGeoidFilename_f(varChar, &ivar);
     return Py_BuildValue("i", 0);
 }

I'll try to fix the other occurrences of this in isce2 to hopefully get us working with python 3.10.

@rtburns-jpl with https://github.com/isce-framework/isce2/pull/560 being merged, I confirmed dem.py works fine under python 3.10 with the changes you laid out. Thank you and cheers!

[EJFielding]

@sisi-ali Please get the new version of ISCE2. We have finally found the necessary change to the DEM downloading code in #560.

[sisi-ali]

Hi everyone I want to calculate the closure_phase_bias.py using the MintPy software that I per-processed on ISCE, but I cant calculate with (https://github.com/insarlab/MintPy/blob/main/mintpy/closure_phase_bias.py) I would be grateful if you could guide me

[EJFielding]

@sisi-ali It would be better to ask on the MintPy discussion site https://github.com/insarlab/MintPy/discussions or on their Google Group.

[sisi-ali]

hi dear Eric Fielding

I want to calculate the closure_phase_bias.py using the MintPy software that I per-processed on ISCE, but I cant calculate with ( https://github.com/insarlab/MintPy/blob/main/mintpy/closure_phase_bias.py) I would be grateful if you could guide me

‫‪Eric Fielding‬‏ @.***‬‏> در تاریخ پنجشنبه ۱۱ اوت ۲۰۲۲ ساعت ۲:۵۳ نوشت:‬

@sisi-ali https://github.com/sisi-ali Please get the new version of ISCE2. We have finally found the necessary change to the DEM downloading code in #560 https://github.com/isce-framework/isce2/pull/560.

— Reply to this email directly, view it on GitHub https://github.com/isce-framework/isce2/issues/458#issuecomment-1211344296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ3AKUSOWIULYCD26XTE3T3VYQTW7ANCNFSM5QX4THTA . You are receiving this because you were mentioned.Message ID: @.***>

[krive050]

Hello,

I running ISCE 2.6.1. The topsApp.py crashes at the DEM steps when it is trying to convert to DEM.

I am getting the same error "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats"

Is anyone able to help me with this? Thanks!

[EJFielding]

@krive050 The easy workaround is to change your Python to version 3.9 instead of version 3.10.

[rtburns-jpl]

@krive050 Please check if https://github.com/isce-framework/isce2/pull/534 resolves your issue. I will merge that PR once I have verification that any workflow is fixed by it.

[JolinnaYoung]

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

[EJFielding]

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

Hi Jolinna, you should upgrade to ISCE2 version 2.6.3. It has been updated to work with Python 3.10.

[JolinnaYoung]

@yunjunz Hi Dr.Yunjun! I met the same question when using dem.py with Python's version = 3.10. I've tried to add #define PY_SSIZE_T_CLEAN to [correct_geoid_i2_srtmmodule.cpp#L30] but nothing changed at all. Could you please give me hints on solving this problem? Many thanks!

Hi Jolinna, you should upgrade to ISCE2 version 2.6.3. It has been updated to work with Python 3.10.

Hi Eric, thanks for your kind reply! I'll try it!