Toil problem on Python 3.10 / Ubuntu 22.04

[glennhickey]

I started getting an error when importing files after upgrading cactus docker images to ubuntu 22.04:

I was able to fix it by not installing toil[all] (using just toil[aws] instead) ... from the log I'd imagine the [google] extra's at fault.

Traceback (most recent call last):
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/lib/memoize.py", line 40, in new_f
    return memory[args]
KeyError: (FileJobStore(/data/js),)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cactus/cactus_env/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/progressive/cactus_progressive.py", line 406, in main
    runCactusProgressive(options)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/progressive/cactus_progressive.py", line 441, in runCactusProgressive
    project.inputSequenceIDMap[genome] = toil.importFile(seq)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/lib/compatibility.py", line 12, in call
    return func(*args, **kwargs)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1129, in importFile
    return self.import_file(srcUrl, sharedFileName, symlink)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1143, in import_file
    return self._jobStore.import_file(src_uri, shared_file_name=shared_file_name, symlink=symlink)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/jobStores/abstractJobStore.py", line 385, in import_file
    otherCls = self._findJobStoreForUrl(parseResult)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/jobStores/abstractJobStore.py", line 333, in _findJobStoreForUrl
    for jobStoreCls in self._jobStoreClasses:
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/lib/memoize.py", line 42, in new_f
    r = f(*args)
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/jobStores/abstractJobStore.py", line 314, in _jobStoreClasses
    module = import_module(moduleName)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/jobStores/googleJobStore.py", line 27, in <module>
    from google.cloud import exceptions, storage
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/google/cloud/storage/__init__.py", line 38, in <module>
    from google.cloud.storage.blob import Blob
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 54, in <module>
    from google.cloud.iam import Policy
  File "/home/cactus/cactus_env/lib/python3.10/site-packages/google/cloud/iam.py", line 38, in <module>
    class Policy(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
________________________ TestCase.testEvolverPrepareWDL ________________________
Traceback (most recent call last):
  File "/builds/comparativegenomicstoolkit/cactus/test/evolverTest.py", line 695, in testEvolverPrepareWDL
    self._run_evolver_decomposed_wdl("wdl")
  File "/builds/comparativegenomicstoolkit/cactus/test/evolverTest.py", line 248, in _run_evolver_decomposed_wdl
    subprocess.check_call(['java', '-jar', './cromwell-49.jar', 'run',
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['java', '-jar', './cromwell-49.jar', 'run', '/builds/comparativegenomicstoolkit/cactus/tmp_EQnDvG5RLQ/prepared.wdl', '-o', '/builds/comparativegenomicstoolkit/cactus/tmp_EQnDvG5RLQ/options.json']' returned non-zero exit status 1.

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1183

[adamnovak]

@glennhickey What Toil version are you using here, exactly? Might this have been fixed in #4114 which bumped the max versions of some Google modules?

[glennhickey]

I'm using 5.6.0 -- that commit almost surely fixes it. Probably the same type of story for #4138 ...

[Hexotical]

This was fixed by updating our google requirement.