`requirements.txt` doesn't support Python 2

[wescpy]

The requirements.txt file is not Python 2 compatible. The following errors appear when executing the directions in the README to deploy this as a Python 2 app to App Engine.

$ pip2 install -t lib -r requirements.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
ERROR: Could not find a version that satisfies the requirement Flask==2.1.0 (from -r requirements-orig.txt (line 1)) (from versions: 0.1, 0.2, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.5.2, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.9, 0.10, 0.10.1, 0.11, 0.11.1, 0.12, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4)
ERROR: No matching distribution found for Flask==2.1.0 (from -r requirements.txt (line 1))

The reason for the error is that specific version numbers are hardcoded into requirements.txt and most if not all of them only work with Python 3. There are 2 possible fixes, so pick one:

1. Remove all version numbers from all packages in requirements.txt so the latest versions are installed regardless of whether the user runs Python 2 or 3.
2. Leave the existing requirements.txt alone, meaning geared towards Python 3 users and create an alternative named requirements2.txt with the version numbers either stripped out or locked down to the final versions available to Python 2. Then update the instructions for Python 2 users to execute: pip2 install -t lib -r requirements2.txt instead. (This suggestion is modeled off of having app.yaml for Python 3 users and app27.yaml for Python 2 users. [I suppose requirements27.txt also suffices.])

Regardless of which option is chosen, there is also a requirement to explicitly list grpcio in either requirements.txt (option 1) or requirements2.txt (option 2). The reason for this is that without list it, grpcio, v1.41 is installed in lib, and pip install fails towards the end with a conflict:

$ pip2 install -t requirements2-bad.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting flask
  Using cached Flask-1.1.4-py2.py3-none-any.whl (94 kB)
Collecting google-cloud-pubsub
  Using cached google_cloud_pubsub-1.7.2-py2.py3-none-any.whl (144 kB)
. . .
. . .
. . .
Collecting pyasn1>=0.1.3
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Using legacy 'setup.py install' for grpcio, since package 'wheel' is not installed.
Installing collected packages: click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, flask, six, protobuf, enum34, googleapis-common-protos, setuptools, pytz, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, pyparsing, packaging, urllib3, certifi, chardet, idna, requests, futures, grpcio, google-api-core, grpc-google-iam-v1, google-cloud-pubsub
    Running setup.py install for grpcio ... done
ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
google-cloud-ndb 1.11.1 requires grpcio<1.40dev; python_version < "3.0", but you'll have grpcio 1.41.1 which is incompatible.

That error means that one of the other packages (flask or google-cloud-pubsub) has some dependency leading to grpcio v1.41.1, and it appears that it has to be older than v1.40dev (per output above), presumably this is the boundary of Python 2 vs. 3 compatibility.

Furthermore, once you add that extra requirement and pip install succeeds, as does deployment to App Engine, there's another issue using Pub/Sub (its client library) with the Python 2 runtime whereby an AttributeError exception is thrown when doing transport cleanup (close()), so I filed a bug against that separately.

[leahecole]

Hey @wescpy ! Which sample is this for? I can comment on a few of your points and explain some further reasoning but if you point to the sample in general that can help us with our strategy.

I will say that pretty confidently we won't be using option 1 - we consider pinning requirements.txt to be best practice because it leaves no question as to which version of each dependency is being installed, and it also makes it easier for us to debug when maintaining. Additionally, it enables us to utilize our bot friends to keep things up to date and to test samples on latest dependency versions, and correct them if a dependency causes a test failure. For some requirements files, we do use requirement specifiers to pin a specific version of a dependency for a particular version or range of python versions. For others, like with Cloud Composer, we may include a constraints.txt file as part of the samples installation (this is less common, but it's a recommendation of Airflow so I use it 🤷🏻‍♀️ )

Regarding option 2, I'll say it probably depends which sample this refers to. Python 2 is deprecated in the vast majority of products and we have not supported it in current versions of our samples for awhile. There are a few exceptions but they are few and far between.

I also want to cc @dandhlee if he has anything else he wants to add - he is now the python samples fearless leader 😁 🐍

[dandhlee]

Hey @wescpy! I think the issue here might just be that Renovate(Mend now?) might have accidentally upgraded the versions when it shouldn't. If you could point towards which specific sample it is, we can backtrack and find versions and update to the versions we should have it pointed towards.

For example:

Flask == 2.0 ; python_version > '3.6'
Flask == 1.4 ; python_version < '3.7'

is ideal. However, Renovate has been ignoring this to update for all python versions instead:

Flask == 2.1 ; python_version > '3.6'
Flask == 2.1 ; python_version < '3.7' # Should not touch this :/

Sometimes they get overlooked and easy to miss if there's >30 files being touched :/

[wescpy]

Heya, thanks for the speedy reply. Apologies I left out which sample... I'm talking about the Cloud Pub/Sub sample for App Engine.

Hope that helps. Addressing Leah's & Dan's comments (some of this I should've put in the OP too):

• @dandhlee : I'm sure the tool is doing the right thing because Py3 is primary, and it shouldn't be expected otherwise. That said, an alternative needs to be available for Py2 developers.
• @leahecole :
  • Agree most Py2 samples/support s/b waning, but because of the App Engine legacy runtime (public) commitment, explicitly including Python 2, those/these samples have an extended lifetime vs. std samples.
  • This is illustrated by the fact that this specific sample was tweaked to support both Py2 & 3 relatively recently.
  • Per the 2nd bullet in the OP, this sample already has an app27.yaml, so it makes sense to have a requirements27.txt as well as updating the README's instructions for 2.x devs, and if locking specific versions is the practice, then that file should lock-in the final 2.x versions available.
• I could be wrong but don't believe google-api-python-client is used in this sample and can be removed from requirements*.txt.
• CC @engelke since I think he wrote this sample.
• There's also a bug in the Py2 instructions anyway. If we add requirements27.txt, those instructions s/b updated to remove any existing lib as well as include "install":

$ rm -rf lib
$ pip install -t lib -r requirements27.txt  # or `pip2` if both 2.x & 3.x are installed
$ gcloud app deploy app27.yaml

[dandhlee]

The tool shouldn't try to update dependencies to versions that's not compatible for Py2 if we've specified it, however I can understand the technical limitations around it. Thus, it's been up to the reviewer to ensure that the restrictions are respected and edited.

The specified sample is specific for Python3, so I wouldn't expect to be able to run it in Python2. For all other AppEngine samples that should be able to run in Python2, I'll work with Charlie and other sample owners to fix it appropriately.

[wescpy]

Hey Dan, per the PR, this sample was explicitly made to support both Py2 and 3.... It used to be 3.x-only but that PR was to "backport" it to 2.7 so as to support both simultaneously. Regardless, I'll let you and Charlie to take care of it. I'm working on related content to help App Engine users move off its legacy APIs like pull tasks (TaskQueue) and move to Cloud Pub/Sub, and having a working Pub/Sub GAE sample for both 2.x & 3.x audiences would greatly help them move both to Pub/Sub as well as to Python 3 as soon as possible. Thanks!

[dandhlee]

Ah, got it! Thanks for the update. I'll try to find some time this week to get it working.