python-medikit
Additional features for pip dependencies: uniform logging, private repos and comes_from info
Hello, in our project we have been using Medikit for a while, but we were missing some features which were useful for us. With this aim, we created a fork. These changes are now included in this PR for discussion, improvement, and ideally for potential inclusion in your main project if you find them useful.
Added features
These are the newly added features:
Uniform dependencies
Due to the nature of our project, we wanted to have different dependencies for different extras, but in some cases, we needed to have the same versions for all the extras. With this aim, we have added the parameter python.use_uniform_requirements. When True, for a certain requirement, finds a valid version for all the extras using it (including inherited requirements) and sets all of them for all the requirements*.txt files implied. We are not considering setup.py for this feature, to keep some flexibility after the process.
Addition of private repositories
It is now possible to use private repositories using git, for instance including in the python add_requirements something like: requirement @ git+ssh://[email protected]/user_name/requirement_2.git@{checkpoint_or_tag}, which will be then translated into equivalent requirements in the setup.py and requirements*.txt files. More information
Show the requirement which inherits a certain requirement
Just a visualization feature. When python.show_comes_from_info is True, presents a comment with the generated requirements, indicating the requirement which caused the installation of a dependency, to understand better the changes.
Extended information
New parameters provided:
-
python.use_uniform_requirements: If True, requirements are set to be the same for all the extras when there are coincidences, directly or on the inherited dependencies. -
python.show_comes_from_info: If True, information about the origin of a dependency is shown.
Valid requirement examples:
boto3 ~=1.10,
requirement_1 @ git+https://{token}@github.com/user_name/requirement_1.git@{checkpoint_or_tag},
requirement_2 @ git+ssh://[email protected]/user_name/requirement_2.git@{checkpoint_or_tag},
Projectfile usage example:
# Test (see github.com/python-medikit)
from medikit import require
NAME = 'Test'
with require('python') as python:
python.setup(
name=NAME,
description='This is just a test',
license='Apache License, Version 2.0',
url='http://www.an-example.com/',
download_url='http://www.an-example.com/',
author='An author',
author_email='[email protected]',
)
# New parameters are configured as follows
python.use_uniform_requirements = True
python.show_comes_from_info = True
# You can try commenting/uncommenting the following to see the behavior
python.add_requirements(
'boto3 ~=1.10',
# f'httpie @ git+https://github.com/httpie/[email protected]',
dev=[
'boto3 ~= 1.11',
# 'httpie >= 2.2.0',
# f'httpie @ git+https://github.com/httpie/[email protected]',
],
extra1=[
'boto3 == 1.12',
# f'httpie @ git+https://{token}@github.com/httpie/httpie.git@5945845420c019db81e8c09f0ce174a7eeebcdd8',
f'httpie @ git+https://github.com/httpie/[email protected]',
# 'httpie < 2.4.0'
],
extra2=[
'boto3 ~=1.11',
# f'httpie @ git+https://github.com/httpie/[email protected]',
],
extra3=[
# 'boto3 == 1.12',
# 'httpie == 2.2.0',
]
)
with require('git') as git:
pass
with require('make') as make:
pass
# vim: ft=python:
- In a git repository, if there are submodules, it will explore them along all of them
- If
use_uniform_requirements== True, requirement versions will be unified among all the extras. NOTE: In case a private repository version is mixed with a different version from another repository, or even with a packaged version (included inherited versions), an IncompatibleRequirements exception will be triggered.
For instance, the example above will become:
cat requirements.txt
-e .
boto3==1.12.0
botocore==1.15.49 # From: boto3==1.17.45
jmespath==0.10.0 # From: boto3==1.17.45
python-dateutil==2.8.1 # From: botocore==1.20.45
s3transfer==0.3.6 # From: boto3==1.17.45
six==1.15.0 # From: python-dateutil==2.8.1
urllib3==1.25.11 # From: botocore==1.20.45
cat requirements-dev.txt
-e .[dev]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49 # From: boto3==1.17.45
jmespath==0.10.0 # From: boto3==1.17.45
python-dateutil==2.8.1 # From: botocore==1.20.45
s3transfer==0.3.6 # From: boto3==1.17.45
six==1.15.0 # From: python-dateutil==2.8.1
urllib3==1.25.11 # From: botocore==1.20.45
cat requirements-extra1.txt
-e .[extra1]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49 # From: boto3==1.12
certifi==2020.12.5 # From: requests==2.25.1
chardet==4.0.0 # From: requests==2.25.1
docutils==0.15.2 # From: botocore==1.15.49
git+https://github.com/httpie/[email protected]
idna==2.10 # From: requests==2.25.1
jmespath==0.10.0 # From: boto3==1.12
pygments==2.8.1 # From: httpie@ git+https://github.com/httpie/[email protected]
python-dateutil==2.8.1 # From: botocore==1.15.49
requests==2.25.1 # From: httpie@ git+https://github.com/httpie/[email protected]
s3transfer==0.3.6 # From: boto3==1.12
six==1.15.0 # From: python-dateutil==2.8.1
urllib3==1.25.11 # From: botocore==1.15.49
cat requirements-extra2.txt
-e .[extra2]
-r requirements.txt
boto3==1.12.0
botocore==1.15.49 # From: boto3==1.17.45
jmespath==0.10.0 # From: boto3==1.17.45
python-dateutil==2.8.1 # From: botocore==1.20.45
s3transfer==0.3.6 # From: boto3==1.17.45
six==1.15.0 # From: python-dateutil==2.8.1
urllib3==1.25.11 # From: botocore==1.20.45
cat setup.py
# Generated by Medikit 0.8.0 on 2021-04-06.
# All changes will be overriden.
# Edit Projectfile and run “make update” (or “medikit update”) to regenerate.
from setuptools import setup, find_packages
from codecs import open
from os import path
here = path.abspath(path.dirname(__file__))
# Py3 compatibility hacks, borrowed from IPython.
try:
execfile
except NameError:
def execfile(fname, globs, locs=None):
locs = locs or globs
exec(compile(open(fname).read(), fname, "exec"), globs, locs)
# Get the long description from the README file
try:
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()
except:
long_description = ''
# Get the classifiers from the classifiers file
tolines = lambda c: list(filter(None, map(lambda s: s.strip(), c.split('\n'))))
try:
with open(path.join(here, 'classifiers.txt'), encoding='utf-8') as f:
classifiers = tolines(f.read())
except:
classifiers = []
version_ns = {}
try:
execfile(path.join(here, 'Test/_version.py'), version_ns)
except EnvironmentError:
version = 'dev'
else:
version = version_ns.get('__version__', 'dev')
setup(
author='An author',
author_email='[email protected]',
description='This is just a test',
license='Apache License, Version 2.0',
name='Test',
version=version,
long_description=long_description,
classifiers=classifiers,
packages=find_packages(exclude=['ez_setup', 'example', 'test']),
include_package_data=True,
install_requires=['boto3 ~= 1.10'],
extras_require={
'dev': ['boto3 ~= 1.11'],
'extra1': [
'boto3 == 1.12',
'httpie @ git+https://github.com/httpie/[email protected]'
],
'extra2': ['boto3 ~= 1.11']
},
url='http://www.an-example.com/',
download_url='http://www.an-example.com/'.format(version=version),
)
We hope you find these features interesting. Waiting for your comments.
Pull Request Test Coverage Report for Build 529
- 35 of 66 (53.03%) changed or added relevant lines in 1 file are covered.
- No unchanged relevant lines lost coverage.
- Overall coverage increased (+0.5%) to 47.116%
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
|---|---|---|---|
| medikit/feature/python.py | 35 | 66 | 53.03% |
| <!-- | Total: | 35 | 66 |
| Totals | |
|---|---|
| Change from base Build 514: | 0.5% |
| Covered Lines: | 1015 |
| Relevant Lines: | 1936 |
💛 - Coveralls
Pull Request Test Coverage Report for Build 523
- 12 of 68 (17.65%) changed or added relevant lines in 1 file are covered.
- 7 unchanged lines in 1 file lost coverage.
- Overall coverage decreased (-2.0%) to 44.619%
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
|---|---|---|---|
| medikit/feature/python.py | 12 | 68 | 17.65% |
| <!-- | Total: | 12 | 68 |
| Files with Coverage Reduction | New Missed Lines | % |
|---|---|---|
| medikit/feature/python.py | 7 | 45.7% |
| <!-- | Total: | 7 |
| Totals | |
|---|---|
| Change from base Build 514: | -2.0% |
| Covered Lines: | 976 |
| Relevant Lines: | 1938 |