containers
Trigger automated checks to keep python-podman aligned with libpod defined interfaces
Summarize
In the sake of compatibilty in libpod and python-podman I propose to design and implement automated checks to verify if python-podman implementation is fully compatible with libpod varlink defined interface.
The goal of this ticket is to discuss about this feature design and watch how we could implement this and the needs behind this (impact on project workflow, how to proceed etc...).
Draft design
Project lifecycle
I think we need to create a git branch dedicated for each libpod version that we want to maintain to avoid to introduce future features into wrong branches and python-podman releases, in other word we need to isolate development versions.
The master branch will only reflect the libpod master and will help us to synchronize changes between future latest versions only.
I think we need as many version of python-podman than existing libpod versions to keep the python-api compatible with libpod defined interface.
Triggering automated checks
The automated check can be trigger from:
- the libpod side when changes are pushed on https://github.com/containers/libpod/blob/master/cmd/podman/varlink/io.podman.varlink
- the python-podman side daily by triggering cronjob on each opened branches
The automated check need to reflect the relation between version on the both projects, we only need to compare the varlink defined interface in the right libpod tag version and on the right development branch version on python-podman.
If we choose to trigger automated checks on python-podman then we need to pull from the right libpod tag/branch the raw varlink defined interface file (by using http request) and then compare it the python-podman implementation.
How to compare interface with implementation
For the moment I see some solutions to compare if these projects are synchronized, lets describe them.
Maybe you will see other/better solutions to manage this.
Don't forget that the main goal of this issue is to discuss with you if we can do this and how to do this before starting to spend time on it. Also I want to raise pro and con that I may miss.
First solution
The main idea in this solution is to compare function signatures.
Steps needed in order:
- download the raw interface description form libpod repo
- parse the previous download file to extract defined signature
- parse the function implementation available in
python-podman - compare the both and observe if delta exists
- raise error if delta exist
- automatically open a new github python-podman issue with attached job output to inform us and then fix the issue
The main difficulty here is how to ignore perifical function which are part of the python-podman mechanisms and not related to interface implementation... Since python-podman is only compatible with python3 we could by example use the powerful python inspect module coupled with a standardized mechanism which will allow us to analyze keywords in functions's docstrings to detect if the function is part of the interface definition, determine which interface the implementation is linked and if signature need to be compared with interface definition.
All these things can be done in pure python without extra module and dependencies (or just to some mainstream requirements).
We don't need to compile and install libpod here, we just need to refer to the related interface.
I think this one is the more preferable solution and the more safer and the less painful.
Second solution
On travis's instance compile and install the right libpod version and then start its varlink server, and then run unit tests against but I think it's more likely that libpod's varlink defined interface will be ahead of the python-podman implementation. In this case it can be difficult to witness raised exceptions from the python-podman side because the python-podman would not pass undefined params (except if some params are removed from libpod and not yet removed from python-podman).
Maybe I'm wrong by thinking this way, I'm not sure if all params are mandatory and if exceptions can be raised by doing this. If exception are raised then it's can be a way to identify non synchonized interface implemention in python-podman.
My main concern about this solution is how to properly manage travis instance, libpod rights (sudo etc...), rootless needs, etc... on a travis's instance image which could have some limitations. I think it can be painful to follow this way and our goal is not to test libpod but just to observe if implementation is rightly synchronized, I don't want to fight with windmills here.
Third solution
By adding new capabilities to the varlink's mocking feature it will allow us to generate a fake service by lookup form the downloaded raw interface to generate some fake python classes (rather than generating fake interface definition from fake python class as is currently the case).
Then we can mix solution 1 and solution 2 approachs (if enough unit test exists for solution 2), and by example compare generated function's signatures and real implementation signatures.
Like has explained in the second solution I'm not sure if 100 percent of the args are mandatories and if some are missing I'm not sure we will catch exception and error. Anyway with this approach libpod stay ahead of the python-podman implementation so this is the main risk.
This is a cross project solution which ask to us to adding new feature on the varlink side and in a second time to implement the check on python-podman.
The main advantage here is to provide feature which can be reused outside of the scope podman (the feature added to varlink).
Conclusion
I think we need this kind of feature to avoid to spend time fix issue unsynchronized implementation.
Also I think these changes can help us to better handle the support of libpod's versions on python-podman and to more often and more merely release new versions of this API.
Do not hesitate to react to my proposal by sending comments here.
Possibly related to
- #69
- #64
- #14
- #15
A quick-and-dirty option would just be to pull out all the method names from the varlink schema and run an identifier search over the python source (especially libs).
If the identifier doesn't appear, you have pretty high confidence the feature isn't implemented. (Since afaik you're not doing dynamic things.)
If it does appear, you can be mildly certain it is implemented.
You could probably hack an initial version with a regex and a pile of greps. Unless you have a varlink schema library lying around.
I think we should have a discussion on these features. I love seeing python-podman further develop.
We need to coordinate with the APIv2 work that is going on in Podman.
https://github.com/containers/libpod/tree/apiv2
The bottom line we are moving away from varlink to a similar Protocol to the Docker.API.
A quick-and-dirty option would just be to pull out all the method names from the varlink schema and run an identifier search over the python source (especially
libs).If the identifier doesn't appear, you have pretty high confidence the feature isn't implemented. (Since afaik you're not doing dynamic things.)
If it does appear, you can be mildly certain it is implemented.
You could probably hack an initial version with a regex and a pile of greps. Unless you have a varlink schema library lying around.
Yep I agree it can be a first version quickly released, however, we still don't check if defined and implemented signatures and we risk to face this kind of issue #64, so we need to move further in a second time to cover this part too.
I think we should have a discussion on these features. I love seeing python-podman further develop. We need to coordinate with the APIv2 work that is going on in Podman. https://github.com/containers/libpod/tree/apiv2 The bottom line we are moving away from varlink to a similar Protocol to the Docker.API.
Thanks Daniel for the heads up, I will take a look to the APIv2 to avoid to spend time on something who will be deprecated in the next releases.