linkedin
Running pivy-importer fails on Mac
I attempted to run pivy-importer on my MacBook Pro 2015, Siera v. 10.12.3, Java 1.8.0_112 and ended up with this error
MUSRS186002-942:tmp rs186002$ java -jar pivy-importer-0.4.1-all.jar --repo ~/tmp/pypi virtualenv:15.0.1 pip:7.1.2 --replace alabaster:0.7=alabaster:0.7.1,pytz:0a=pytz:2016.4,Babel:0.8=Babel:1.0,sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
alabaster:0.7=alabaster:0.7.1
pytz:0a=pytz:2016.4
Babel:0.8=Babel:1.0
sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
12:53:02.579 INFO c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:15.0.1
12:53:04.256 INFO c.l.p.i.deps.DependencyDownloader - Pulling in pip:7.1.2
12:53:04.777 INFO c.l.p.i.deps.DependencyDownloader - Pulling in pytest:3.0.6
12:53:05.196 INFO c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:1.10
12:53:05.872 INFO c.l.p.i.deps.DependencyDownloader - Pulling in scripttest:1.3
12:53:05.936 INFO c.l.p.i.deps.DependencyDownloader - Pulling in mock:2.0.0
12:53:06.671 INFO c.l.p.i.deps.DependencyDownloader - Pulling in py:1.4.29
12:53:06.825 INFO c.l.p.i.deps.DependencyDownloader - Pulling in setuptools:34.1.1
12:53:06.940 INFO c.l.p.i.deps.DependencyDownloader - Pulling in argparse:1.4.0
12:53:07.010 INFO c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.7
12:53:07.076 INFO c.l.p.i.deps.DependencyDownloader - Pulling in pbr:0.11.0
12:53:07.161 INFO c.l.p.i.deps.DependencyDownloader - Pulling in six:1.9.0
12:53:07.228 INFO c.l.p.i.deps.DependencyDownloader - Pulling in funcsigs:1.0.0
12:53:07.354 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.5.2
12:53:09.066 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.7
12:53:09.386 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Pygments:2.0
12:53:10.152 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.3
12:53:11.369 INFO c.l.p.i.deps.DependencyDownloader - Pulling in unittest2:1.1.0
12:53:11.506 INFO c.l.p.i.deps.DependencyDownloader - Pulling in pip:9.0.1
12:53:11.748 INFO c.l.p.i.deps.DependencyDownloader - Pulling in ordereddict:1.1
12:53:11.811 INFO c.l.p.i.deps.DependencyDownloader - Pulling in six:1.5.0
12:53:11.883 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.3
12:53:12.307 INFO c.l.p.i.deps.DependencyDownloader - Pulling in docutils:0.11
12:53:12.514 INFO c.l.p.i.deps.DependencyDownloader - Pulling in snowballstemmer:1.1.0
12:53:12.637 INFO c.l.p.i.deps.DependencyDownloader - Pulling in Babel:1.3
12:53:13.104 INFO c.l.p.i.deps.DependencyDownloader - Pulling in alabaster:0.7.1
12:53:13.169 INFO c.l.p.i.deps.DependencyDownloader - Pulling in imagesize:0.7.1
12:53:13.232 INFO c.l.p.i.deps.DependencyDownloader - Pulling in requests:2.4.0
12:53:13.403 INFO c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.5
12:53:13.473 INFO c.l.p.i.deps.DependencyDownloader - Pulling in nose:1.3.7
12:53:13.568 INFO c.l.p.i.deps.DependencyDownloader - Pulling in simplejson:3.10.0
12:53:13.638 INFO c.l.p.i.deps.DependencyDownloader - Pulling in html5lib:1.0b10
Exception in thread "main" java.lang.RuntimeException: Unable to find version 2.2 for chardet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at com.linkedin.python.importer.pypi.ProjectDetails.maybeFixVersion(ProjectDetails.groovy:55)
at com.linkedin.python.importer.pypi.ProjectDetails$maybeFixVersion.call(Unknown Source)
at com.linkedin.python.importer.distribution.SourceDistPackage$_parseRequiresText_closure1.doCall(SourceDistPackage.groovy:81)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:5236)
at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:615)
at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:636)
at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:595)
at org.codehaus.groovy.runtime.dgm$1060.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at com.linkedin.python.importer.distribution.SourceDistPackage.parseRequiresText(SourceDistPackage.groovy:50)
at com.linkedin.python.importer.distribution.SourceDistPackage.getDependencies(SourceDistPackage.groovy:41)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGroovyObjectGetProperty(GetEffectivePogoPropertySite.java:68)
at com.linkedin.python.importer.deps.DependencyDownloader.downloadDependency(DependencyDownloader.groovy:72)
at com.linkedin.python.importer.deps.DependencyDownloader$downloadDependency.callCurrent(Unknown Source)
at com.linkedin.python.importer.deps.DependencyDownloader.download(DependencyDownloader.groovy:50)
at com.linkedin.python.importer.ImporterCLI.main(ImporterCLI.java:73)
Ok, figured it out. Had to extend command line by adding these 2 to "--replace" flag:
chardet:2.2=chardet:2.3.0,setuptools:0.6a2=setuptools:34.1.1
But where that list is coming from? Is it hardcoded or is it red from somewhere? The former would be really unfortunate
And why this is not pulling PyYAML lib? In fact - who and how determines which libraries to pull? I see a rather short list of stuff which is for most parts is useless to me. What am I missing? Here's my command:
java -jar pivy-importer-0.4.1-all.jar --repo ~/tmp/ virtualenv:15.0.1 pip:7.1.2 --replace alabaster:0.7=alabaster:0.7.1,pytz:0a=pytz:2016.4,Babel:0.8=Babel:1.0,sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1,chardet:2.2=chardet:2.3.0,setuptools:0.6a2=setuptools:34.1.1
@bostone The pyvi-importer is just an example to get people started quickly and provides just the packages that were needed for our CI runs triggered by commits here.
Internally, at LinkedIn, we have a Python product that uploads external packages into our artifactory + Ivy files created by analyzing the dependencies. Thus, when our developers need a new package, we can easily get it (and its transitive dependencies) into our artifactory (if they are not already there).
Being a Python program, this tool is much better at analyzing the needs of Python packages than pyvi-importer and it has various options. This is a much more sophisticated program and highly customized for internal LinkedIn use, so we did not open-source it. We expect that enterprise users will be able to build something similar for their own needs. I had been contacted by some who used pyvi-importer as a guidance to build a very simple Python program that uploads the dependencies to their custom artifact repository.
If the community feels the need for a tool like this, we can look into producing a simpler open-source version of it, but the timeline for that would depend on our priorities.
For project that advertise itself as dependency resolution solution I found it strange that I have to jump over multiple hoops just to import a single library. It's not a critique but rather an observation. I really think this needs to be described in more details right away so not to build false expectation. In my personal opinion this project can only really take off if it provides easy way of plugging into official and/or Artifactory-based PyPi repo. Initially I thought that it was enough to provide some definition in the "repositories" section. Then I red in the documentation that I need to use pivy-importer and now both of these things are not (essentially) true. It's really frustrating. For light-medium Python project it seems to me that I'm better off just writing few gradle tasks that would run some python scripts and that's a pity because I really like the premise of this project
@bostone Some users have successfully used pyvi-importer for their projects. See for example the contribution from our user: https://github.com/linkedin/pygradle/tree/master/examples/iris-classification
We are open to suggestions and are working with the users to improve documentation and resolution of the pain points. We'll definitely look into providing our Python tool in some form to the community. As @sixninetynine suggested, we'll also look into providing something that will allow easier setup of a localized repo on one's development machine.
Thank you again for the comments and input. We value it, particularly because it's hard to notice these deficiencies in our environment. We use pygradle every day without issues on over thousand products, but we have our internal plugin wrapping it and adding some custom/new features and our repo populated with our tools. It's important to hear from users like you to understand where the issues can arise. Also, our users come from quite different environments with very different needs and each input is important to try and understand the generic solution that would help as many people as possible.
We can work together on these improvements.
Love that I'm getting response from you guys. The initial setup was really easy and impressive until I hit the wall with one simple dependency. I think at this point this is a weakest link. Naturally, I was really disappointed that I cannot easily import from PyPi. How about adding automation that behind the scenes sets local ivy repo and pulls dependencies from PyPi based on what user specifies in the "dependencies" section? How far fetch would that be? I'll look at the example you pointed to
Thanks again. Bo
How about adding automation that behind the scenes sets local ivy repo and pulls dependencies from PyPi based on what user specifies in the "dependencies" section? How far fetch would that be?
Not far-fetched at all! Our internal tool runs a process very similar to this :)
@sixninetynine I forked the project and am working on externalizing list of "packagesToInclude" for pivy-importer. Here's question for you. From the hardcoded list which packages are actually required for pygradle build to work. I'm guessing anything related to virtualenv, pip, setuptools, flake8, pytest, wheel. But what about the rest? I much rather keep the default list and append the dependencies dynamically. Here's what you guys have by default:
def packagesToInclude = ['virtualenv:15.0.1', 'pip:7.1.2', 'setuptools:19.1.1', 'setuptools-git:1.1',
'flake8:2.4.0', 'flake8:2.5.4', 'pytest:2.9.1', 'pytest-cov:2.2.1', 'pytest-xdist:1.14',
'wheel:0.26.0', 'pbr:1.8.0', 'Sphinx:1.4.1', 'six:1.10.0', 'pex:1.1.4',
'requests:2.10.0', 'Flask:0.11.1', 'docutils:0.12', 'appdirs:1.4.0',
'packaging:16.8'].join(" ")
I fixed pyvi-importer behavior in #138.
Also added --latest option and --pre option to allow installation of the latest (highest) supported version of dependency and to allow pre-releases (alpha, beta, rc) which are disallowed by default in the new version.
Can you try using the latest version @bostone (0.6.1 was just released).
Regarding the question what's required -- only the the dependencies we need for the build, and Flask + requests for our examples (we'll consider dropping these two). Otherwise, we bring the dependencies that particular Python project needs and specifies in its build.gradle file. The list is now quite minimal in 0.6.x.