mango icon indicating copy to clipboard operation
mango copied to clipboard
verified publisher icon bigdatagenomics

Need to downgrade py4j during install

Open jpdna opened this issue 7 years ago • 4 comments

When I attempt to install from Master as below, I get the following error:

git clone [email protected]:bigdatagenomics/mango.git
virtualenv venv
source venv/bin/activate
cd mango/mango-viz/
make prepare
cd ../mango-python/
make prepare
cd ..
export SPARK_HOME=/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7
export MANGO_HOME=/Users/paschalj/test9/june14_mango_master/mango

PY4J_ZIP="$(ls -1 "${SPARK_HOME}/python/lib" | grep py4j)"
export PYTHONPATH=${SPARK_HOME}/python:${SPARK_HOME}/python/lib/${PY4J_ZIP}:${PYTHONPATH}
mvn clean package -DskipTests

ASSEMBLY_DIR="${MANGO_HOME}/mango-assembly/target"
ASSEMBLY_JAR="$(ls -1 "$ASSEMBLY_DIR" | grep "^mango-assembly[0-9A-Za-z\_\.-]*\.jar$" | grep -v javadoc | grep -v sources || true)"
export PYSPARK_SUBMIT_ARGS="--jars ${ASSEMBLY_DIR}/${ASSEMBLY_JAR} --driver-class-path ${ASSEMBLY_DIR}/${ASSEMBLY_JAR} pyspark-shell"

mvn clean package -P python -DskipTests

...
[INFO] --- exec-maven-plugin:1.6.0:exec (dev-python) @ mango-python ---
pip install -e .
Obtaining file:///Users/paschalj/test9/june26_mango/mango/mango-python
Exception:
Traceback (most recent call last):
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 194, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 354, in prepare_editable_requirement
    req.check_if_exists(use_user_site)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 930, in check_if_exists
    self.req.name
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 472, in get_distribution
    dist = get_provider(dist)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 344, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 783, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
ContextualVersionConflict: (py4j 0.10.7 (/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages), Requirement.parse('py4j==0.10.4'), set(['pyspark']))
make: *** [develop] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] mango 0.0.2-SNAPSHOT ............................... SUCCESS [  7.278 s]
[INFO] mango: CORE ........................................ SUCCESS [01:19 min]
[INFO] mango: CLI ......................................... SUCCESS [ 43.223 s]
[INFO] Mango: Assembly .................................... SUCCESS [01:00 min]
[INFO] Mango: Python visualization hooks .................. SUCCESS [01:13 min]
[INFO] Mango: Python APIs 0.0.2-SNAPSHOT .................. FAILURE [  3.535 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:28 min
[INFO] Finished at: 2018-06-26T11:16:32-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (dev-python) on project mango-python: Command execution failed.: Process exited with an error: 2 (Exit value: 2) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :mango-python

This can be resolve by doing:

pip install py4j==0.10.4

prior to running maven

jpdna avatar Jun 26 '18 15:06 jpdna

A couple questions:

  • Are you running in a virtualenv?
  • what is your python version?
  • Can you print out and report the output for your env PY4J_ZIP?

akmorrow13 avatar Jun 26 '18 16:06 akmorrow13

Yes, used a virtualenv, seen in the commands included at start of the issue.

echo $PY4J_ZIP
py4j-0.10.4-src.zip

echo $PYTHONPATH
/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7/python:/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:

python --version
Python 2.7.10

jpdna avatar Jun 26 '18 18:06 jpdna

I am also running into similar problems. Any advice yet to solve?

Cheers

ricky-lim avatar Dec 12 '18 09:12 ricky-lim

Can you please specify the specific problem you are having, as well as the Spark, Mango, and python version?

akmorrow13 avatar Dec 12 '18 15:12 akmorrow13