marvin-python-toolbox icon indicating copy to clipboard operation
marvin-python-toolbox copied to clipboard

Adding support to build docker images with the engine-httpserver running as default

Open lucasbm88 opened this issue 8 years ago • 1 comments

Now we have a CLI to build docker images.

This fixes issue #35

This first version is able to build two type of docker images:

  • base - This is the base docker image that should be extended by any other marvin image.
  • spark - This is a docker image that uses marvin base and adds spark on top of it. Ideally should be used by engines that are spark-dependent.

Requirements:

  • The host machine should have docker installed.
  • The user running marvin CLI should be part of docker group (to avoid running with sudo). See https://docs.docker.com/engine/installation/linux/linux-postinstall/

How to test:

  • Clone this branch
  • Generate a new project: marvin engine-generate
  • Customize with your code (optional)
  • Build the docker image: marvin engine-dockerbuild (run --help and play around with parameters)
  • Run the image just build: docker run -it marvinai/marvin-spark
  • Call the REST pipeline endpoint and check that everything is working fine.

Custom spark configuration:

  • If you want to use custom spark files, replace the base files in <engine_home>/docker/marvin-spark-docker/spark-conf/* and build the image.

lucasbm88 avatar Nov 29 '17 19:11 lucasbm88

Codecov Report

Merging #37 into master will increase coverage by 0.75%. The diff coverage is 23.07%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #37      +/-   ##
==========================================
+ Coverage   49.57%   50.33%   +0.75%     
==========================================
  Files          28       25       -3     
  Lines        2104     2110       +6     
  Branches      210      211       +1     
==========================================
+ Hits         1043     1062      +19     
+ Misses       1057     1044      -13     
  Partials        4        4
Impacted Files Coverage Δ
marvin_python_toolbox/management/engine.py 27.52% <23.07%> (-0.32%) :arrow_down:
...es/python-engine/project_package/_compatibility.py
...emplates/python-engine/project_package/_logging.py
...on-engine/project_package/data_handler/__init__.py
marvin_python_toolbox/common/config.py 96.96% <0%> (+71.96%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f1c170f...658b23c. Read the comment docs.

codecov-io avatar Nov 30 '17 13:11 codecov-io