containers icon indicating copy to clipboard operation
containers copied to clipboard
verified publisher icon databricks

Update gpu image

Open lu-wang-dl opened this issue 1 year ago • 0 comments

When starting the GPU docker image, we will get the following error:

Error: Internal error message: Spark error: Driver down cause: java.io.IOException: Cannot run program "virtualenv" (in directory "/local_disk0/.ephemeral_nfs/cluster_libraries"): error=2, No such file or directory

This PR is fixing the image issue. In this PR,

  • Unified the virtualenv setup in ubuntu/gpu/cuda-11.8/venv/Dockerfile to be the same as ubuntu/python/Dockerfile
  • Add new user for cluster library installation ubuntu/gpu/cuda-11.8/base/Dockerfile. This is following the same way in ubuntu/minimal/Dockerfile
  • Add python-lsp

Manually upload the docker to docker hub and create the cluster: Screenshot 2024-07-01 at 9 03 37 PM

lu-wang-dl avatar Jul 02 '24 05:07 lu-wang-dl