submarine icon indicating copy to clipboard operation
submarine copied to clipboard

SUBMARINE-1283. copy data for experiment before it running via distcp to minio

Open FatalLin opened this issue 3 years ago • 0 comments

What is this PR for?

DONT MERGE THIS, this PR is for debugging for now steps to test stuck distcp job:

  1. install submarine in regular way, the submarine cr would be installed in submarine-user-test(or anywhere you assigned)
  2. create a minio in the same namespace with submarine cr, https://gist.github.com/FatalLin/12f98f90bae5b16e19ed0e51c3ce863e
  3. cd ./dev-support/docker-images/experiment-prehandler, execute ./build.sh to build image(please make sure you've change to minikube environment.
  4. create a pre-handler pod in the same with submarine cr, https://gist.github.com/FatalLin/0a0fe69abccf0aebac5074c4e4b126b3
  5. use kubectl exec to the pre-handler pod, the hadoop package has been deployed in /opt/hadoop-3.3.3, enter the hadoop directory and execute ./bin/hadoop distcp -Dfs.s3a.access.key=submarine_minio -Dfs.s3a.secret.key=submarine_minio -Dfs.s3a.endpoint=http://<your hdfs host>:<your hdfs port> hdfs://host.docker.internal:9000/tmp/* s3a://submarine-minio-dev:9000/ let's see what'll happen here.

What type of PR is it?

[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]

Todos

  • [ ] - Task

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

  • Do the license files need updating? Yes/No
  • Are there breaking changes for older versions? Yes/No
  • Does this need new documentation? Yes/No

FatalLin avatar Aug 15 '22 03:08 FatalLin