submarine
submarine copied to clipboard
SUBMARINE-1283. copy data for experiment before it running via distcp to minio
What is this PR for?
DONT MERGE THIS, this PR is for debugging for now steps to test stuck distcp job:
- install submarine in regular way, the submarine cr would be installed in submarine-user-test(or anywhere you assigned)
- create a minio in the same namespace with submarine cr, https://gist.github.com/FatalLin/12f98f90bae5b16e19ed0e51c3ce863e
- cd
./dev-support/docker-images/experiment-prehandler, execute./build.shto build image(please make sure you've change to minikube environment. - create a pre-handler pod in the same with submarine cr, https://gist.github.com/FatalLin/0a0fe69abccf0aebac5074c4e4b126b3
- use
kubectl execto the pre-handler pod, the hadoop package has been deployed in/opt/hadoop-3.3.3, enter the hadoop directory and execute./bin/hadoop distcp -Dfs.s3a.access.key=submarine_minio -Dfs.s3a.secret.key=submarine_minio -Dfs.s3a.endpoint=http://<your hdfs host>:<your hdfs port> hdfs://host.docker.internal:9000/tmp/* s3a://submarine-minio-dev:9000/let's see what'll happen here.
What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
Todos
- [ ] - Task
What is the Jira issue?
How should this be tested?
Screenshots (if appropriate)
Questions:
- Do the license files need updating? Yes/No
- Are there breaking changes for older versions? Yes/No
- Does this need new documentation? Yes/No