apache
[SUPPORT] setup_demo.sh can support MacOS M1
Tips before filing an issue
-
Have you gone through our FAQs?
-
Join the mailing list to engage in conversations and get faster support at [email protected].
-
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
[+] Running 34/34
✔ Network hudi Created 0.0s
✔ Container kafkabroker Started 2.0s
✔ Container namenode Started 2.5s
✔ Container graphite Started 1.8s
✔ Container zookeeper Started 2.3s
✔ Container hive-metastore-postgresql Started 2.2s
! zookeeper The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! kafka The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! hive-metastore-postgresql The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! namenode The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
✔ Container hivemetastore Started 3.6s
✔ Container historyserver Started 3.6s
! historyserver The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! hivemetastore The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
✔ Container datanode1 Started 4.9s
✔ Container trino-coordinator-1 Started 5.1s
✔ Container hiveserver Started 4.9s
✔ Container presto-coordinator-1 Started 4.8s
! presto-coordinator-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! trino-coordinator-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! hiveserver The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! datanode1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
✔ Container sparkmaster Started 6.0s
✔ Container trino-worker-1 Started 6.1s
✔ Container presto-worker-1 Started 5.8s
! presto-worker-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! trino-worker-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! sparkmaster The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
✔ Container spark-worker-1 Started 7.3s
✔ Container adhoc-2 Started 7.6s
✔ Container adhoc-1 Started 7.6s
! spark-worker-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! adhoc-1 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
! adhoc-2 The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
-
Hudi version :
-
Spark version :
-
Hive version :
-
Hadoop version :
-
Storage (HDFS/S3/GCS..) :
-
Running on Docker? (yes/no) :
Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
Hi @xiedeyantu , could you try
setup_demo.sh --mac-aarch64on M1 Macbook?
Sorry, there is also some erros, following:
image with reference docker.io/wurstmeister/kafka:2.12-2.0.1 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64
image with reference wurstmeister/kafka:2.12-2.0.1 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64
Error response from daemon: No such container: adhoc-1
Error response from daemon: No such container: adhoc-2
@xiedeyantu Did you tried using --mac-aarch64, Are you still facing this error with that flag?
@xiedeyantu you'll need to first stop the demo with stop_demo.sh and stop_demo.sh --mac-aarch64 to shutdown all existing instances and restart the demo from fresh with setup_demo.sh --mac-aarch64. Let us know how it goes.
@xiedeyantu Did you tried using
--mac-aarch64, Are you still facing this error with that flag?
yes
@xiedeyantu you'll need to first stop the demo with
stop_demo.shandstop_demo.sh --mac-aarch64to shutdown all existing instances and restart the demo from fresh withsetup_demo.sh --mac-aarch64. Let us know how it goes.
This doesn't work for me, I even killed every docker.I have also tried the method you mentioned, but the error reported above.
@xiedeyantu Can you tell us which version you are using? What command you ran for maven build? Also if possible mentioned detailed steps which will help us to either identify the root cause OR we can try to reproduce this issue.
@xiedeyantu Can you tell us which version you are using? What command you ran for maven build? Also if possible mentioned detailed steps which will help us to either identify the root cause OR we can try to reproduce this issue.
@ad1happy2go I uesd master, I only ran setup_demo.sh --mac-aarch64 without maven build. But image has been downloaded is x86 platform on mac m1.I also use build_local_docker_images.sh, but I find no hudi image.
@xiedeyantu Can you try building it with 0.13.0 version once. That version works for sure. If that succeed, we can try triaging the issue with master.
Please share the MVN command you ran to build the project, Before running setup_demo we must build the project.
@xiedeyantu Follow the doc, it has mvn build command also. https://hudi.apache.org/docs/docker_demo/
you can use this to build faster - mvn -T 2C clean package -DskipTests
@xiedeyantu Follow the doc, it has mvn build command also. https://hudi.apache.org/docs/docker_demo/
you can use this to build faster - mvn -T 2C clean package -DskipTests It can help me to build hudi, but can not build docker images such as hivemetastore, sparkmaster, ... . I also try to exec
build_local_docker_images.sh, no images can get.
You don't need to build local images to run docker. setup_demo.sh will pull the docker images it needs. @xiedeyantu Just build the hudi and run setup_demo
You don't need to build local images to run docker. setup_demo.sh will pull the docker images it needs. @xiedeyantu Just build the hudi and run setup_demo
@ad1happy2go Some images can not support arm64 on MacOS, I need to build. Anyway, I also want to know how to build local images.
@xiedeyantu I never tried that to build local imaged. I saw you ran - ./build_local_docker_images.sh
What error you facing while running this?
@xiedeyantu I confirmed with the latest release - 0.14.0 and docker demo is working fine. Can you please try.
wurstmeister/kafka:2.12-2.0.1 seems to be unavailable for arm64. Is there an suggested alternative?
