fairing icon indicating copy to clipboard operation
fairing copied to clipboard
verified publisher icon kubeflow

Is threre any instruction on examples/simple for docker registry case?

Open sakaia opened this issue 5 years ago • 2 comments

/kind bug

What steps did you take and what happened: I try to execute fairing on local repository but it failed. Is there any comments?

Edit examples/simple/main.py

$ diff -urpN main.py main0.py
--- main.py     2020-06-02 02:30:46.801637906 +0000
+++ main0.py    2020-06-11 02:43:49.412253759 +0000
@@ -4,8 +4,7 @@ import tensorflow as tf
 from kubeflow import fairing
 # Setting up google container repositories (GCR) for storing output containers
 # You can use any docker container registry istead of GCR
-GCP_PROJECT = fairing.cloud.gcp.guess_project_name()
-DOCKER_REGISTRY = 'gcr.io/{}/fairing-job'.format(GCP_PROJECT)
+DOCKER_REGISTRY = 'localhost:5000/fairing-job'
 fairing.config.set_builder(
     'append', base_image='tensorflow/tensorflow:1.13.1-py3', registry=DOCKER_REGISTRY, push=True)
 fairing.config.set_deployer('job')

execute python script

python3 main.py

Waiting for continues

[W 200611 05:23:09 append:54] Image successfully built in 2.4282387730017945s.
[W 200611 05:23:09 append:94] Pushing image localhost:5000/fairing-job/fairing-job:9289FA5B...
[I 200611 05:23:09 docker_creds_:234] Loading Docker credentials for repository 'localhost:5000/fairing-job/fairing-job:9289FA5B'
[W 200611 05:23:09 append:81] Uploading localhost:5000/fairing-job/fairing-job:9289FA5B
[I 200611 05:23:09 docker_session_:280] Layer sha256:22d8b84cd8f1ede5daa97ffb3539c4737fb4f050075bd71ba411f4c127123f89 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:7e65918542624be36faca5e547a5b84c4eca21dd4e979c9309f4ad79b95d00ce exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:089d60cb4e0a6a5f884454ad8f9ea43021ac8c4a8584698f8751fab2437450e3 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:fe293195091d6aba06481f8c65f61eb206d1aaadfc6028a432194dd49e237d1c exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:29303e8416d52b21fcf37eef03df2d402dd4c772860dd67890862ebe3389e1c8 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:45085432511a479b977e19d3334f4948200141d06802c222e743e3d3bafabd05 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:0a5fb8dc4fa0b9c02e5e64782ebf3288cc2a8b795e9d92ff722b06a425b756de exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:9c461696bc0925124c5fe76298c2590a4d4b4009eab16a2346c0010269007f74 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:12bb05a3cac8e4cfbc016bffbef28c71cb8602fb35caef657e8dc15907aa8dc7 exists, skipping
[I 200611 05:23:09 docker_session_:280] Layer sha256:b816d6e919ba26085fe7d7b3e56390efee286c2701f1ad57ea1af54a1fa4892e exists, skipping
[I 200611 05:23:09 docker_session_:284] Layer sha256:33824972951988895d9203b193b528539b5e12957190b46e492bfbbe65fcbb8e pushed.
[I 200611 05:23:09 docker_session_:284] Layer sha256:4780bad7ba12dc1fe2019001f179655c913138103eecf298e0251138fc9904ff pushed.
[I 200611 05:23:09 docker_session_:334] Finished upload of: localhost:5000/fairing-job/fairing-job:9289FA5B
[W 200611 05:23:09 append:99] Pushed image localhost:5000/fairing-job/fairing-job:9289FA5B in 0.7465651339989563s.
[W 200611 05:23:10 job:101] The job fairing-job-k5z7b launched.
[W 200611 05:23:12 manager:296] Waiting for fairing-job-k5z7b-c4tmc to start...
[W 200611 05:23:12 manager:296] Waiting for fairing-job-k5z7b-c4tmc to start...
[W 200611 05:23:13 manager:296] Waiting for fairing-job-k5z7b-c4tmc to start...

What did you expect to happen: example/simple/main.py works fine on local repository.

Anything else you would like to add: I am looking into "docker logs registry" it returns 404 and 500 error. 500 says invalid character '\x1f' looking for beginning of value

log.txt

time="2020-06-08T05:48:43.804235165Z" level=info msg="response completed" go.version=go1.11.2 http.request.contenttype="application/json" http.request.host="localhost:5000" http.request.id=b705834e-1a61-4fef-b258-5ef8ee277a65 http.request.method=GET http.request.remoteaddr="172.17.0.1:39648" http.request.uri="/v2/" http.request.useragent="//fairing/client:main0.py" http.response.contenttype="application/json; charset=utf-8" http.response.duration=33.597662ms http.response.status=200 http.response.written=2
172.17.0.1 - - [08/Jun/2020:05:48:43 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "//fairing/client:main0.py"
time="2020-06-08T05:48:43.806907172Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=v2/fairing-job/fairing-job revision=sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=5f85a1c8-fd88-44c1-8739-6144fb7ad299 http.request.method=GET http.request.remoteaddr="172.17.0.1:39648" http.request.uri="/v2/v2/fairing-job/fairing-job/manifests/sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245" http.request.useragent="//fairing/client:main0.py" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.42223ms http.response.status=404 http.response.written=202 vars.name="v2/fairing-job/fairing-job" vars.reference="sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245"
172.17.0.1 - - [08/Jun/2020:05:48:43 +0000] "GET /v2/v2/fairing-job/fairing-job/manifests/sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245 HTTP/1.1" 404 202 "" "//fairing/client:main0.py"
time="2020-06-08T05:48:43.806907172Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=v2/fairing-job/fairing-job revision=sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=5f85a1c8-fd88-44c1-8739-6144fb7ad299 http.request.method=GET http.request.remoteaddr="172.17.0.1:39648" http.request.uri="/v2/v2/fairing-job/fairing-job/manifests/sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245" http.request.useragent="//fairing/client:main0.py" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.42223ms http.response.status=404 http.response.written=202 vars.name="v2/fairing-job/fairing-job" vars.reference="sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245"
172.17.0.1 - - [08/Jun/2020:05:48:43 +0000] "GET /v2/v2/fairing-job/fairing-job/manifests/sha256:2607929a5f8c1167748e1d5deb439853f0225e5166a6e859e635e7203727e245 HTTP/1.1" 404 202 "" "//fairing/client:main0.py"
172.17.0.1 - - [08/Jun/2020:05:48:43 +0000] "HEAD /v2/v2/fairing-job/fairing-job/blobs/sha256:fe293195091d6aba06481f8c65f61eb206d1aaadfc6028a432194dd49e237d1c HTTP/1.1" 404 157 "" "//fairing/client:main0.py"

(Update 06/11 ) for 500 response code it seems invalid character \x1f.

time="2020-06-11T06:54:44.042538405Z" level=error msg="response completed with error" err.code=unknown err.detail="invalid character '\x1f' looking for beginning of value" err.message="unknown error" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=48fd2001-b63a-486b-b3ae-9f5cfcd8590a http.request.method=GET http.request.remoteaddr="172.17.0.1:38468" http.request.uri="/v2/fairing-job/fairing-job/manifests/sha256:e52eb5f69eb32d21b14166bc2917608187a0f0258cc8531f16b6cfa98de97308" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=34.673483ms http.response.status=500 http.response.written=80 vars.name="fairing-job/fairing-job" vars.reference="sha256:e52eb5f69eb32d21b14166bc2917608187a0f0258cc8531f16b6cfa98de97308"

$ curl http://localhost:5000/v2/fairing-job/fairing-job/manifests/sha256:b767e032bb64a3ef94ef6958e13c23ab1f479be57c24ba3242f3836fb76328a8
{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Offset":1}}]}

for 404 error, blob or manifest is not found.

time="2020-06-11T06:34:22.510799652Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=fairing-job/fairing-job revision=sha256:e2e711fdd28404d6300174174de907a5dd602c3fa67cf4ac64b9c9115cb1690a" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=de43cfac-01fc-4060-9c1c-e6df07750de8 http.request.method=GET http.request.remoteaddr="172.17.0.1:49438" http.request.uri="/v2/fairing-job/fairing-job/manifests/sha256:e2e711fdd28404d6300174174de907a5dd602c3fa67cf4ac64b9c9115cb1690a" http.request.useragent="//fairing/client:main0.py" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.095207ms http.response.status=404 http.response.written=199 vars.name="fairing-job/fairing-job" vars.reference="sha256:e2e711fdd28404d6300174174de907a5dd602c3fa67cf4ac64b9c9115cb1690a"

time="2020-06-11T06:19:47.50939995Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:fecdd531c2d67877ed00fe7537276d5333ca871af813dce1ddaf09c3db360087 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=f1c9bc2d-8b1e-46e6-a39f-37242b70aef7 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:37134" http.request.uri="/v2/fairing-job/fairing-job/blobs/sha256:fecdd531c2d67877ed00fe7537276d5333ca871af813dce1ddaf09c3db360087" http.request.useragent="//fairing/client:main0.py" http.response.contenttype="application/json; charset=utf-8" http.response.duration=72.207535ms http.response.status=404 http.response.written=157 vars.digest="sha256:fecdd531c2d67877ed00fe7537276d5333ca871af813dce1ddaf09c3db360087" vars.name="fairing-job/fairing-job"

$ curl http://localhost:5000/v2/v2/fairing-job/fai
ring-job/blobs/sha256:b767e032bb64a3ef94ef6958e13c23ab1f479be57c24ba3242f3836fb76328a8
{"errors":[{"code":"BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:b767e032bb64a3ef94ef6958e13c23ab1f479be57c24ba3242f3836fb76328a8"}]}

Environment:

  • Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"):0.7.1.1
  • Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):1.0.2
  • Minikube version: none(microk8s)
  • Kubernetes version: (use kubectl version):1.15.11
  • OS (e.g. from /etc/os-release): ubuntu 19.10

NOTE: If you are using fair from master, please provide us the git commit hash.

sakaia avatar Jun 11 '20 05:06 sakaia

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

issue-label-bot[bot] avatar Jun 11 '20 05:06 issue-label-bot[bot]

You can try the e2e example: https://github.com/kubeflow/fairing/tree/master/examples/mnist

jinchihe avatar Jun 18 '20 07:06 jinchihe