error message when deploy to openshift

[gzhhong]

Hello,

I can run the docker image by "docker run -it -p1000:8080 notaitech/nudenet:classifier", and everything works fine.

The correct log as below:

2020-12-02 03:46:53.690994: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-12-02 03:46:53.699721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2799925000 Hz 2020-12-02 03:46:53.700472: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x48196c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-02 03:46:53.700529: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-12-02:03:46:53,701 WARNING [deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. Waiting for prediction loop to begin. 2020-12-02:03:46:57,194 INFO [saver.py:1284] Restoring parameters from /root/.NudeNet/classifier_model_tf/variables/variables Waiting for prediction loop to begin. 2020-12-02:03:47:00,505 INFO [_utils.py:186] Warming up .. 2020-12-02:03:47:03,238 INFO [_utils.py:226] Time per sample for batch_size: 1 is 0.2546215057373047 2020-12-02:03:47:04,776 INFO [_utils.py:226] Time per sample for batch_size: 2 is 0.2562824487686157 2020-12-02:03:47:04,776 INFO [_utils.py:241] optimum batch size is 1 2020-12-02:03:47:04,820 INFO [_loop.py:35] Starting prediction loop Waiting for prediction loop to begin. 2020-12-02:03:47:05,107 INFO [_utils.py:129] AVAILABLE FREE MEMORY: 44066.109375; CACHE: 2203.3054687500003 MB 2020-12-02:03:47:05,126 INFO [_generate_run_sh.py:11] WORKERS=2; batch_size=1; cpu_count=4 2020-12-02:03:47:05,791 INFO [_utils.py:129] AVAILABLE FREE MEMORY: 44066.09765625; CACHE: 2203.3048828125 MB [2020-12-02 03:47:05 +0000] [48] [INFO] Starting gunicorn 20.0.4 [2020-12-02 03:47:05 +0000] [48] [INFO] Listening at: http://0.0.0.0:8080 (48)

I tag this docker by docker tag command, then push the docker image to docker registry on openshift. Then I deploy the image and run it on openshift:

I get the error message when I run the pod:

  | File "_loop.py", line 8, in   | import _utils   | File "/app/_utils.py", line 42, in   | open(file_path, "wb").write(base64.b64decode(b64_string.encode("utf-8")))   | FileNotFoundError: [Errno 2] No such file or directory: './example_test/00.jpg'   | Traceback (most recent call last):   | File "_manager_loop.py", line 8, in   | import _utils   | File "/app/_utils.py", line 42, in   | open(file_path, "wb").write(base64.b64decode(b64_string.encode("utf-8")))   | FileNotFoundError: [Errno 2] No such file or directory: './example_test/00.jpg'   | Waiting for prediction loop to begin.   | Waiting for prediction loop to begin.   | Waiting for prediction loop to begin.   | Waiting for prediction loop to begin.

The "Waiting for prediction loop to begin." will printed line by line but never end.

What's the reason of the problem and how can I fix it?

Thanks,

James

[bedapudi6788]

Hi James, If I understand correctly, you are doing docker tag notaitech/nudenet:classifier NEW_REPO/NEW_TAG and doing docker push? If this is the case, it is very weird cause it should be the exact same image.

[gzhhong]

Hello Bedapudi,

I just found the script in the docker will download some other script then replace some script in the container, what's the purpose of such action? Can the classifier work offline, I mean the container can start and to predict an image without data connection?

Thanks,

James

[bedapudi6788]

@gzhhong the docker image will work offline.

I just found the script in the docker will download some other script then replace some script in the container,

Are you referring to https://github.com/notAI-tech/fastDeploy/blob/0fa6506936b2ccc288c387425b2ceed0ca18504f/service/_run.sh#L1?

This only happens if the environment variable UPDATE is set to true and is for updating the serving code without having to rebuild the docker image. In the above logs you posted, this doesn't happen.

In the above example, the environment variable UPDATE was not explicitly set and the container doesn't download anything.

If the container was trying to download new scripts (i.e: UPDATE is set to true), you will see explicit logs like in the above image.

Does this help?

[bedapudi6788]

Also, you can verify that the container needs no internet connection by passing --network none. This will disable the internet access for the container. If you pass --network none and -e UPDATE=true you will notice the error logs for failed download and if -e UPDATE=true is not passed, it will not print any error logs and work as expected.

[gzhhong]

ok, many thanks for the support. Another question, can we start the docker container without the role of root user? because it is forbidden on openshift

[bedapudi6788]

Yes, if you can run any docker image without sudo, you will be able to run nudenet's image also without sudo. Simply, if docker run hello-world works, running nudenet's image also will work.

[thinkall]

Yes, if you can run any docker image without sudo, you will be able to run nudenet's image also without sudo. Simply, if docker run hello-world works, running nudenet's image also will work.

If @gzhhong means running the container as a non-root user, then docker run --user 10000:10000 hello-world would be the command.

[thinkall]

@bedapudi6788 I tried with non-root user, PermissionError raised as below:

docker run --user 10000:10000 notaitech/nudenet:classifier 
Traceback (most recent call last):
  File "_loop.py", line 8, in <module>
    import _utils
  File "/app/_utils.py", line 42, in <module>
    open(file_path, "wb").write(base64.b64decode(b64_string.encode("utf-8")))
PermissionError: [Errno 13] Permission denied: './example_test/00.jpg'
Traceback (most recent call last):
  File "_manager_loop.py", line 8, in <module>
    import _utils
  File "/app/_utils.py", line 42, in <module>
    open(file_path, "wb").write(base64.b64decode(b64_string.encode("utf-8")))
PermissionError: [Errno 13] Permission denied: './example_test/00.jpg'

Then I changed the permission of all the files under /app and /root with chmod -R 777 /root && chmod -R 777 /app, it still raises PermissionError as below:

Traceback (most recent call last):
  File "_loop.py", line 137, in <module>
    from predictor import predictor
  File "/app/predictor.py", line 4, in <module>
    classifier = NudeClassifier()
  File "/usr/local/lib/python3.6/dist-packages/nudenet/classifier.py", line 29, in __init__
    os.mkdir(model_folder)
PermissionError: [Errno 13] Permission denied: '/.NudeNet/'

I think it's a result of the base image notaitech/fastdeploy:tf_1_15_cpu, files in base image are created with root user. Can you build an image which support arbitrary user? Thanks.

[bedapudi6788]

@thinkall I think i misunderstood the issue. Are you trying to use docker without using sudo to start it or do you want to change the user inside the docker container?

[thinkall]

@thinkall I think i misunderstood the issue. Are you trying to use docker without using sudo to start it or do you want to change the user inside the docker container?

@bedapudi6788 I'd like to change the user inside the docker container from root user to a non-root user.

[gzhhong]

@bedapudi6788 , can you provide new docker image? thank you so much!