during classification ,I am doing performance testing on AWS with inception model flask api with gunicorn (creating multiple process) Error: OOM when allocating tensor with shape[800,1280,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Cast = CastDstT=DT_FLOAT, SrcT=DT_UINT8, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[daoud]

I am doing performance testing on AWS with inception model flask api with gunicorn (creating multiple process) Error: OOM when allocating tensor with shape[800,1280,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Cast = CastDstT=DT_FLOAT, SrcT=DT_UINT8, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'Cast', defined at: File "/usr/local/bin/gunicorn", line 11, in sys.exit(run()) File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 61, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 223, in run super(Application, self).run() File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 72, in run Arbiter(self).run() File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 212, in run self.manage_workers() File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 545, in manage_workers self.spawn_workers() File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 616, in spawn_workers self.spawn_worker() File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 134, in init_process self.run() File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 124, in run self.run_for_one(timeout) File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 68, in run_for_one self.accept(listener) File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 30, in accept self.handle(listener, client, addr) File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 135, in handle self.handle_request(listener, req, client, addr) File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 176, in handle_request respiter = self.wsgi(environ, resp.start_response) File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1997, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1598, in dispatch_request return self.view_functionsrule.endpoint File "/home/ubuntu/cv_workspace/computer_vision_services.py", line 1480, in classify_bulk input_operation, output_operation, tf_session=sess) File "/home/ubuntu/cv_workspace/src/apis/ImgClassification.py", line 228, in classifyImageInSess t = read_tensor_from_image_file(file_name) File "/home/ubuntu/cv_workspace/src/apis/ImgClassification.py", line 51, in read_tensor_from_image_file float_caster = tf.cast(image_reader, tf.float32) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py", line 758, in cast return gen_math_ops.cast(x, base_type, name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 919, in cast "Cast", x=x, DstT=DstT, name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3160, in create_op op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1625, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

################ GPU