Batch_size issue

[SrivatsaN225]

The issue is :I want to see how much time my predict() function is taking.Although I set batch_size to the number of rows of data I am sending but it is still batching it randomly and then calling the predict function. (We can know this by print(len(inputs)) )

Is there any way to disable the automatic batching that clipper is doing? So that the data is sent in one shot. I also tried increasing the slo_micros value but no luck there.

Here is the create_endpoint() code: keras_deployer.create_endpoint(clipper_conn=clipper_conn, name="keras", version="1", input_type="doubles", func=predict, model_path_or_object="/home/ubuntu/keras_experiments/model.h5", batch_size=1000,slo_micros=300000000, pkgs_to_install=['pandas'])

Thanks in advance.

[robberlang]

With PR #753 you can get all the inputs of your batch sent to your model function at once.

[SrivatsaN225]

Issue not Resolved: Hi I tried on the client end {'input_batch':s} where s is a list. I sent 100K rows and batch_size=100K it is still automatically batching the input rows.How to stop this? If is use only{'input':s} then it's throwing an error.Please explain your answer with some example code. I want to send the whole input in one go to the model how to do this?

[robberlang]

It's not possible that your single query of 100K inputs (using input_batch) get sent to your model as a single request. I came across the same problem. The PR I mentioned is a patch to fix this.

[SrivatsaN225]

I have just now installed from your branch and checked it, is still not working. So I sent 130000 rows and batch_size=50000 and when I checked the logs the input_rows length was 2357,4866 etc.That it is still automatically batching the input dataset. Even for 10K input rows and batch_size=1K still automatically batching.Please have a look.

[robberlang]

DId you build the query frontend docker image with my branch, and specify that image when starting Clipper?