online inference or predict

Open jacklone opened this issue 8 years ago • 2 comments

Thank you for sharing, I would like to provide a demo translation service using serving or something, and I found this similar issue: https://github.com/google/seq2seq/issues/114. when I use tf.contrib.learn.estimator, each call to predict() reloads the model, same as https://github.com/tensorflow/tensorflow/issues/4648. Could you give me any suggestions?

Nov 23 '17 07:11 jacklone

@jacklone tf.contrib.learn.estimator has the problem that would reload the model every time when calling the functions. Have you ever tried this solutions? https://github.com/marcsto/rl/blob/master/src/fast_predict.py

Nov 23 '17 09:11 tobyyouup

@tobyyouup I think this solution has problems when dealing with multi-thread request in my web demo. Maybe I am wrong?

Nov 23 '17 10:11 jacklone