MLServer Multi-language Runtimes

Create multi-language wrappers that can run Java, C++ and R models. For this, we can leverage the existing research in Seldon Core which leverages tools like JNI and PyBind to bridge from Python to other runtimes.

May 06 '21 13:05 axsaucedo

Adding to the above issue as indicated in https://github.com/SeldonIO/seldon-core/issues/4531

MLeap is used to deploy Spark mllib models (including Spark Xgboost and others) for inference through Java runtime. With Seldon core the Java language wrapper could be used to wrap MLeap format models and deploy using Java JNI wrapper. The JNI wrapper however had many vulnerabilities https://github.com/SeldonIO/seldon-core/issues/4103.

Please support deployment of MLeap serialized Spark MLlib models in MLServer so that Java JNI wrapper vulnerabilities can be avoided.

Jan 05 '23 22:01 indranilr

Hey @indranilr ,

Thanks for expressing interest on this one!

Just as a head up, this is on the roadmap, but probably won't come through until the 1.4.0 release.

Jan 10 '23 16:01 adriangonz

Just out of interest would this include Julia support?

Jun 21 '23 10:06 dtpryce

We've already done something along these lines for C++ and Java for SCv1 so that would be the initial priority. We are not planning to explore adding support for Julia yet.

Jun 21 '23 10:06 adriangonz

Totally understood! Will be good to see those first few languages and we could always try to mimic for Julia if we decide we really need it :) thank @adriangonz!

Jun 21 '23 10:06 dtpryce

Hey @indranilr ,

Thanks for expressing interest on this one!

Just as a head up, this is on the roadmap, but probably won't come through until the 1.4.0 release.

@adriangonz since 1.4.0 is released now, has the support for MLeap serialized Spark Xgboost model incorporated ? Could you please point to relevant documentation ?

Mar 05 '24 10:03 indranilr

Hey @indranilr,

I don't work on Seldon or this project anymore. Best would be to check with @ramonpzg on what's the status of this one.

Mar 05 '24 11:03 adriangonz

Hey @indranilr,

I don't work on Seldon or this project anymore. Best would be to check with @ramonpzg on what's the status of this one.

Thanks Adrian, @ramonpzg appreciate if you can provide guidance w.r.t this question

Mar 05 '24 13:03 indranilr

Hi @indranilr -- We recently moved things around in the roadmap of MLServer and have started dedicating more capacity to improving different features of it and its runtime. It might take us a couple more versions before we can fully dedicate time for multi-language wrappers (it would definitely be very valuable to add these :raised_hands: ).

Mar 22 '24 13:03 ramonpzg