bentoml
bug: bentoml.ray.deployment is broken with Ray 2.8.0
Describe the bug
The implementation of src/bentoml/_internal/ray/init.py had dependencies upon internal Ray object which may change across releases leading to incompatibilities. In particular it references ASGIHTTPSender (which was only present before ray 2.5), https://github.com/bentoml/BentoML/pull/4191 enabled support for Ray 2.5. This has now broken again for Ray 2.8 because there is neither a BufferedASGISender class which is referenced in src/bentoml/_internal/ray/init.py and grpc-client/bentoml/_internal/ray/init.py in the following code fragment.
if pkg.pkg_version_info("ray")[:2] >= (2, 5):
from ray.serve._private.http_util import BufferedASGISender as ASGIHTTPSender
else:
from ray.serve._private.http_util import ASGIHTTPSender
_get_service_deployment (http & grpc) should be implemented in a manner that doesn't use Ray private interfaces to make this resilient to Ray version changes. Hopefully, that isn't impossible.
To reproduce
Follow instructions at https://docs.bentoml.org/en/latest/integrations/ray.html with Ray 2.8.0 installed
Example stack with Ray 2.8.0
module = importlib.import_module(module_name)
File "/usr/local/Cellar/[email protected]/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/rcorujo/bento_testing/./bento_ray.py", line 8, in <module>
classifier = bentoml.ray.deployment('mnist_classifier:oxk7lydkxkq2jlg6')
File "/Users/rcorujo/.virtualenvs/aioli/lib/python3.9/site-packages/bentoml/_internal/utils/lazy_loader.py", line 69, in __getattr__
self._module = self._load()
File "/Users/rcorujo/.virtualenvs/aioli/lib/python3.9/site-packages/bentoml/_internal/utils/lazy_loader.py", line 48, in _load
module = importlib.import_module(self.__name__)
File "/usr/local/Cellar/[email protected]/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/rcorujo/.virtualenvs/aioli/lib/python3.9/site-packages/bentoml/ray.py", line 1, in <module>
from ._internal.ray import deployment
File "/Users/rcorujo/.virtualenvs/aioli/lib/python3.9/site-packages/bentoml/_internal/ray/__init__.py", line 11, in <module>
from ray.serve._private.http_util import BufferedASGISender as ASGIHTTPSender
ImportError: cannot import name 'BufferedASGISender' from 'ray.serve._private.http_util'
Expected behavior
Successful startup
Environment
bentoml: 1.1.7 python: 3.9 ray: 2.8.0
@jerryharrow Did you find a workaround for this?
@NrKhader unfortunately not. You can drop back to Ray 2.7 which should still work. I fixed an earlier incompatibility with https://github.com/bentoml/BentoML/pull/4257, but then we hit this one when Ray 2.8.0 came out. We gave up trying to work with Ray + BentoML because BentoML is dependent upon Ray internals to work, so this type of incompatibly is likely to continue to crop up as thing evolve.
Same. +1. Seek Ray Serve to continue supporting BentoML because it provides for scalable serving platform, beyond individual containers.
pip install "starlette<=0.34.0" seemed to work on 2.7
Will it be resolved?
