DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Requests.exceptions.ConnectionError:

Open Weigaa opened this issue 1 year ago • 2 comments

For some well known reason we don't have direct access to huggingface.co, we usually use : export HF_ENDPOINT="https://hf-mirror.com" to get around this. But this method fails when we execute the following statement. python mii-sd.py

the error list is: Traceback (most recent call last): File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connectionpool.py", line 403, in _make_request self._validate_conn(conn) File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn conn.connect() File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() ^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f6c93e23e10>: Failed to establish a new connection: [Errno 101] Network i s unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/requests/adapters.py", line 487, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models?cursor=eyJfaWQiOnsiJGd0IjoiNjI xZmZkYzEzNjQ2OGQ3MDlmMTc4ZTA1In19 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6c93e23e10>: Failed to establish a new connect ion: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/lthpc/nvmessd/wj/DeepSpeed-MII/mii/legacy/examples/benchmark/txt2img/mii-sd.py", line 20, in mii.deploy(task='text-to-image', File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/mii/legacy/deployment.py", line 72, in deploy mii_config = mii.config.MIIConfig(**mii_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/deepspeed/runtime/config_utils.py", line 56, in init

File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/fields.py", line 881, in validate v, errors = self._validate_singleton(v, values, loc, cls) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/fields.py", line 1098, in _validate_singleton return self._apply_validators(v, values, loc, cls, self.validators) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/fields.py", line 1154, in apply_validators v = validator(cls, v, values, self, self.model_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/class_validators.py", line 337, in return lambda cls, v, values, field, config: validator(v) ^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/main.py", line 711, in validate return cls(**value) ^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/deepspeed/runtime/config_utils.py", line 56, in init super().init(**data) File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/main.py", line 339, in init values, fields_set, validation_error = validate_model(pydantic_self.class, data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/pydantic/v1/main.py", line 1100, in validate_model values = validator(cls, values) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/mii/legacy/config.py", line 257, in validate_model_and_task mii.utils.check_if_task_and_model_is_valid(task, model) File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/mii/legacy/utils.py", line 102, in check_if_task_and_model_is_valid valid_task_models = _get_hf_models_by_type(None, task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/mii/legacy/utils.py", line 44, in _get_hf_models_by_type model_data["model_list"] = [ ^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/mii/legacy/utils.py", line 44, in model_data["model_list"] = [ ^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1488, in list_models for item in items: File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/huggingface_hub/utils/_pagination.py", line 44, in paginate r = session.get(next_page, headers=headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/requests/sessions.py", line 600, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 67, in send return super().send(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lthpc/anaconda3/envs/wjtorch2.0.1/lib/python3.11/site-packages/requests/adapters.py", line 520, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models?cursor=eyJfaWQiOnsiJGd0IjoiNjIxZmZkYzEzNjQ2OGQ3MDlmMTc4ZTA1In19 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6c93e23e10>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 69ff26d3-7e2e-48b0-9fd1-2c6a3b36c521)')

Weigaa avatar Mar 02 '24 01:03 Weigaa

same error occurs on all examples on https://github.com/microsoft/DeepSpeed-MII/tree/main/mii/legacy/examples/local @mrwyattii Is there any way to avoid visiting huggingface.co or implement offline deployment? I actually already have the models I need in .cache.

Weigaa avatar Mar 02 '24 07:03 Weigaa

@Weigaa please set "skip_model_check": True in your mii_config: https://github.com/microsoft/DeepSpeed-MII/blob/773b735d6294a98dd842d82ef024d0d9b050f66a/mii/legacy/config.py#L98

This should avoid connecting to the HF server to validate the model/task

mrwyattii avatar Mar 21 '24 22:03 mrwyattii

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=1GNc0HUee5YQH4Vtlk8ZbDjyJBYTEyabo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc3f5ebb0a0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

How do you solve it?

yang-W517 avatar Jul 17 '24 17:07 yang-W517

This appears to be resolved so closing this issue for now. Please comment if not.

loadams avatar Aug 22 '24 23:08 loadams