ragas
ragas copied to clipboard
explodinggradients
Timeout Error When Evaluating answer_similarity and answer_correctness in Ragas
Describe the bug
When attempting to evaluate answer_similarity and answer_correctness using the Ragas framework, I encounter a timeout error. While I can successfully retrieve metrics for context_relevancy and context_recall, the progress bar stalls for a long time and eventually times out when evaluating answer_similarity and answer_correctness.
Ragas version: ragas-0.0.23.dev37+g041b20c Python version: 3.11 DATA: 63521_1.json
Code to Reproduce
import os
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_similarity
from config import OPENAI_API_KEY, OPENAI_BASE_URL
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
os.environ["OPENAI_API_BASE"] = OPENAI_BASE_URL
def evaluate_ragas(file_path):
with open(file_path, 'r', encoding='utf-8') as file:
data = json.load(file)
dataset = Dataset.from_dict(data)
result = evaluate(
dataset,
metrics=[answer_similarity],
)
print(result)
if __name__ == '__main__':
file_path = "63521_1.json"
evaluate_ragas(file_path)
Error trace
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import AzureChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatVertexAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import AzureOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import OpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import VertexAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import AzureOpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import FastEmbedEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import OpenAIEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:115: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use langchain_openai.ChatOpenAI instead.
warn_deprecated(
Evaluating: 0%| | 0/5 [07:32<?, ?it/s]
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
sock = socket.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
raise exceptions[0]
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
stream = self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
with map_exceptions(exc_map):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
response = self._client.send(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
with map_httpcore_exceptions():
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
sock = socket.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
raise exceptions[0]
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
stream = self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
with map_exceptions(exc_map):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
response = self._client.send(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
with map_httpcore_exceptions():
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
sock = socket.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
raise exceptions[0]
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
yield
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
response = connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
raise exc
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
stream = self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
with map_exceptions(exc_map):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
response = self._client.send(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
with map_httpcore_exceptions():
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 176, in evaluate
raise e
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 159, in evaluate
results = executor.results()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 118, in results
raise e
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 114, in results
r = future.result()
^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 36, in wrapped_callable
return counter, callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 75, in score
raise e
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 71, in score
score = self._score(row=row, callbacks=group_cm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/_answer_similarity.py", line 70, in _score
embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 668, in embed_documents
return self._get_len_safe_embeddings(texts, engine=engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 494, in _get_len_safe_embeddings
response = embed_with_retry(
^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 116, in embed_with_retry
return embeddings.client.create(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/resources/embeddings.py", line 106, in create
return self._post(
^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 1088, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 853, in request
return self._request(
^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 893, in _request
raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 40, in <module>
evaluate_ragas("63521", result_id="1")
File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 24, in evaluate_ragas
result = evaluate(
^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 178, in evaluate
result = Result(
^^^^^^^
File "<string>", line 6, in __init__
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 207, in __post_init__
for cn in self.scores[0].keys():
~~~~~~~~~~~^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2795, in __getitem__
return self._getitem(key)
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2779, in _getitem
pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 583, in query_table
_check_valid_index_key(key, size)
File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 0 is out of bounds for size 0
Expected behavior
I expected the answer_similarity and answer_correctness evaluations to succeed without a timeout error, similar to the other metrics.
Additional context
In my use of Ragas for evaluation, other metrics like context_relevancy and context_recall work normally. The issue arises solely when evaluating answer_similarity and answer_correctness.
I'm also having the same issue for answer_correctness and answer_similarity.
I'm getting the following error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 93, in run
results = self.loop.run_until_complete(self._aresults())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 81, in _aresults
raise e
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 76, in _aresults
r = await future
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro
return await coro
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 109, in wrapped_callable_async
return counter, await callable(*args, **kwargs)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore
raise e
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore
score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/_answer_similarity.py", line 65, in _ascore
embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 23, in embed_text
embs = await self.embed_texts([text], is_async=is_async)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 33, in embed_texts
return await aembed_documents_with_retry(texts)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
return await fn(*args, **kwargs)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
do = self.iter(retry_state=retry_state)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 325, in iter
raise retry_exc.reraise()
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 158, in reraise
raise self.last_attempt.result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
result = await fn(*args, **kwargs)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 66, in aembed_documents
return await self.embeddings.aembed_documents(texts)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 526, in aembed_documents
return await self._aget_len_safe_embeddings(texts, engine=engine)
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 435, in _aget_len_safe_embeddings
token = encoding.encode(
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer
Traceback (most recent call last):
File "/Users/dcast/Documents/GitHub/all-about-rag/src/scripts/evaluate_auto_merging_ragas.py", line 67, in <module>
result = evaluate(
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 228, in evaluate
raise e
File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 210, in evaluate
raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
2024-03-10 19:28:51,571 ERROR asyncio: Task was destroyed but it is pending!
task: <Task pending name='Task-48' coro=<as_completed.<locals>.sema_coro() running at /Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:558]>
2024-03-10 19:28:51,571 ERROR asyncio: Task was destroyed but it is pending!
task: <Task pending name='Task-43' coro=<as_completed.<locals>.sema_coro() running at /Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:558]>
Just found the issue. This was due to having null values in the ground_truth column. Interestingly, those NAs are from synthetically generated questions using generate_with_llamaindex_docs.
@shahules786 might be good to raise a warning in these cases. What do you think?
Happy to help with a PR, if that makes sense.
