Portkey-AI
Misleading HTTP 200 Status Code on Network Connection Lost Error via Portkey
What Happened?
Issue Summary:
When making a request to OpenAI's API through Portkey, I received an HTTP 200 status code despite the response body indicating a "Network connection lost." error. This behavior is misleading and prevents automatic retry mechanisms from triggering, as they typically rely on non-200 status codes to indicate failure.
{ "status": 200, "headers": { "content-type": "application/json" }, "body": { "html-message": "Network connection lost." }, "responseTime": 108, "lastUsedOptionJsonPath": "config" }
Steps to Reproduce: Send a POST request to https://api.openai.com/v1/chat/completions via Portkey with the following headers and body:
Headers: (List only the relevant headers, especially those related to Portkey and retry logic) Content-Type: application/json X-Portkey-Retry: {"attempts": 5} (Include any other headers that are crucial to replicating the issue) Body: (Simplify to illustrate the request without exposing sensitive information)
see the request json below
{ "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "headers": { "accept": "application/json", "accept-encoding": "gzip, br", "cf-visitor": "{\"scheme\":\"https\"}", "connection": "Keep-Alive", "content-length": "7325", "content-type": "application/json", "host": "api.portkey.ai", "user-agent": "OpenAI/Python 1.14.2", "x-forwarded-proto": "https", "x-portkey-retry": "{\"attempts\": 5}", "x-real-ip": "213.27.249.134", "x-stainless-arch": "x64", "x-stainless-async": "false", "x-stainless-lang": "python", "x-stainless-os": "MacOS", "x-stainless-package-version": "1.14.2", "x-stainless-runtime": "CPython", "x-stainless-runtime-version": "3.10.0" }, "body": { "messages": [ { "role": "system", "content": "<my_system_prompt>" }, { "role": "user", "content": "<my content>" } ], "model": "gpt-4-1106-preview", "n": 1, "stream": false, "temperature": 0 }, "portkeyHeaders": { "x-portkey-api-key": "rD********Hg=", "x-portkey-config": "{\"cache\":{\"mode\":\"simple\",\"maxAge\":864000}}", "x-portkey-provider": "openai", "x-portkey-trace-id": "f2fc3222-e3e5-4081-9d3f-f92cf2d312b0" } }
What Should Have Happened?
In the event of a network connection issue or similar error, I expect to receive an HTTP status code indicative of an error (e.g., 4xx or 5xx). This would allow client-side logic, such as automatic retries, to function correctly based on the status code.
Actual Behavior: Received an HTTP 200 status code, suggesting a successful response, despite the body containing an error message indicating a lost network connection.
Suggested Fix: Adjust the API or Portkey's error handling logic to return a more appropriate HTTP status code when an error occurs, especially for network-related issues.
Relevant Code Snippet
I trigger this request from a python script using langchain, as below
from langchain_community.chat_models import ChatOpenAI from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
headers = createHeaders( provider="openai", api_key=PORTKEY_API_KEY, # Cache Options config={ "cache": {"mode": "simple", "maxAge": 864000} }, # in seconds (10 days) retry={"attempts": 5}, # Metadata environment="test", user="<user>", organisation="<org>", prompt="test", )
after initializing an LLM object
llm = ChatOpenAI( model_name=model_name, temperature=0, base_url=PORTKEY_GATEWAY_URL, default_headers=headers, )
Your Twitter/LinkedIn
https://www.linkedin.com/in/nathanpamart/
I faced this bug too. An additional side effect of this bug is Portkey's cache caches the error response (assume because of the 200 response code). You don't get a valid response till you force cache refresh.
Hi @Nathanpamart @rootAvish sorry for missing out on this! Tagging @narengogi who can take a look at this
@rootAvish @Nathanpamart This issue has since been fixed, the gateway now handles http failures correctly
