Portkey-AI
feat: Add Deep Infra Provider
Title: Add Deep Infra
- Add provider as Deep Infra
- Support /chat/completions route
Made the addition of Deep Infra provider using the docs provided for reference, followed the structure from other already existing API. Let me know if I need to make changes or if I have missed any variables.
Related Issues: (optional)
- #110
Hey @Suraj-Bhandarkar-S . Thanks for the PR. Will do a detailed review and also run tests for these once. Looks good at first glance.
I have found one issue. You have not added deepinfra in this file: src/providers/index.ts. Without that, the integration will not work. Can you please add that. Also, please make sure that all the flows are tested on your end.
The split pattern used by deepinfra between each stream chunk is \r\n\r\n. You can check this by printing JSON.stringify() of stream chunks that come in response from deepinfra. This pattern needs to be defined in src/utils.ts in this function
https://github.com/Portkey-AI/gateway/blob/b5dbfadd5bc697c5c212cf44a3bc8bea898f3216/src/utils.ts#L4
Without this, gateway is not able to recognize when a particular stream chunk ends.
@VisargD when I am trying to test in local for deepinfra provider,
`curl --location '127.0.0.1:8787/v1/openai/chat/completions' \ --header 'x-portkey-provider: deepinfra' \ --header 'Authorization: Bearer (added my key)' \ --header 'Content-Type: application/json' \ --data '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "deepinfra/airoboros-70b"}'`
I am getting
{ "status": "failure", "message": "Invalid provider passed" }
I have added the provider as you had mentioned I am not sure if I have missed anything. (Using PostMan to Test) Thanks again!
when I am trying to test in local for deepinfra provider, curl --location '127.0.0.1:8787/v1/openai/chat/completions' \ --header 'x-portkey-provider: deepinfra' \ --header 'Authorization: Bearer (added my key)' \ --header 'Content-Type: application/json' \ --data '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "deepinfra/airoboros-70b"}'
- The url needs to be
http://127.0.0.1:8787/v1/chat/completions. You do not have to send openai in the url as it is picked up from the config itself. - Also looks like the formatting of the curl is wrong. Here is a working version
curl --location 'http://127.0.0.1:8787/v1/chat/completions' \
--header 'x-portkey-provider: deepinfra' \
--header 'Authorization: Bearer (added my key)' \
--header 'Content-Type: application/json' \
--data '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "deepinfra/airoboros-70b"}'
I tried the above still same not sure why, if I change to other provider like mistral-ai, openai at least the request is being hit error or unauthorized access.
It might be possible that you are on a different branch or you forgot to restart your local server after making the latest changes.
It might be possible that you are on a different branch or you forgot to restart your local server after making the latest changes.
I tried but still same issue, are you able to use the provider from your side ?
Yes. Try putting in some logs at some checkpoints to see whats happening. Also make sure that the port is correct (if you are using wrangler in local)
Please request a re-review once you are done with all the changes and testing
Was able to run the provider and test the requests in postman, I had to use wrangler to run in local. Thank you again :)