Run returning obfuscated error (400 error given a valid request)

Open bartolomej opened this issue 9 months ago • 0 comments

I wanted to run evaluations for this model https://replicate.com/cuuupid/gte-qwen2-7b-instruct.

But I kept getting this error when I triggered larger number of "run" requests:

Unknown Error: invalid character '<' looking for beginning of value

I couldn't get the info I needed from the default client error response, so I forked the repo to add more debug info: https://github.com/bartolomej/replicate-go/pull/1

This is the full http body sent to the API:

{
  "input": {
    "text": [
      "About 300 words of normal sanitized text (without any special characters/newlines/...."
    ]
  },
  "version": "67b1736bae9312a321217b2e10547882943b9e4a285eac4cba4043fab954b954"
}

And this is the full response body:

<html>
  <head>
    <title>400 Bad Request</title>
  </head>
  <body>
    <center><h1>400 Bad Request</h1></center>
    <hr>
    <center>cloudflare</center>
  </body>
</html>

Example Go code I used:

func (r *EmbeddingModel) Create(ctx context.Context, reqs []types.CreateEmbeddingRequest) ([][]float32, error) {
	text := make([]string, 0, len(reqs))

	for _, r := range reqs {
                // r.Instruct and r.Query were both sanitized, without any special characters or newlines
		text = append(text, fmt.Sprintf("Instruct: %s Query: %s", r.Instruct, r.Query))
	}

	res, err := r.client.Run(ctx, r.modelID, map[string]any{
		"text": text,
	}, nil)
	if err != nil {
		return nil, errors.Wrap(err, "replicate run")
	}

	// ... irrelevant response parsing code ...

	return out, nil
}

Then I refactored my code to send the request via barebones http client and started receiving 429 (error limit) responses:

payload := map[string]interface{}{
		"version": r.modelVersion,
		"input": map[string]interface{}{
			"text": text,
		},
	}

	body, err := json.Marshal(payload)
	if err != nil {
		return nil, errors.Wrap(err, "marshal json")
	}

	req, err := http.NewRequestWithContext(ctx, "POST", "https://api.replicate.com/v1/predictions", bytes.NewReader(body))
	if err != nil {
		return nil, errors.Wrap(err, "build request")
	}

	req.Header.Set("Authorization", "Bearer "+r.apiToken)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Prefer", "wait")

	client := &http.Client{Timeout: 60 * time.Second}
	resp, err := client.Do(req)
	if err != nil {
		return nil, errors.Wrap(err, "http request")
	}
	defer resp.Body.Close()

	respBody, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, errors.Wrap(err, "read response body")
	}

In summary:

I'm I just misusing the replicate client (doesn't seem like it)?
... or is the Replicate API / proxy returning some weird responses for certain requests (400 instead of 429)?
I feel like these edge case responses could be better handled to include more debug info for cases like these

Apr 17 '25 19:04 bartolomej