ChatGPTNextWeb
[Feature Request]: Vercel Functions for Hobby can now run up to 60 seconds
Problem Description
You'll get a timeout error if no data is returned from the API after about 25 seconds when deploying the application in Vercel Hobby.
Solution Description
Vercel Functions for Hobby can now run up to 60 seconds. https://vercel.com/changelog/vercel-functions-for-hobby-can-now-run-up-to-60-seconds I'm wondering if the above change can be implemented.
Alternatives Considered
No response
Additional Context
No response
The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
So I think maybe Edge runtime is involved. According to https://vercel.com/docs/functions/configuring-functions/duration.
You can't configure a maximum duration for functions using the Edge runtime. They can run indefinitely provided they send an initial response within 25 seconds.
About api/openai/[...path]: If using edge, as long as the first byte arrives within 25 seconds, the streaming can continue indefinitely. If using nodejs, regardless of when the first byte arrives, the total transmission time will be cut off if it exceeds the max-duration. So the conclusion is to keep api/openai/[...path] as it is——set it as an edge function.
https://github.com/Hk-Gosuto/ChatGPT-Next-Web-LangChain/issues/258#issuecomment-2128771868
@maxduke Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
https://github.com/vercel/ai/issues/487#issuecomment-1847754042
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂 I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
@maxduke Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂 I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
reopened. Thanks a lot for the information. Let’s see.
@SuiYunsy
Here's a workaround for this problem (comments are in Chinese)
codes changed in app/api/google.ts from line 116 to the end. old codes are commented.
// --- 添加一个两秒的心跳，防止流式一直不出结果 ---
let intervalId: any;
let responseStream: ReadableStream;
const transformStream = new TransformStream();
const writer = transformStream.writable.getWriter();
const encoder = new TextEncoder();
// 启动一个心跳定时器，每 2 秒发送一个无用 SSE 块，以保持及时响应
intervalId = setInterval(() => {
writer.write(encoder.encode(': Waiting...\n\n'));
console.log("[Alive] Sent keep-alive");
}, 2000);
// 异步运行“真流”
(async () => {
try {
const res = await fetch(fetchUrl, fetchOptions);
if (!res.ok) {
const errorBody = await res.text();
const errorEvent = `data: ${JSON.stringify({ error: true, message: `Upstream Error: ${res.status} ${res.statusText}`, body: errorBody })}\n\n`;
writer.write(encoder.encode(errorEvent));
throw new Error(`[Alive] Upstream fetch failed with status ${res.status}`);
}
// @ts-ignore
const reader = res.body.getReader();
const decoder = new TextDecoder();
let done = false;
let isFirstChunk = true;
// 将“真流”的输入写入我们一开始立即返回的“假流”里
while (!done) {
const { value, done: isDone } = await reader.read();
done = isDone;
if (value) {
writer.write(encoder.encode(decoder.decode(value, { stream: true })));
}
if (isFirstChunk) {
clearInterval(intervalId);
intervalId = null;
isFirstChunk = false;
console.log("[Alive] First chunk received, clearing keep-alive interval.");
}
}
} catch (e) {
console.error("[Alive] Stream fetch error:", e);
const errorMessage = `data: ${JSON.stringify({ error: true, message: (e as Error).message })}\n\n`;
writer.write(encoder.encode(errorMessage));
} finally {
clearTimeout(timeoutId);
if (intervalId) {
clearInterval(intervalId);
intervalId = null;
console.log("[Alive] Stream ended, clearing interval.");
}
writer.close();
console.log("[Alive] Stream writer closed.");
}
})().catch(e => {
console.error("[Alive] Other error:", e);
});
// 立即输出一个“假流”，这样保证有返回值
responseStream = transformStream.readable;
return new Response(responseStream, {
status: 200,
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
},
});
// try {
// const res = await fetch(fetchUrl, fetchOptions);
// // to prevent browser prompt for credentials
// const newHeaders = new Headers(res.headers);
// newHeaders.delete("www-authenticate");
// // to disable nginx buffering
// newHeaders.set("X-Accel-Buffering", "no");
// return new Response(res.body, {
// status: res.status,
// statusText: res.statusText,
// headers: newHeaders,
// });
// } finally {
// clearTimeout(timeoutId);
// }
