aphrodite-engine
aphrodite-engine copied to clipboard
[Bug]: Docker container refuses connection (read ECONNRESET)
Your current environment
This is about the Docker environment and the standard image, so probably does not apply
Docker related info: command to start:
docker run -e MODEL_NAME="TheBloke/Mistral-7B-Instruct-v0.2-GPTQ" -p 2242:2242 -e DATATYPE='half' --gpus 4 -e NUM_GPUS="4" --ipc host alpindale/aphrodite-engine
this is basically the command from the Readme here, except --gpus all would not actually work unless the env variable NUM_GPUS is set
also, my GPUs required -e DATATYPE='half'
Docker container is definitely running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
76c02028bbe1 alpindale/aphrodite-engine "/app/aphrodite-engi…" 27 minutes ago Up 27 minutes 0.0.0.0:2242->2242/tcp, :::2242->2242/tcp, 7860/tcp infallible_ishizaka
🐛 Describe the bug
SillyTaverns is running on the same host and the connection setting is for Aphrodite and http://127.0.0.1:2242/ and
I am always getting this error:
Trying to connect to API: {
api_server: 'http://127.0.0.1:2242',
api_type: 'aphrodite',
legacy_api: false
}
FetchError: request to http://127.0.0.1:2242/v1/models failed, reason: read ECONNRESET
at ClientRequest.<anonymous> (/home/boss/ai/SillyTavern/node_modules/node-fetch/lib/index.js:1505:11)
at ClientRequest.emit (node:events:519:28)
at Socket.socketErrorListener (node:_http_client:500:9)
at Socket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET'
Additional test with Curl to eliminate SillyTavern issues:
curl http://localhost:2242/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ",
"prompt": "Hello, who are you?",
"stream": false,
"mirostat_mode": 2,
"mirostat_tau": 6.5,
"mirostat_eta": 0.2
}'
curl: (56) Recv failure: Connection reset by peer
I have also tried to launch aphrodites from /docker subdirectory using docker compose up -d and the command line (above) - always the same result, it would not accept connections. Local SillyTavern, remote - Docker never accepts connections. Help!