aphrodite-engine icon indicating copy to clipboard operation
aphrodite-engine copied to clipboard

[Bug]: Docker container refuses connection (read ECONNRESET)

Open elabz opened this issue 1 year ago • 0 comments

Your current environment

This is about the Docker environment and the standard image, so probably does not apply

Docker related info: command to start:

docker run  -e MODEL_NAME="TheBloke/Mistral-7B-Instruct-v0.2-GPTQ" -p 2242:2242 -e DATATYPE='half' --gpus 4 -e NUM_GPUS="4" --ipc host alpindale/aphrodite-engine

this is basically the command from the Readme here, except --gpus all would not actually work unless the env variable NUM_GPUS is set also, my GPUs required -e DATATYPE='half'

Docker container is definitely running:

$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS          PORTS                                                 NAMES
76c02028bbe1   alpindale/aphrodite-engine   "/app/aphrodite-engi…"   27 minutes ago   Up 27 minutes   0.0.0.0:2242->2242/tcp, :::2242->2242/tcp, 7860/tcp   infallible_ishizaka

🐛 Describe the bug

SillyTaverns is running on the same host and the connection setting is for Aphrodite and http://127.0.0.1:2242/ and I am always getting this error:

Trying to connect to API: {
  api_server: 'http://127.0.0.1:2242',
  api_type: 'aphrodite',
  legacy_api: false
}
FetchError: request to http://127.0.0.1:2242/v1/models failed, reason: read ECONNRESET
    at ClientRequest.<anonymous> (/home/boss/ai/SillyTavern/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:519:28)
    at Socket.socketErrorListener (node:_http_client:500:9)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNRESET',
  code: 'ECONNRESET'

Additional test with Curl to eliminate SillyTavern issues:

curl http://localhost:2242/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ",
  "prompt": "Hello, who are you?",
  "stream": false,
  "mirostat_mode": 2,
  "mirostat_tau": 6.5,
  "mirostat_eta": 0.2
}'
curl: (56) Recv failure: Connection reset by peer

I have also tried to launch aphrodites from /docker subdirectory using docker compose up -d and the command line (above) - always the same result, it would not accept connections. Local SillyTavern, remote - Docker never accepts connections. Help!

Screenshot 2024-06-04 at 12 59 46 AM

elabz avatar Jun 04 '24 05:06 elabz