cli icon indicating copy to clipboard operation
cli copied to clipboard

`Waiting for health checks...` running forever.

Open Shyrogan opened this issue 1 year ago • 3 comments

Describe the bug Recently I have been unable to launch my supabase local project using the supabase start command. The command remains stuck on Waiting for health check and never runs succesfully.

  1. Studio is available normally.
  2. Authentication or any service related to the database seem to work.

System information Rerun the failing command with --create-ticket flag.

  • Ticket ID: d888349bfa5f409fb91e7b4eb4f71d98
  • Version of OS: macOS d888349bfa5f409fb91e7b4eb4f71d98
  • Version of CLI: 1.178.2
  • Version of Docker: Docker version 26.1.4, build 5650f9b102
  • Versions of services:
        SERVICE IMAGE      │      LOCAL       │ LINKED
  ─────────────────────────┼──────────────────┼─────────
    supabase/postgres      │ 15.1.1.61        │ -
    supabase/gotrue        │ v2.151.0         │ -
    postgrest/postgrest    │ v12.0.1          │ -
    supabase/realtime      │ v2.28.32         │ -
    supabase/storage-api   │ v1.0.6           │ -
    supabase/edge-runtime  │ v1.54.6          │ -
    supabase/studio        │ 20240422-5cf8f30 │ -
    supabase/postgres-meta │ v0.80.0          │ -
    supabase/logflare      │ 1.4.0            │ -
    supabase/supavisor     │ 1.1.56           │ -
    darthsim/imgproxy      │ v3.8.0           │ -

✨  Done in 0.19s.

Shyrogan avatar Jun 25 '24 20:06 Shyrogan

Could you report which services are unhealthy? ie. when cli says Waiting for health check..., run the following command in another terminal.

docker ps -a

sweatybridge avatar Jun 27 '24 09:06 sweatybridge

CONTAINER ID   IMAGE                                             COMMAND                  CREATED              STATUS                        PORTS                                                                  NAMES
f6e2d0a98988   public.ecr.aws/supabase/studio:20240422-5cf8f30   "docker-entrypoint.s…"   45 seconds ago       Up 43 seconds (healthy)       0.0.0.0:54323->3000/tcp, :::54323->3000/tcp                            supabase_studio_mentor
12431e0a3e21   public.ecr.aws/supabase/postgres-meta:v0.80.0     "docker-entrypoint.s…"   45 seconds ago       Up 44 seconds (healthy)       8080/tcp                                                               supabase_pg_meta_mentor
818fcf85ea2e   public.ecr.aws/supabase/edge-runtime:v1.54.6      "sh -c 'cat <<'EOF' …"   45 seconds ago       Up 44 seconds                 0.0.0.0:32768->8081/tcp, :::32768->8081/tcp                            supabase_edge_runtime_mentor
2d16a4fadf28   public.ecr.aws/supabase/imgproxy:v3.8.0           "imgproxy"               46 seconds ago       Up 44 seconds (healthy)       8080/tcp                                                               supabase_imgproxy_mentor
1fc177aa7072   public.ecr.aws/supabase/storage-api:v1.0.6        "docker-entrypoint.s…"   46 seconds ago       Up 45 seconds (healthy)       5000/tcp                                                               supabase_storage_mentor
48ddd978df6c   public.ecr.aws/supabase/postgrest:v12.0.1         "postgrest"              46 seconds ago       Up 45 seconds                 3000/tcp                                                               supabase_rest_mentor
9461c1d93559   public.ecr.aws/supabase/realtime:v2.28.32         "/usr/bin/tini -s -g…"   47 seconds ago       Up 46 seconds (healthy)       4000/tcp                                                               supabase_realtime_mentor
4df7ed5dc2a5   public.ecr.aws/supabase/inbucket:3.0.3            "/start-inbucket.sh …"   47 seconds ago       Up 46 seconds (healthy)       1100/tcp, 2500/tcp, 0.0.0.0:54324->9000/tcp, :::54324->9000/tcp        supabase_inbucket_mentor
02a8792d2d43   public.ecr.aws/supabase/gotrue:v2.151.0           "auth"                   47 seconds ago       Up 46 seconds (healthy)       9999/tcp                                                               supabase_auth_mentor
c8e38227fed7   public.ecr.aws/supabase/kong:2.8.1                "sh -c 'cat <<'EOF' …"   47 seconds ago       Up 46 seconds (healthy)       8001/tcp, 8443-8444/tcp, 0.0.0.0:54321->8000/tcp, :::54321->8000/tcp   supabase_kong_mentor
d7ab72e1bbfc   public.ecr.aws/supabase/postgres:15.1.1.61        "sh -c 'cat <<'EOF' …"   About a minute ago   Up About a minute (healthy)   0.0.0.0:54322->5432/tcp, :::54322->5432/tcp                            supabase_db_mentor

Shyrogan avatar Jun 27 '24 09:06 Shyrogan

Here are the logs for postgrest:

27/Jun/2024:09:41:01 +0000: Starting PostgREST 12.0.1 (cd38da5)...
27/Jun/2024:09:41:01 +0000: Attempting to connect to the database...
27/Jun/2024:09:41:01 +0000: Connection successful
27/Jun/2024:09:41:01 +0000: Admin server listening on port 3001
27/Jun/2024:09:41:01 +0000: Listening on port 3000
27/Jun/2024:09:41:01 +0000: Listening for notifications on the pgrst channel
27/Jun/2024:09:41:01 +0000: Config reloaded
27/Jun/2024:09:41:01 +0000: Schema cache loaded

and edge-runtime:

Download https://deno.land/std/http/status.ts
Download https://deno.land/x/[email protected]/index.ts
Implicitly using latest version (0.224.0) for https://deno.land/std/http/status.ts
Download https://deno.land/[email protected]/http/status.ts
Download https://deno.land/x/[email protected]/jwe/compact/decrypt.ts
Download https://deno.land/x/[email protected]/jwe/flattened/decrypt.ts
Download https://deno.land/x/[email protected]/jwe/general/decrypt.ts
Download https://deno.land/x/[email protected]/jwe/general/encrypt.ts
Download https://deno.land/x/[email protected]/jws/compact/verify.ts
Download https://deno.land/x/[email protected]/jws/flattened/verify.ts
Download https://deno.land/x/[email protected]/jws/general/verify.ts
Download https://deno.land/x/[email protected]/jwt/verify.ts
Download https://deno.land/x/[email protected]/jwt/decrypt.ts
Download https://deno.land/x/[email protected]/jwe/compact/encrypt.ts
Download https://deno.land/x/[email protected]/jwe/flattened/encrypt.ts
Download https://deno.land/x/[email protected]/jws/compact/sign.ts
Download https://deno.land/x/[email protected]/jws/flattened/sign.ts
Download https://deno.land/x/[email protected]/jws/general/sign.ts
Download https://deno.land/x/[email protected]/jwt/sign.ts
Download https://deno.land/x/[email protected]/jwt/encrypt.ts
Download https://deno.land/x/[email protected]/jwk/thumbprint.ts
Download https://deno.land/x/[email protected]/jwk/embedded.ts
Download https://deno.land/x/[email protected]/jwks/local.ts
Download https://deno.land/x/[email protected]/jwks/remote.ts
Download https://deno.land/x/[email protected]/jwt/unsecured.ts
Download https://deno.land/x/[email protected]/key/export.ts
Download https://deno.land/x/[email protected]/key/import.ts
Download https://deno.land/x/[email protected]/util/decode_protected_header.ts
Download https://deno.land/x/[email protected]/util/decode_jwt.ts
Download https://deno.land/x/[email protected]/util/errors.ts
Download https://deno.land/x/[email protected]/key/generate_key_pair.ts
Download https://deno.land/x/[email protected]/key/generate_secret.ts
Download https://deno.land/x/[email protected]/util/base64url.ts
Download https://deno.land/x/[email protected]/lib/buffer_utils.ts
Download https://deno.land/x/[email protected]/runtime/base64url.ts
Download https://deno.land/x/[email protected]/runtime/decrypt.ts
Download https://deno.land/x/[email protected]/runtime/zlib.ts
Download https://deno.land/x/[email protected]/lib/is_disjoint.ts
Download https://deno.land/x/[email protected]/lib/is_object.ts
Download https://deno.land/x/[email protected]/lib/decrypt_key_management.ts
Download https://deno.land/x/[email protected]/lib/cek.ts
Download https://deno.land/x/[email protected]/lib/validate_crit.ts
Download https://deno.land/x/[email protected]/lib/validate_algorithms.ts
Download https://deno.land/x/[email protected]/lib/encrypt_key_management.ts
Download https://deno.land/x/[email protected]/runtime/verify.ts
Download https://deno.land/x/[email protected]/lib/check_key_type.ts
Download https://deno.land/x/[email protected]/lib/jwt_claims_set.ts
Download https://deno.land/x/[email protected]/runtime/encrypt.ts
Download https://deno.land/x/[email protected]/lib/iv.ts
Download https://deno.land/x/[email protected]/runtime/sign.ts
Download https://deno.land/x/[email protected]/jwt/produce.ts
Download https://deno.land/x/[email protected]/runtime/digest.ts
Download https://deno.land/x/[email protected]/runtime/fetch_jwks.ts
Download https://deno.land/x/[email protected]/runtime/env.ts
Download https://deno.land/x/[email protected]/runtime/asn1.ts
Download https://deno.land/x/[email protected]/runtime/key_to_jwk.ts
Download https://deno.land/x/[email protected]/runtime/jwk_to_key.ts
Download https://deno.land/x/[email protected]/runtime/generate.ts
Download https://deno.land/x/[email protected]/lib/check_iv_length.ts
Download https://deno.land/x/[email protected]/runtime/check_cek_length.ts
Download https://deno.land/x/[email protected]/runtime/timing_safe_equal.ts
Download https://deno.land/x/[email protected]/runtime/webcrypto.ts
Download https://deno.land/x/[email protected]/lib/crypto_key.ts
Download https://deno.land/x/[email protected]/lib/invalid_key_input.ts
Download https://deno.land/x/[email protected]/runtime/is_key_like.ts
Download https://deno.land/x/[email protected]/runtime/aeskw.ts
Download https://deno.land/x/[email protected]/runtime/ecdhes.ts
Download https://deno.land/x/[email protected]/runtime/pbes2kw.ts
Download https://deno.land/x/[email protected]/runtime/rsaes.ts
Download https://deno.land/x/[email protected]/lib/aesgcmkw.ts
Download https://deno.land/x/[email protected]/runtime/random.ts
Download https://deno.land/x/[email protected]/runtime/subtle_dsa.ts
Download https://deno.land/x/[email protected]/runtime/check_key_length.ts
Download https://deno.land/x/[email protected]/runtime/get_sign_verify_key.ts
Download https://deno.land/x/[email protected]/lib/epoch.ts
Download https://deno.land/x/[email protected]/lib/secs.ts
Download https://deno.land/x/[email protected]/lib/format_pem.ts
Download https://deno.land/x/[email protected]/runtime/bogus.ts
Download https://deno.land/x/[email protected]/lib/check_p2s.ts
Download https://deno.land/x/[email protected]/runtime/subtle_rsaes.ts
Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.54.6 (compatible with Deno v1.43.0)

Shyrogan avatar Jun 27 '24 09:06 Shyrogan

Could you kindly check if this is still reproducible in the beta release?

npx supabase@beta start

sweatybridge avatar Jul 06 '24 15:07 sweatybridge

Same issue here, but the beta did not change it. Could it be an issue with the config.toml file?

config.toml.txt

Got this when health check failed.

Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.55.0 (compatible with Deno v1.43.0)
Stopping containers...
failed to execute http request: Head "http://127.0.0.1:54321/rest-admin/v1/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to execute http request: Head "http://127.0.0.1:54321/functions/v1/_internal/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Try rerunning the command with --debug to troubleshoot the error.

sosensible avatar Jul 12 '24 13:07 sosensible

Hmm I'm not sure what's exactly wrong as I can't seem to reproduce locally on macOS. But as a temporary workaround, you can pass in supabase start --ignore-health-check flag.

https://supabase.com/docs/reference/cli/introduction

sweatybridge avatar Jul 12 '24 13:07 sweatybridge

Same issue here, but the beta did not change it. Could it be an issue with the config.toml file?

config.toml.txt

Got this when health check failed.

Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.55.0 (compatible with Deno v1.43.0)
Stopping containers...
failed to execute http request: Head "http://127.0.0.1:54321/rest-admin/v1/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to execute http request: Head "http://127.0.0.1:54321/functions/v1/_internal/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Try rerunning the command with --debug to troubleshoot the error.

Got it working by recreating the project to default configuration, looks like there is a setting in newer versions blocking us, maybe you could post your project configuration to see ?

Shyrogan avatar Jul 12 '24 14:07 Shyrogan

Hmm I'm not sure what's exactly wrong as I can't seem to reproduce locally on macOS. But as a temporary workaround, you can pass in supabase start --ignore-health-check flag.

https://supabase.com/docs/reference/cli/introduction

I'm having this same problem, and oddly even with the --ignore-health-check flag, the cli still hangs on Waiting for health checks...

taylorkangbeck avatar Jul 14 '24 20:07 taylorkangbeck

Hmm I'm not sure what's exactly wrong as I can't seem to reproduce locally on macOS. But as a temporary workaround, you can pass in supabase start --ignore-health-check flag. https://supabase.com/docs/reference/cli/introduction

I'm having this same problem, and oddly even with the --ignore-health-check flag, the cli still hangs on Waiting for health checks...

Can you try the solution I sent above ?

Shyrogan avatar Jul 15 '24 14:07 Shyrogan

Same issue here. I running on macOS 15. Could the issue be related to this macOS version somehow?

rogix avatar Jul 17 '24 16:07 rogix

Same issue here, but the beta did not change it. Could it be an issue with the config.toml file? config.toml.txt Got this when health check failed.

Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.55.0 (compatible with Deno v1.43.0)
Stopping containers...
failed to execute http request: Head "http://127.0.0.1:54321/rest-admin/v1/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to execute http request: Head "http://127.0.0.1:54321/functions/v1/_internal/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Try rerunning the command with --debug to troubleshoot the error.

Got it working by recreating the project to default configuration, looks like there is a setting in newer versions blocking us, maybe you could post your project configuration to see ?

What do you mean by "recreating the project to default configuration"? Would you mind sharing how to do this? Thanks

rogix avatar Jul 17 '24 16:07 rogix

+1 on how you recreated the project to default configuration.

sosensible avatar Jul 17 '24 18:07 sosensible

recreated the project to default configuration

You can try these commands to reinitialise the local project config. I'm not 100% sure if it solves the problem though.

supabase stop --no-backup
supabase init --force

sweatybridge avatar Jul 17 '24 20:07 sweatybridge

Same issue here, but the beta did not change it. Could it be an issue with the config.toml file? config.toml.txt Got this when health check failed.

Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.55.0 (compatible with Deno v1.43.0)
Stopping containers...
failed to execute http request: Head "http://127.0.0.1:54321/rest-admin/v1/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to execute http request: Head "http://127.0.0.1:54321/functions/v1/_internal/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Try rerunning the command with --debug to troubleshoot the error.

Got it working by recreating the project to default configuration, looks like there is a setting in newer versions blocking us, maybe you could post your project configuration to see ?

What do you mean by "recreating the project to default configuration"? Would you mind sharing how to do this? Thanks

+1 on how you recreated the project to default configuration.

Simply save your current configuration somewhere and reinitialize it.

Here is the commit that fixed the project on my side (I believe): Screenshot 2024-07-18 at 09-11-48 chore fix supabase · Shyrogan_mentor@3c57073

Shyrogan avatar Jul 18 '24 07:07 Shyrogan

Yep! Just switching enabled to false worked for me. Thanks!

rogix avatar Jul 18 '24 16:07 rogix

@sweatybridge closing the issue as there is a "fix" but feel free to re-open it if you need it.

Happy coding everyone!

Shyrogan avatar Jul 22 '24 22:07 Shyrogan

Getting the same error despite following all of the above instructions on supabase 1.223.10

Debug log:

Task db:up pnpm supabase start "--debug"
Supabase CLI 1.223.10
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Sent Header: Content-Length [265]
2024/11/20 01:19:18 Sent Header: Content-Type [application/json]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Sent Header: Content-Length [121]
2024/11/20 01:19:18 Sent Header: Content-Type [application/json]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Sent Header: Content-Length [125]
2024/11/20 01:19:18 Sent Header: Content-Type [application/json]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Sent Header: Content-Length [12735]
2024/11/20 01:19:18 Sent Header: Content-Type [application/json]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:18 Recv First Byte
2024/11/20 01:19:18 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:18 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:18 Sent Header: Content-Length [0]
2024/11/20 01:19:18 Send Done
2024/11/20 01:19:19 Recv First Byte
2024/11/20 01:19:19 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:19 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:19 Send Done
2024/11/20 01:19:19 Recv First Byte
2024/11/20 01:19:20 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:20 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:20 Send Done
2024/11/20 01:19:20 Recv First Byte
2024/11/20 01:19:21 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:21 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:21 Send Done
2024/11/20 01:19:21 Recv First Byte
2024/11/20 01:19:22 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:22 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:22 Send Done
2024/11/20 01:19:22 Recv First Byte
2024/11/20 01:19:23 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:23 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:23 Send Done
2024/11/20 01:19:23 Recv First Byte
2024/11/20 01:19:24 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:24 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:24 Send Done
2024/11/20 01:19:24 Recv First Byte
2024/11/20 01:19:25 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:25 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:25 Send Done
2024/11/20 01:19:25 Recv First Byte
2024/11/20 01:19:26 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:26 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:26 Send Done
2024/11/20 01:19:26 Recv First Byte
2024/11/20 01:19:27 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:27 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:27 Send Done
2024/11/20 01:19:27 Recv First Byte
2024/11/20 01:19:28 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:28 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:28 Send Done
2024/11/20 01:19:28 Recv First Byte
2024/11/20 01:19:29 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:29 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:29 Send Done
2024/11/20 01:19:29 Recv First Byte
2024/11/20 01:19:30 Sent Header: Host [api.moby.localhost]
2024/11/20 01:19:30 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2024/11/20 01:19:30 Send Done
2024/11/20 01:19:30 Recv First Byte
2024/11/20 01:19:30 Connect Start: tcp 127.0.0.1:54322
2024/11/20 01:19:32 Connect Error: tcp 127.0.0.1:54322 dial tcp 127.0.0.1:54322: i/o timeout
Stopping containers...
Pruned containers: [3f649d1d9c453146d7db79a70b79caee2b73e435a29bdd7f83fb8e68e27b3899]
Pruned volumes: [supabase_db_platform supabase_config_platform]
Pruned network: [supabase_network_platform]
failed to connect to postgres: failed to connect to `host=127.0.0.1 user=postgres database=postgres`: dial error (timeout: dial tcp 127.0.0.1:54322: i/o timeout)

therecluse26 avatar Nov 20 '24 06:11 therecluse26