PrefectHQ
Graphql Apollo service failure under Ubuntu 20.04 LTS
Description
The initial problem was detected by following the prefect installation instructions using pip under python 3.7.6. After a successful installation sequence and starting the local server using 'prefect server start', the service stack runs up but an error connecting to the graphql server fails. Attempts using 'prefect backend server' caused no change in the problem. Editing of both the local .prefect/config.toml and the one installed under site-packages/prfect/config.toml to turn on debugging and force different IP address assignments didn't produce any difference in behavior.
2020-06-22T21:26:17.180Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.28.0.4:4201 apollo_1 | 2020-06-22T21:26:17.181Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) apollo_1 | 2020-06-22T21:26:17.182Z apollo_1 | Trying again in 3 seconds...
Corresponding to this error was a API status problem on the web UI:
Several different system configurations (conda 3.7, python 3.8 pip install no conda) and modifications of the configuration file were tried. The developer instructions :
https://github.com/PrefectHQ/prefect/tree/master/server
Were then followed to download and build sub-components of the service stack. In particular the apollo service was independently tested :
prefect/server/services/apollo$ npm install> [email protected] postinstall /home/av_developer/Downloads/prefect/server/services/apollo/node_modules/@babel/node/node_modules/core-js
node -e "try{require('./postinstall')}catch(e){}"
Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
https://opencollective.com/core-js https://www.patreon.com/zloirock
Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
[email protected] postinstall /home/av_developer/Downloads/prefect/server/services/apollo/node_modules/apollo-env/node_modules/core-js node -e "try{require('./postinstall')}catch(e){}"
[email protected] postinstall /home/av_developer/Downloads/prefect/server/services/apollo/node_modules/core-js node -e "try{require('./postinstall')}catch(e){}"
@apollo/[email protected] postinstall /home/av_developer/Downloads/prefect/server/services/apollo/node_modules/@apollo/protobufjs node scripts/postinstall
[email protected] postinstall /home/av_developer/Downloads/prefect/server/services/apollo/node_modules/nodemon node bin/postinstall || exit 0
Love nodemon? You can now support the project via the open collective:
https://opencollective.com/nodemon/donate
npm WARN apollo No repository field. npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents): npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 1090 packages from 610 contributors and audited 1165 packages in 16.751s
and then tested:
prefect/server/services/apollo
nodemon --exec babel-node src/index.js
[nodemon] 1.19.4
[nodemon] to restart at any time, enter rs
[nodemon] watching dir(s): src//* ../../src//*
[nodemon] watching extensions: js,mjs,coffee,litcoffee,json,py
[nodemon] starting babel-node src/index.js
2020-06-22T21:30:34.188Z Error fetching GraphQL health: FetchError: request to http://localhost:4201/health failed, reason: connect ECONNREFUSED 127.0.0.1:4201
2020-06-22T21:30:34.197Z Error: Could not safely build a schema!
at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js:126:11)
at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema!
at safelyBuildSchema (/home/av_developer/Downloads/prefect/server/services/apollo/src/index.js:126:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
2020-06-22T21:30:34.198Z
Trying again in 3 seconds...
producing the same error as in the prior installation attempts.
Expected Behavior
I expected the installation guide instructions to result in a working test platform and the grahql service to startup and allow proper connection.
Reproduction
This problem occurred consistently for me for all install attempts using the Ubuntu 20.04 LTS installation under VMware Fusion virtualization. The method of reproduction involving the least amount of software was to follow the developer instructions and then build and test only the apollo service sub-component under prefect/server/services/apollo.
Environment
Ubuntu 20.04 LTS platform with all updates running under VMware Fusion 11.5.5 under Mac OS X 10.15.4. Anaconda 4.8.3 was used for many of the installation attempts with python 3.7.6. npm version was 6.14.4.
Example 3.8 env :
prefect diagnostics { "config_overrides": { "server": { "ui": { "graphql_url": true } } }, "env_vars": [], "system_information": { "platform": "Linux-5.4.0-37-generic-x86_64-with-glibc2.29", "prefect_version": "0.12.0", "python_version": "3.8.2" } }
Example anaconda env :
prefect diagnostics { "config_overrides": { "server": { "ui": { "graphql_url": true } } }, "env_vars": [], "system_information": { "platform": "Linux-5.4.0-37-generic-x86_64-with-debian-bullseye-sid", "prefect_version": "0.12.0", "python_version": "3.7.6" } }
@fdlind Yeah I don't expect the apollo service to start up without the rest of the stack running (it's the top layer that requires the graphql service which requires postgres etc.) When you run prefect server start do you see the migrations being applied to the Postgres database? Wondering if somehow they're being missed or are unable to run. There might also be a port issue where some service is not able to be created due to something using the default port. You could try setting different ports for the prefect server start command and that may resolve it if there's a conflict:
--postgres-port TEXT Port used to serve Postgres, defaults to '5432'
--hasura-port TEXT Port used to serve Hasura, defaults to '3001'
--graphql-port TEXT Port used to serve the GraphQL API, defaults to '4001'
--ui-port TEXT Port used to serve the UI, defaults to '8080'
--server-port TEXT Port used to serve the Core server, defaults to '4200'
Hi,
Make a clean conda install per standard instructions. Tried different port combinations from the command line as above. Doesn't seem to actually change the server ports. Also edited the toml config file manually and that will change ports but it doesn't fix the issue.
The trace associated with the clean conda install and server startup with an alternate graphql port is :
(av_prefect2) av_developer@ubuntu:~$ prefect server start --graphql-port 3100 Pulling postgres ... done Pulling hasura ... done Pulling graphql ... done Pulling apollo ... done Pulling scheduler ... done Pulling ui ... done Creating network "prefect-server" with the default driver Creating tmp_postgres_1 ... done Creating tmp_hasura_1 ... done Creating tmp_graphql_1 ... done Creating tmp_scheduler_1 ... done Creating tmp_apollo_1 ... done Creating tmp_ui_1 ... done Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_scheduler_1, tmp_apollo_1, tmp_ui_1 apollo_1 | apollo_1 | > @ serve /apollo apollo_1 | > node dist/index.js apollo_1 | graphql_1 | graphql_1 | Running Alembic migrations... graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 723da8052cbd, create extensions graphql_1 | INFO [alembic.runtime.migration] Running upgrade 723da8052cbd -> 2f249ccdcba7, set utc timezone graphql_1 | INFO [alembic.runtime.migration] Running upgrade 2f249ccdcba7 -> 7ff37edbf446, Initial migration graphql_1 | INFO [alembic.runtime.migration] Running upgrade 7ff37edbf446 -> 6c48e68b5e97, Add task run trigger graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6c48e68b5e97 -> aa594a91210c, Add run timing details trigger graphql_1 | INFO [alembic.runtime.migration] Running upgrade aa594a91210c -> 8666a0ceb70e, Add trigger to update runs when states are inserted graphql_1 | INFO [alembic.runtime.migration] Running upgrade 8666a0ceb70e -> 6b0389a57a0a, Add updated trigger to all tables graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6b0389a57a0a -> 7e65dadba625, fix log default timestamp graphql_1 | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5989db8d0>: Failed to establish a new connection: [Errno 111] Connection refused')) graphql_1 | graphql_1 | Could not upgrade the database! hasura_1 | {"type":"startup","timestamp":"2020-07-15T21:58:58.629+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}} hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host "postgres" (172.31.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"} postgres_1 | The files belonging to this database system will be owned by user "postgres". postgres_1 | This user must also own the server process. postgres_1 | postgres_1 | The database cluster will be initialized with locale "en_US.utf8". postgres_1 | The default database encoding has accordingly been set to "UTF8". postgres_1 | The default text search configuration will be set to "english". postgres_1 | postgres_1 | Data page checksums are disabled. postgres_1 | postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok postgres_1 | creating subdirectories ... ok postgres_1 | selecting default max_connections ... 100 postgres_1 | selecting default shared_buffers ... 128MB postgres_1 | selecting default timezone ... Etc/UTC postgres_1 | selecting dynamic shared memory implementation ... posix postgres_1 | creating configuration files ... ok postgres_1 | running bootstrap script ... ok postgres_1 | performing post-bootstrap initialization ... ok postgres_1 | syncing data to disk ... ok postgres_1 | postgres_1 | WARNING: enabling "trust" authentication for local connections postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or postgres_1 | --auth-local and --auth-host, the next time you run initdb. postgres_1 | postgres_1 | Success. You can now start the database server using: postgres_1 | postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start postgres_1 | postgres_1 | waiting for server to start....2020-07-15 21:58:58.417 UTC [45] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-07-15 21:58:58.442 UTC [46] LOG: database system was shut down at 2020-07-15 21:58:58 UTC postgres_1 | 2020-07-15 21:58:58.450 UTC [45] LOG: database system is ready to accept connections postgres_1 | done postgres_1 | server started postgres_1 | CREATE DATABASE postgres_1 | postgres_1 | postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/ postgres_1 | postgres_1 | waiting for server to shut down...2020-07-15 21:58:58.832 UTC [45] LOG: received fast shutdown request postgres_1 | .2020-07-15 21:58:58.834 UTC [45] LOG: aborting any active transactions postgres_1 | 2020-07-15 21:58:58.835 UTC [45] LOG: background worker "logical replication launcher" (PID 52) exited with exit code 1 postgres_1 | 2020-07-15 21:58:58.836 UTC [47] LOG: shutting down scheduler_1 | [2020-07-15 21:59:01,520] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 264 seconds... postgres_1 | 2020-07-15 21:58:58.848 UTC [45] LOG: database system is shut down postgres_1 | done postgres_1 | server stopped postgres_1 | postgres_1 | PostgreSQL init process complete; ready for start up. postgres_1 | postgres_1 | 2020-07-15 21:58:58.953 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres_1 | 2020-07-15 21:58:58.953 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres_1 | 2020-07-15 21:58:58.965 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres_1 | 2020-07-15 21:58:58.981 UTC [63] LOG: database system was shut down at 2020-07-15 21:58:58 UTC postgres_1 | 2020-07-15 21:58:58.996 UTC [1] LOG: database system is ready to accept connections postgres_1 | 2020-07-15 21:59:00.483 UTC [71] ERROR: duplicate key value violates unique constraint "pg_extension_name_index" postgres_1 | 2020-07-15 21:59:00.483 UTC [71] DETAIL: Key (extname)=(pgcrypto) already exists. postgres_1 | 2020-07-15 21:59:00.483 UTC [71] STATEMENT: CREATE EXTENSION IF NOT EXISTS pgcrypto SCHEMA public ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾 apollo_1 | 2020-07-15T21:59:02.760Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.31.0.4:4201 apollo_1 | 2020-07-15T21:59:02.761Z Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema! apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11) apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) apollo_1 | 2020-07-15T21:59:02.762Z apollo_1 | Trying again in 3 seconds... apollo_1 | apollo_1 | 2020-07-15T21:59:05.772Z Building schema... apollo_1 | 2020-07-15T21:59:06.464Z Building schema complete! apollo_1 | Server ready at http://0.0.0.0:4200 🚀 apollo_1 | 2020-07-15T21:59:06.577Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"009531e9-baa7-42f6-8ba0-266a47db7dd1"}}