PrefectHQ
Prefect not picking up my Postgres database.
I added my database properties to both backend.toml and config.toml files. When I execute 'prefect config', the config json shows my database credentials. But when I execute 'prefect server start', prefect pulls docker image for postgres and does not use my external database.
@madkinsz Unfortunately, I had terminated that Virtual Machine from my cloud platform. I tried to create another setup but am seeing some different problem. I execute the following commands:
pip install prefect prefect backend server
I am getting the following error: ImportError: cannot import name 'stringify' from 'dask.utils' (/anaconda/envs/py37_default/lib/python3.7/site-packages/dask/utils.py)
I have docker and docker-compose installed on my machine. Please advise for this error.
This looks like an issue with your dask/distributed versions. I can't provide additional advice without more information
@madkinsz Attaching 2 files. The 1st one is Config.txt. It shows my docker is active. I have 2 config files - backend.toml and config.toml. I have a requirement to use my local Postgres database with Prefect. So I have added details to config.toml. The 2nd file is prefect server start error related to docker service. I have docker running, even then I get error related to it.
I tried command - ip route | grep docker0 | awk '{print $9}' Received response - 172.17.0.1. Tried setting DOCKER_HOST=172.17.0.1. But it did not work.
Please review the details that I have provided and advise on how to more forward with Prefect. Let me know in case you still need any more details.
Do you have problems when you use an unmodified config? I do not believe this is an issue with Prefect and looks like a problem with your docker setup.
Can you run docker-compose pull from the src/prefect/cli/ directory? (if you're not sure where this is -- python -c "import prefect; print(prefect.__file__)" can you get started)
docker-compose up won't work without all the environment variables set but you should able to use it to debug past this current error.
@madkinsz 1. I saw a problem using unmodified config. Later installing psutil fixed it. Without configuration, I am able to start Prefect. Refer file - Prefect Server Start without Config.txt 2. docker-compose pull - Worked for me. docker-compose up - Did not work. Refer attachment docker-compose executions.txt to see the errors. 3. Further I tried to start Prefect with my configuration file. It started but I am not sure if it connected Postgres database installed in my machine or not. Refer Prefect Server Start with Config.txt for more details. I used prefect server start --no-postgres-port to start Prefect because my local database is also running on 5432 and it was conflicting. I believe that --no-postgres-port does not configure postgres from docker image and uses my database. If not then please correct me.
Prefect Server Start without Config.txt Prefect Server Start with Config.txt docker-compose executions.txt
This isn't really a supported pattern for prefect server start but there may be a workaround. There's no way to stop the postgres container from being created so --no-postgres-port stops the created container from binding to your local address so they no longer conflict. However, it looks like at https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/server.py#L19 the localhost is replaced with postgres to use the docker-compose internal communication so it is likely that it is connecting to the wrong postgres instance still. I believe if you change localhost to 127.0.0.1 or 0.0.0.0 it will no longer be replaced and it should connect to your database!
I've created an issue to improve this functionality at https://github.com/PrefectHQ/prefect/issues/3891
@madkinsz Thank you for your analysis and issue reporting. I already had localhost mapped to 127.0.0.1 in my system. I tried changing it to 0.0.0.0. Please refer attached file. I still do not see my local database getting connected with Prefect. Please advise.
Do you think I should wait for https://github.com/PrefectHQ/prefect/issues/3891 to be fixed first?
@madkinsz Thank you for your analysis and issue reporting. I already had localhost mapped to 127.0.0.1 in my system. I tried changing it to 0.0.0.0. Please refer attached file. I still do not see my local database getting connected with Prefect. Please advise.
I'm talking about changing the host of your database in the prefect the config to 127.0.0.1 explicitly instead of localhost so it is not replaced by postgres by the line I linked.
Do you think I should wait for PrefectHQ/prefect#3891 to be fixed first?
If that's an option. I do not know how soon it will be fixed though.
@madkinsz This did not work. I get an error : Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? When I use the same credentials in DBeaver, I am able to connect to the database which means database is accepting connections. So not able to understand why the connection request s not going fine with Prefect.