PayloadTooLargeError and graphql foreign key violation on server start

[gitanshu]

Description

I am seeing these errors when I run prefect server start on 0.13.8 on a particular machine. Link to error log gist

I was running an older version of prefect, either 0.13.4 or 0.13.6, ran a workflow over dask executor, then upgraded to 0.13.8 and restarted it. I've tried clearing all docker images and volumes, and ~/.prefect/flows, and I still see the same error.

Expected Behavior

No error.

Reproduction

I've tried reproducing this on 2 other machines but haven't been able to.

Environment

Ubuntu 18.04

This is my config.toml

backend = "server"

[server]
host = "http://172.18.1.3"

    [server.ui]
    apollo_url = "http://172.18.1.3:4200/graphql"

[logging]
# The logging level: NOTSET, DEBUG, INFO, WARNING, ERROR, or CRITICAL
level = "WARNING"

[context.secrets]
SLACK_WEBHOOK_URL = "<redacted>"

[cicdw]

Hi @gitanshu - the error payloads you are seeing are:

• PayloadTooLargeError: this is caused by sending too large a payload to the API, probably on the order of tens of megabytes if I had to guess
• Foreign key violation. insert or update on table "log" violates foreign key constraint "log_flow_run_id_fkey"': this is being caused by an API request that is attempting to write a log for a flow run ID that is not present in the database. All logs must be associated with a valid flow run ID.

Both of these errors are only triggered when you make a request to the API, so could you describe more where these requests might be coming from?