catherinedevlin
connection hangs randomly
Hi,
At random times during my notebook session, a sql statement like %sql select count(*) from table will just hang even though the calculation should be fast. This hang cannot be interrupted by jupyter, so I always have to restart the kernel. I can confirm that my postgresql server is still running and I can run the same command when I log in through a client, so this should not be a server side problem.
Wondering if anyone else has seen this and if there's a workaround or solution.
@princesscrypto1337 does it still happen to you (I know this issue is pretty old)
Huge thanks and appreciation to the contributors of the ipython-sql project! Very grateful for the project!
Hi @idomic. This problem is occurring for me. As @princesscrypto1337 mentions, the problem is random. Usually occurs after a lot of interaction with the Jupyter notebook. Like the issue reported, I am also using %sql select count(*). It also occurs in %sql create or replace table.... The issue seems to happen if I run those statements a couple or few times. In my case, I am using DuckDB. At first, I was using JupySQL and I thought it was a DuckDB issue. However, after troubleshooting the issue, and eliminating %sql and using a direct con.execute() the issue was no longer persistent. Just for context, using con.execute() the select count(*) only takes around 40-50 seconds. Letting %sql select count(*) run, can run for hours until stopped. I uninstalled JupySQL and tried ipython-sql (version: 0.4.1) and the issue appeared again (once again random). When the progress bar from DuckDB is available, the progress bar usually indicates the hang occurs at different times (32%, 38%, 41%, 88%, 90%, etc.). When the progress bar is not available, I just get the * indicating the code cell is currently executing. In my situation, I can interrupt the kernel. After interrupting the kernel, I tried %reload_ext sql and the problem still persists. Restarting the kernel, I can usually use ipython-sql again for a %sql select * from table limit 5, but then the issue starts to happen again when I do %sql select count(*). For now, I have suspended use. Thanks for your interest!
Hi. After exhaustive trial and error, the DuckDB problem previously mentioned does not appear to be related to the ipython-sql or JupySQL extensions. The issue with the cell hanging in Jupyter occurs when the DuckDB configuration is SET enable_progress_bar=true;. There appears to be an issue already already opened in the DuckDB repository - 5407.
Sorry to confuse the issue reported by @princesscrypto1337 for PostgreSQL. However, if someone is encountering cell hangs in Jupyter with DuckDB, try disabling the progress bar until the issue is resolved. Thanks!
The main issue here is to reproduce the problem. However, I think it'd still be valuable to provide a workaround, perhaps a timeout? So that cells that take too long abort, and there isn't a need to restart the kernel.
Would this help? @princesscrypto1337
