trino-python-client icon indicating copy to clipboard operation
trino-python-client copied to clipboard

Don't prepare statements unless explicitly specified

Open mewa opened this issue 4 years ago • 3 comments

This is linked to #148 but since it explores a different/wider aspect of this issue, I opted for creating another issue.

The root cause in #148 is that statements are prepared, which combined with a large number of values results in too large header sizes.

As I explored it, it got me thinking about whether we should be preparing each and every query before executing it. Especially since currently it seems everything is happening implicitly, without the knowledge of the user. Any thoughts?

mewa avatar Dec 29 '21 11:12 mewa

If I am reading the code correctly the prepared statements path is only used when params are provided to execute. If not plain SQL is sent as it is.

Are you observing different behaviour?

hashhar avatar Dec 29 '21 12:12 hashhar

I had to dig into the stack traces since we're not calling execute directly, but yes, the params are provided.

For context, we're using:

  1. pandas - which in turn calls sqlalchemy
  2. sqlalchemy - which relies on sqlalchemy-trino dialect to communicate with Trino
  3. finally, sqlalchemy-trino then calls the Trino client

I think the issue here is that the client is doing more than it's asked for. In my opinion, deciding which statement should be prepared (or not) is the end-user's responsibility.

mewa avatar Dec 29 '21 15:12 mewa

There is no way today (or before the PR which added prepared statements support) to be able to pass parameters to a query without using prepared statements.

If you want to avoid prepared statements you need to stop passing params by constructing the SQL as a string and then calling cursor.execute(sql) instead of cursor.execute(sql, params).

I'm not sure if there's a way to have both:

  1. Ability for client to handle params.
  2. Not use prepared statements when params are passed.

What solution are you suggesting?

hashhar avatar Dec 29 '21 17:12 hashhar