sqlx icon indicating copy to clipboard operation
sqlx copied to clipboard

[Next] [Feature] Generalized query placeholders (including support for expanding arrays into multiple placeholders)

Open mehcode opened this issue 5 years ago • 13 comments

@abonander and I have been discussing the benefits of providing a more Python style of SQL placeholders for parameters

In Python database connectors, it looks like this:

conn.execute("SELECT %(name)s", name="Hello")

The connector takes care of:

  • Using ?, $1, or @p1 depending on the database driver
  • Expanding arrays into ?, ?, ?, ...
  • Allowing named parameters where none are supported such as PostgreSQL

The really interesting part is Intellij understands that %(name)s is a SQL parameter and correctly parses that query and still provides proper code completion ( as a side note, Intellij Rust recently gained support for SQL highlighting and code completion inside of sqlx::query! ).


We are proposing that we add support for the following ( both in query! and query ):

Positional Parameter

// "SELECT ?" in MySQL
// "SELECT $1" in PostgreSQL

sqlx::query!("SELECT {0}", 10);

Named Parameter

// "SELECT @id" in MSSQL
// "SELECT $id" in SQLite
// "SELECT $1" in PostgreSQL

sqlx::query!("SELECT {id}", id=10);

Inline Parameter (only macros)

// "SELECT @id" in MSSQL
// "SELECT $id" in SQLite
// "SELECT $1" in PostgreSQL

let id = 10;
sqlx::query!("SELECT {id}");

Array Parameter

// MySQL = "SELECT * FROM foo WHERE id IN (?, ?, ?, ...)"
// PostgreSQL = "SELECT * FROM foo WHERE id IN ($1, $2, $3, ...)"

sqlx::query!("SELECT * FROM foo WHERE id IN ({ids+})", ids=[1, 2, 3]);
  • {ids+} or {ids*} becomes a comma-delimited list of parameters
  • {ids+} for a 0 length array becomes NULL
  • {ids*} for a 0 length array becomes `` (nothing)

I was on the fence about doing something like this but we can teach Intellij to recognize {...} as a SQL parameter which would let Intellij continue to parse and provide completion in our SQL queries.

Thoughts?

mehcode avatar Dec 04 '20 05:12 mehcode

Big thumbs up from me :+1:

If the inline form could also permit {foo.bar}, that would be amazing!

jplatte avatar Dec 04 '20 12:12 jplatte

If the inline form could also permit {foo.bar}, that would be amazing!

We should probably follow the syntax recommendation here if we want to allow expressions: https://rust-lang.github.io/rfcs/2795-format-args-implicit-identifiers.html#alternative-solution---interpolation

let foo = Foo { bar: 50 };

// SELECT $1
sqlx::query!("SELECT {(foo.bar)}");

This also lets our Kleene operator work with these complex expressions:

let foo = Foo { bar: [1, 2, 3] };

// SELECT $1, $2, $3
sqlx::query!("SELECT {(foo.bar)*}");

mehcode avatar Dec 04 '20 12:12 mehcode

I think that 90% of my uses of this are going to be {(foo.bar)} or {(foo.bar.baz)}. But since it's backwards-compatible to allow this without the extra parentheses later on, it seems fine to start without that.

jplatte avatar Dec 04 '20 12:12 jplatte

The IN ({ids*}) syntax would be a god send for Sqlite in particular.

I wonder if it would be possible to add a minimal extensions of this syntax for INSERT as well?

I'd like to insert all the fields of a struct succinctly - as a nice ergonomic for developers who want to prototyping quickly and leave choosing specific fields as a later optimization.

Perhaps if there was a way to generate 'static str or const column names via a macro they could be expanded into the query at compile time?

#[derive(sqlx::Columns)]
struct Foo { a: i32, b: i32, c: i32 }
let foo = Foo { a: 1, b: 2, c: 3 };

let values = foo.column_values();
// INSERT INTO foos (a, b, c) VALUES (?, ? ,?)
sqlx::query!("INSERT INTO foos ({Foo::COLUMNS*}) VALUES (values*)");

D1plo1d avatar Feb 08 '21 16:02 D1plo1d

@D1plo1d I would probably prefer something a little less implicit like the following: https://github.com/launchbadge/sqlx/issues/591

That would work okay with this proposal, although if you use the ...foo syntax we'll have to disable the ability to implicitly bind variables from the scope for that query (since we can't know whether to generate code to get a given bind parameter from foo or from the surrounding scope).

However, we can't implicitly expand the column list from the fields of foo while maintaining the guarantees of query!() as there's no way to get the fields of Foo in a proc macro without some hacky side-channel communication between #[derive(sqlx::Columns)] and sqlx::query!() (which I realize is the pot calling the kettle black in this situation but I'd prefer to limit the hackiness where possible).

abonander avatar May 03 '21 21:05 abonander

Is there any update to when this might be implemented, or alternatively what blockers currently exist for implementation?

I have a use-case of passing a vec of arguments for a WHERE x IN (?) clause with SQLite and cannot use the listed postgres workaround due to the database type, this feature would be incredibly helpful towards that goal.

cbondurant avatar Dec 12 '22 19:12 cbondurant

I would love this as I'm currently writing a lot of code to manually trick indices for postgres. Having this done by name instead would be amazing

blagowtf avatar Jul 13 '23 14:07 blagowtf

Not being able to use WHERE ... IN (?) in SQLite makes it really hard to batch WHERE operations (for example in a data loader). With Postgres it works fine because of ANY($1).

marvin-j97 avatar Nov 28 '23 16:11 marvin-j97

Would funding this issue improve time to completion?

andrewgazelka avatar May 22 '25 12:05 andrewgazelka

Hi everyone, I've made a macro in SeaQuery - took some ideas from this discussion thread.

Would love your feedback, here is a write-up!

https://www.sea-ql.org/blog/2025-08-15-sea-query-raw-sql/

let a = 1;
struct B { b: i32 }
let b = B { b: 2 };
let c = "A";
let d = vec![3, 4, 5];

let query = sea_query::raw_query!(
    PostgresQueryBuilder,
    r#"SELECT ("size_w" + {a}) * {b.b} FROM "glyph"
       WHERE "image" LIKE {c} AND "id" IN ({..d})"#
);

assert_eq!(
    query.sql,
    r#"SELECT ("size_w" + $1) * $2 FROM "glyph"
       WHERE "image" LIKE $3 AND "id" IN ($4, $5, $6)"#
);
assert_eq!(
    query.values,
    Values(vec![1.into(), 2.into(), "A".into(), 3.into(), 4.into(), 5.into()])
);

tyt2y3 avatar Aug 14 '25 21:08 tyt2y3

@tyt2y3 if the proc_macro tightly bound to the sea-query internals or would be feasible to lift the mapping logic into the sqlx macros?

lu-zero avatar Aug 25 '25 07:08 lu-zero

@tyt2y3 if the proc_macro tightly bound to the sea-query internals or would be feasible to lift the mapping logic into the sqlx macros?

the macro itself is not tightly coupled with sea-query, but the test cases are. so it requires some effort to port and maintain it separately.

tyt2y3 avatar Aug 25 '25 09:08 tyt2y3

Having many positional parameters in a query with no way to name them is not only unsafe, it is dangerous.

(So why haven't the Postgres authors implemented this in the latest version of the wire protocol to save hundreds of client libraries from re-inventing the wheel?)

Granted: sqlx is one of the few libraries in any language that supports compile-time query checking, and the difficulty of implemention of this feature cannot be overstated.

I don't know the primary drivers for this project, but have the authors of this library ever experienced the following (albeit oversimplified) kind of frustration?

sqlx::query!(r#"INSERT INTO people
(first_name, last_name, email1, email2, phone1, phone2, address1, address2, address3, city)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)"#,
person.first_name,
person.last_name,
person.address1, // OOPS! ALL Strings, NO COMPILE ERROR!
person.address2,
person.city,
...
)...

This feature request is 5 years old. Seems like a "lifetime" to me in this ecosystem.

What will it take? The community needs safe, named parameters! Is this something contributors can collaborate on? Granted, LLMs are still primitive, stupid, often wrong, and frustrating to work with, but I've successfully used them in a couple of forks of major crates (thiserror -> wherror, tracing_subscriber -> better_tracing) to implement seemingly difficult features that the authors could not or would not implement, and that were beyond my ability given time constraints to figure out.

It's not wrong to try throwing a couple of LLMs on the codebase and getting suggestions on how to implement something like this, if that's the hurdle.

Imagine how much time this feature would save in production codebases. It would certainly make up the development time many times over. SQL was invented in the 1980s, when parameter names would take up too much memory! It's almost 2026, we have the tools and the power to achieve this! And since format_args! can successfully take named parameters, why not the sqlx macros?

Can't sqlx::query! and friends just rewrite the names to numbers internally based on their order in the query or something?

tgrushka avatar Oct 29 '25 21:10 tgrushka