sqlparse icon indicating copy to clipboard operation
sqlparse copied to clipboard

Allow customizing lexer instance in FilterStack

Open illes opened this issue 9 months ago • 0 comments

Is your feature request related to a problem? Please describe.

The top-level entry points, notably sqlparse.parse(sql), use the singleton Lexer instance via FilterStack. Users that extend sqlparse (e.g. to parse SQL with a custom Lexer instance/subclass depending on the SQL's dialect) can't re-use top level functions/FilterStack, because they always use the singleton Lexer instance.

Describe the solution you'd like

I propose to make a subtle change that only affects the internals of FilterStack, adding a self.lexer = Lexer.get_default_instance() that users can override.

Describe alternatives you've considered #717 attempts to solve this problem by adding a lexer parameter to top-level methods. My proposed change would not require any updates to public APIs/docs, like #715 does.

Additional context

Current usage with singleton lexer:

import sqlparse


sql = "SELECT id::!int FROM t"

statements = sqlparse.parse(sql)

Note that parse(sql) boils down to:

def parse(sql):
     stack = engine.FilterStack()
     stack.enable_grouping()
     return tuple(stack.run(sql))  # uses Lexer.get_default_instance() via lexer.tokens(...)

New alternative extended usage (instead of unchanged top-level parse(sql)):

my_stack = engine.FilterStack()
my_stack.lexer = MyLexer()  # extends sqlparse.lexer.Lexer
my_stack.enable_grouping()
statements = tuple(my_stack.run(sql))

illes avatar Apr 23 '25 16:04 illes