piperider icon indicating copy to clipboard operation
piperider copied to clipboard

CLI incorrectly parses schema containing "." character

Open Vfisa opened this issue 2 years ago • 3 comments

In my case I have tried with this schema: in.c-data-diff-prep resulted in this error: Screenshot 2023-05-18 at 11 22 32

Vfisa avatar May 19 '23 05:05 Vfisa

@Vfisa Thank you for filing the issue. We can reproduce it and will try to fix it.

for internal tracking: sc-31444

ctiml avatar May 23 '23 04:05 ctiml

This issue is a bit more complicated than we expect.

We use the SQLAlchemy library at the underlying level to connect to databases and data warehouses. In the Snowflake engine, we have tried various approaches but have been unable to retrieve table metadata from a schema that contains dots in its name. Below is an very simple example

from sqlalchemy import create_engine
import urllib.parse

account = '<account>'
user = '<user>'
password = '<password>'
database = '<database>'
schema = 'DBT.TEST'
# schema = '"DBT.TEST"'  
warehouse = '<warehouse>' 

connection_string = f'snowflake://{user}:{password}@{account}/{database}'
if schema:
    connection_string += f'?schema={urllib.parse.quote(schema)}'
if warehouse:
    connection_string += f'&warehouse={warehouse}'


engine = create_engine(connection_string)

query the columns in a table.

from sqlalchemy import Table, MetaData
Table("mytable", MetaData(), autoload_with=engine, schema=schema)

We will investigate the same issue in sqlalchemy+duckdb, to see if it is sqlalchemy issue or snowflake-sqlalchemy

popcornylu avatar Jun 12 '23 09:06 popcornylu

Related issue https://github.com/snowflakedb/snowflake-sqlalchemy/issues/448

popcornylu avatar Sep 21 '23 02:09 popcornylu