sqlparse icon indicating copy to clipboard operation
sqlparse copied to clipboard

added some builtin types that can appear in Spark

Open mrmasterplan opened this issue 4 years ago • 0 comments

Spark table schema has some compond builtin types that need keywords so that they can be tokenized correctly, You can see them, here: https://spark.apache.org/docs/latest/sql-ref-datatypes.html I also found a reference to Hive SQL data types here http://hortonworks.com/wp-content/uploads/2016/05/Hortonworks.CheatSheet.SQLtoHive.pdf

Here is an example of the kind of statement that I need to parse.

CREATE TABLE IF NOT EXISTS my_db1.tbl1(
a int,
b int,
c string,
cplx struct<
    someId:string,
    QrCode:string,
    details:struct<id:string>,
    blabla : array< int >
    >,
d timestamp,
m map<int,string>
)
USING DELTA
COMMENT "Dummy Database 1 table 1"
LOCATION "/tmp/foo/bar/my_db1/tbl1/"

without the changes in this PR, tokenizing will fail.

mrmasterplan avatar Sep 22 '21 06:09 mrmasterplan