sqlparse
sqlparse copied to clipboard
added some builtin types that can appear in Spark
Spark table schema has some compond builtin types that need keywords so that they can be tokenized correctly, You can see them, here: https://spark.apache.org/docs/latest/sql-ref-datatypes.html I also found a reference to Hive SQL data types here http://hortonworks.com/wp-content/uploads/2016/05/Hortonworks.CheatSheet.SQLtoHive.pdf
Here is an example of the kind of statement that I need to parse.
CREATE TABLE IF NOT EXISTS my_db1.tbl1(
a int,
b int,
c string,
cplx struct<
someId:string,
QrCode:string,
details:struct<id:string>,
blabla : array< int >
>,
d timestamp,
m map<int,string>
)
USING DELTA
COMMENT "Dummy Database 1 table 1"
LOCATION "/tmp/foo/bar/my_db1/tbl1/"
without the changes in this PR, tokenizing will fail.