PyHive icon indicating copy to clipboard operation
PyHive copied to clipboard

Added SparkSQL support based on HiveDialect

Open tanhaoqin opened this issue 7 years ago • 10 comments

SparkSQL is made to be almost compatible with HiveQL. However, the show tables syntax for Spark SQL is slightly different. As such, a new SparkSQL dialect is created based on the original HiveDialect with get_table_names modified to accommodate the results returned from Spark SQL's 'SHOW TABLES'.

tanhaoqin avatar Feb 05 '18 02:02 tanhaoqin

This fixes https://github.com/dropbox/PyHive/issues/150 without breaking the original Hive engine.

tanhaoqin avatar Feb 05 '18 02:02 tanhaoqin

Any progress on this?

xqzhou avatar Feb 27 '18 05:02 xqzhou

+1

alexandrejuma avatar Mar 12 '18 16:03 alexandrejuma

+1 to get this PR merged :)

kentore82 avatar May 07 '18 06:05 kentore82

Do you thought also about changing get_view_names, get_table_names result based on isTemporary column ?

maver1ck avatar May 29 '18 14:05 maver1ck

bump

tooptoop4 avatar Sep 08 '18 02:09 tooptoop4

@tanhaoqin this is failing the tests for some reason, maybe rebasing would fix the issue?

mistercrunch avatar Sep 10 '18 15:09 mistercrunch

Changed some stuff so there's no longer any conflicts.

I am no longer working on the project that required this feature though. Didn't have time to write tests as I am not using any Hive/Presto on my side so the original Hive/Presto tests will fail anyway.

tanhaoqin avatar Sep 11 '18 01:09 tanhaoqin

Many tests done by sqlalchemy_test_case will fail due the lack of support of some types by spark (SPARK-21529).

gmcoringa avatar Oct 08 '18 17:10 gmcoringa

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Apr 16 '22 21:04 CLAassistant