Support for complex types

[john-bodley]

I was wondering why PyHive doesn't support complex types like ARRAY, MAP, STRUCT, etc?

[jingw]

Not sure what you're requesting. Complex types work: https://github.com/dropbox/PyHive/blob/master/pyhive/tests/test_hive.py#L41

Are you asking for support for parsing the result strings into something nicer?

[john-bodley]

Apologies @jingw I meant for Presto. I was looking at sqlalchemy_presto.py and didn't notice any, but per the Presto equivalent link you provided it seems like they are mentioned in test_presto.py.

[josephfrancis]

@jingw We are also seeing this with hive, querying a struct in hive returns as a string. https://github.com/dropbox/PyHive/blob/master/pyhive/sqlalchemy_hive.py#L98 Was trying to use it through superset's sqllab and found that we cant query fields inside a struct.

[josephfrancis]

@jingw I'm interested in getting this working at the very least for presto. I'm happy to make a PR if you have any suggestions on how best to represent the complex types in the current response format.

[jingw]

If you meant Presto, that currently returns a struct as a list: https://github.com/dropbox/PyHive/blob/master/pyhive/tests/test_presto.py#L76

If you meant Hive, that returns strings: https://github.com/dropbox/PyHive/blob/master/pyhive/tests/test_hive.py#L65 We could add a flag parse_complex_types to the connection object (default false for compatibility) and have it return something similar to Presto. Sadly the return format isn't JSON compatible, but it looks like ast.literal_eval could be good enough. Need to check on what happens with fancier types like unicode characters and binary.

I'm not sure what you meant by linking to sqlalchemy_presto.py#L137

[kakipipi23]

There is a PR pending, waiting for over a week. Much frustrating.

[panoptikum]

This is quite a limitation for us. Any chance that hive returns a struct as list or similar?