databricks-sql-python icon indicating copy to clipboard operation
databricks-sql-python copied to clipboard

Add example with conversion to pandas DataFrame

Open timvink opened this issue 3 years ago • 2 comments

The example in the README.md and the documentation all print each row individually.

In practice many users will try to convert the data to a pandas dataframe. Why not include that example in the getting started. We currently use:

import pandas as pd

cursor.execute(query) 
result = cursor.fetchall() 
df = pd.DataFrame(result, columns=[x[0] for x in cursor.description]) 

timvink avatar Jul 25 '22 09:07 timvink

Writing it in this issue since it's kinda related, would it make sense to have an actual option to get the result set as pandas dataframe?

I'm not seeing something important, perhaps, but this code is confusing: https://github.com/databricks/databricks-sql-python/blob/main/src/databricks/sql/client.py#L657 Why not have an option to return the actual dataframe, why is it converted back and forth?

drapadubok avatar Nov 10 '22 08:11 drapadubok

I guess the same would be helpful for writes. I was just wondering what the best way of writing a pandas table to Databricks would be.

Triamus avatar Feb 03 '23 17:02 Triamus