plotly
Support for Google BigQuery
From https://community.plot.ly/t/support-for-google-bigquery/8894
~~This is a good first issue~~, since @google-cloud/bigquery provides all the needed functionality:
Have never contributed to js projects, but since the feature request is mine I am willing to help wherever I can. Looking at the codebase of falcon, it lookst like a new bigquery.js persistent datastore is needed. That means wrapping Bigquery node.js client libraries.
Is that right?
@dkapitan I can help you with this. I did the Athena connector and can walk you through the process. @n-riesco helped me a lot with setting this up. I would clone the project and look at the Contributing.md and New_connection.md as good starting points. Feel free to ask questions.
@dkapitan As @shannonlal says, the best place to start is https://github.com/plotly/falcon-sql-client/blob/master/NEW_CONNECTION.md
If you get stuck, at any point, just, give us a shout.
And to begin simple, I'd require that users have to set the dataset in the connection form (we can drop this requirement later).
@shannonlal and @n-riesco
Off topic, but to deepen my understanding: the Sequelize library is used to create the connections. I need to interface with MS SQL via ODBC as well, which would require integrating sequelize-odbc-mssql as a new dialect.
Is this correct?
@dkapitan
the Sequelize library is used to create the connections.
only some of the connectors use Sequelize
I need to interface with MS SQL via ODBC as well, which would require integrating sequelize-odbc-mssql as a new dialect.
Is this correct?
I'm not familiar with sequelize-odbc-mssql. The main difficulties I see:
- it requires an ODBC driver (see configuration of
dialectOptions.driver) - and it's only for windows.
If you want to experiment with sequelize-odbc-mssql, you'd need to edit these lines in Sql.js.
And I guess, you'll also have to add sequelize-odbc-mssql and @ratanakvlun/node-odbc to webpack.config.base.js.
@dkapitan I think the Plot.ly team was swamped the last couple of weeks fixing some bugs and getting the latest Falcon release working. I am looking at using Big Query at work soon so I could make sometime to help with you this. For Big Query I don't think it makes sense to use sequelize and I would look at using the https://www.npmjs.com/package/@google-cloud/bigquery project. For Athena, we already had the AWS SDK and I just wrote the Athena wrapper for this. If your still available you can ping me directly and I can give you a lot of advice on this. @n-riesco helped me a lot to get my first Connector up and running and I would be more then willing to help you.
@shannonlal thanks for your help. I am currently swamped myself, and in the process of implementing a new data science platform on Google Cloud. So I don't expect to have time in the coming months. Maybe over summer I could start this, when things are a bit less hectic at work.
@shannonlal has started to look into implementing this connector. Unfortunately, there are a few difficulties to overcome:
- webpack fails to bundle
@google-cloud/bigquery(see issues https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1821 and https://github.com/GoogleCloudPlatform/google-cloud-node/issues/531) - and more worryingly (judging by the error logs in https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1821),
@google-cloud/bigquerydepends on native modules that usenode-pre-gyp.
Any updates on this integration? What are the rough timelines, when we can expect google bigquery integration with plotly?
@sumedhsakdeo I am only about 10 hours away from getting this working just need to finish this off. I was having a problem with using the official npm for bigquery but @n-riesco helped me resolve it. I have the UI and stubbed code for backend just need to get everything completed and tested. Shouldn't be too long
