spark-redshift

spark-redshift copied to clipboard

TableName assumes PUBLIC schema when no schema is set

6

https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/TableName.scala#L42 Shouldn't this follow the rules of Redshift and respect the search_path of the logged in user: http://docs.aws.amazon.com/redshift/latest/dg/r_search_path.html

emmanueljob

py4j.protocol.Py4JJavaError: An error occurred while calling o95.save

29

I am executing the following command using PySpark ``` df.write.format('com.databricks.spark.redshift') \ .options(url='%s?rewriteBatchedStatements=true' % config.red_url, dbtable=config.sfeed_table_stage, user=config.red_user, password=config.red_passwd, tempdir=config.s3_tempdir, batchsize=50000).mode('append').save() ``` When I do this, it gives me the error message...

gffield

NullPointerException occurd while trying to read the S3 bucket lifecycle configuration

2

"com.databricks" %% "spark-redshift" % "3.0.0-preview1" , "com.amazonaws" % "aws-java-sdk" % "1.11.397" is used in my project, Line 140 of com.databricks.spark.redshift.Utils.scala `rule.getStatus == BucketLifecycleConfiguration.ENABLED && key.startsWith(rule.getPrefix)` throws NullPointerException because com.amazonaws.services.s3.model.BucketLifecycleConfiguration.Rule.getPrefix returns...

athletic-geek

WARN Utils: An error occurred while trying to read the S3 bucket lifecycle configuration java.lang.NullPointerException

18

Hello guys, I am getting this warn ``` WARN Utils$: An error occurred while trying to read the S3 bucket lifecycle configuration java.lang.NullPointerException at java.lang.String.startsWith(String.java:1385) at java.lang.String.startsWith(String.java:1414) at com.databricks.spark.redshift.Utils$$anonfun$3.apply(Utils.scala:102) at...

pedromb

Spark 3.3 support

1

We would like to use RedShift connector with Spark 3.3 and Scala 2.13. Could you share a road map to understand your plans for such support? Thanks!

o-shevchenko

Query information_schema not working (Exception thrown in awaitResult)

4

When i try to query the list of tables in a schema from a Redshift DB. I get following error. I have tried to use query and dbtable options with...

gfallerBB

I am getting below Exception while connecting to PostgreSQL. Please help to fix it. I have correct values for URL. user& password. java.sql.SQLException: [Amazon][JDBC](11380) Null pointer exception. at com.amazon.redshift.client.PGClient.startSession(Unknown Source)...

jayakumarim

Pyspark timestamps cast issue when reading MySQL DB & writing to Redshift

4

``` Python 2.7 Pyspark 2.2.1 JDBC format for MySQL->Spark DF For writing Spark DF-> AWS Redshift i am using the `Spark-Redshift` driver from Databricks. ``` I am reading data into...

foivosana

I want to use pyspark to write the result to hive table(table format is orc) by yarn-client, but show "py4j.protocol.Py4JJavaError: An error occurred while calling o96.save." error everytime, and if...

xfysq1

Fix executing empty sql statement in preactions and postactions

5

Say for example: you need to execute preactions/postactions `val preactions = "update table1 set col1=2; update table2 set col2 = 2; "` Looks nothing wrong, correct? Only when you try...

meetchandan