blueshift
blueshift copied to clipboard
uswitch
Automate copying data from S3 into Amazon Redshift
If I have multiple unimported csv files, and each has one change of same record, that record is written multiple times in database. Is there a way to tell blueshift...
Wanted to get your thoughts. In my use case, I actually need two usage modes; one is like yours, which I affectionately term _Upsert/Clobber_ mode and the other I term...
We have alot of small s3 files and the delete operation ins blueshift is the biggest bottleneck right now. Any suggestions since i dont know implications of using something like...
The JDBC URL that appears in Redshift's dashboard starts with jdbc:redshift://... . However, Blueshift expects jdbc:postgresql://... in the manifest file. When using the first one, Blueshift raises this error when...
I had this software working a few weeks ago, but now it does not work. The S3 bucket scanner finds my two directories with 'manifest.edn' files, but does not add...
so over the weekend s3 eventual consistency on list operations because really slow and it seems the s3 list index was retuning files that were actually deleted, therefore i was...
Great project. One concern about passwords. According to the design.md documentation, the manifest.edn file containing the JDBC connection username and password sits in S3. Shouldn't this info be better placed...
Ensure blueshift handles manifests that fail validation errors. Should log the error and try again after the usual delay. ``` clojure.lang.ExceptionInfo: Invalid manifest. Check map for more details. at clojure.core$ex_info.invoke(core.clj:4403)...
When an import fails query the stl_load_errors table for the errors during that transaction, store them in a file on S3 so its easier to see when/why imports don't process-...