Guy Freeman comments

Results 5 comments of


                                            Guy Freeman

URLs in start_urls are not affected by this middleware

This is now documented under NOTE 1: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud

[ELE-33] [Feature] Support Clickhouse

Hi @Maayan-s is there a guide for integrating Elementary with community dbt adapters such as Clickhouse's? I'd be happy to try to contribute if so.

Incorrect S3 url when using Digital Ocean Spaces as staging and Clickhouse as destination

Two things are required: 1) use_https = True 2) to set bucket_url _without_ a trailing slash, i.e. `s3://bucket_name` and not `s3://bucket_name/`

feat(duckdb): add microbatch support for incremental strategies

Oh wow, thanks for the tip! Will join the Slack to deduplicate efforts :)

feat(duckdb): add microbatch support for incremental strategies

I updated how timestamps in the fixtures are defined to overcome 'timestamp is not UTC' errors