airflow icon indicating copy to clipboard operation
airflow copied to clipboard

feat: implement amazon s3 to dynamodb transfer operator

Open dondaum opened this issue 1 year ago • 0 comments

closes: #39377

Implements a new transfer operator for loading Amazon S3 data into an Amazon DynamoDB table. The operator uses the existing ImportTable AWS Service. Yet, import into existing tables is not currently supported by ImportTable. Until this is supported the operator uses an combination of staging first into a temporary DynamoDB table and then use scan to load data into the target DynamoDB table.

All related unit tests, CI, and build-docs inlcuding system test pass locally.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

dondaum avatar May 15 '24 21:05 dondaum