Status of testing Providers that were prepared on May 12, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.22.0rc1

• [ ] S3DeleteObjectsOperator Added ability to filter keys by last modified time (#39151): @idantepper
• [ ] Amazon Bedrock - Knowledge Bases and Data Sources (#39245): @ferruzzi
• [ ] Scheduler to handle incrementing of try_number (#39336): @dstandish
• [ ] ECS Executor: Set tasks to RUNNING state once active (#39212): @o-nikolas
• [ ] enh(amazon_hook): in file download, change type of raised exception due to 404 (#39509): @keygrocery

Provider apache.iceberg: 1.0.0rc1

• [ ] Add Iceberg provider (#39155): @romsharon98

Provider google: 10.18.0rc2

• [ ] Add templated fields for Google Cloud Generative AI Operators (#39174): @CYarros10
• [ ] Add logic to handle on_kill for BigQueryInsertJobOperator when deferrable=True (#38912): @sunank200
• [ ] Create CloudComposerRunAirflowCLICommandOperator operator (#38965): @MaksYermak
• [x] Add the deferrable mode to the Dataflow sensors (#37693): @e-galan
• [ ] Deferrable mode for Custom Training Job operators (#38584): @e-galan
• [x] Enhancement for SSL-support in CloudSQLExecuteQueryOperator (#38894): @moiseenkov
• [ ] Create GKESuspendJobOperator and GKEResumeJobOperator operators (#38677): @MaksYermak
• [x] Add support for role arn for aws creds in storage transfer job operators (#38911): @wlinamchurch
• [x] Add encryption_configuration parameter to BigQueryCheckOperator and BigQueryTableCheckOperator (#39432): @molcay
• [ ] Add job_id parameter to BigQueryGetDataOperator (#39315): @shahar1 Linked issues:
  • [ ] Linked Issue #39127: @lopezvit
• [x] Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator (#39018): @e-galan
• [x] Fix batching for BigQueryToPostgresOperator (#39233): @moiseenkov
• [ ] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. (#39230): @sunank200
• [x] Fix GCSObjectExistenceSensor operator to return the same XCOM value in deferrable and non-deferrable mode (#39206): @VladaZakharova
• [x] Bugfix BigQueryToMsSqlOperator (#39171): @moiseenkov
• [x] fix: add retry logic in case of google auth refresh credential error (#38961): @dondaum Linked issues:
  • [x] Linked Issue #38532: @nathadfield
• [ ] fix: BigQueryCheckOperator skip value and error check in deferrable mode (#38408): @kacpermuda Linked issues:
  • [ ] Linked Issue #37885: @kacpermuda
• [ ] [OpenLineage] Fix datasets in GCSDeleteObjectsOperator (#39059): @kacpermuda Linked issues:
  • [ ] Linked Issue #35838: @kacpermuda
• [ ] [OpenLineage] Use prefixes instead of file paths for datasets in GCSToGCSOperator (#39058): @kacpermuda Linked issues:
  • [ ] Linked Issue #31350: @mobuchowski
• [ ] [OpenLineage] Fix datasets in GCSTimeSpanFileTransformOperator (#39064): @kacpermuda Linked issues:
  • [ ] Linked Issue #35838: @kacpermuda
• [ ] Fix generation temp filename in DataprocSubmitPySparkJobOperator (#39498): @Taragolis Linked issues:
  • [ ] Linked Issue #39478: @RubenBBlazquez
• [ ] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocSubmitJobOperator (#39447): @sunank200 Linked issues:
  • [ ] Linked Issue #39230: @sunank200
• [ ] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for BigQueryInsertJobOperator (#39442): @sunank200 Linked issues:
  • [ ] Linked Issue #38912: @sunank200
• [ ] Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator (#39446): @sunan Linked issues:
  • [ ] Linked Issue #39130: @sunank200
• [ ] Fix DataprocCreateBatchOperator with result_retry raises AttributeError (#39462): @tatiana Linked issues:
  • [ ] Linked Issue #39394: @tatiana
• [x] Bugfix yaml parsing for GKEStartKueueInsideClusterOperator (#39234): @moiseenkov
• [ ] Fix validation for BQ label values for BigQueryInsertJobOperator (#39568): @kisssam Linked issues:
  • [ ] Linked Issue #39567: @kisssam
• [ ] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup (#39130): @sunank200
• [x] Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
• [ ] Handle ImportError for KubernetesDeleteJobOperator in google provider (#39036): @vatsrahul1001 Linked issues:
  • [ ] Linked Issue #37793: @MaksYermak
• [x] Deprecation of AutoML services: Add deprecation warnings and raise exceptions for already deprecated ones (#38673): @molcay Linked issues:
  • [x] Linked Issue #38633: @shahar1
• [x] Disconnect GKE operators from deprecated hooks (#39434): @moiseenkov

Provider microsoft.azure: 10.1.0rc2

• [x] add dns_config and diagnostics parameters to AzureContainerInstancesOperator (#39156): @Kylea650 Linked issues:
  • [x] Linked Issue #39149: @Kylea650
• [ ] Add stacklevel into the AzureSynapsePipelineHook deprecation warnings (#39192): @Taragolis
• [x] Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
• [ ] Pass through more parameters to AzureBlobFileSystem to mirror handlin… (#38747): @tomrutter Linked issues:
  • [ ] Linked Issue #38746: @tomrutter
• [x] Implement run-method on KiotaRequestAdapterHook and move logic away from triggerer to hook (#39237): @dabla
• [x] Implemented MSGraphSensor as a deferrable sensor (#39304): @dabla
• [x] Fix: Only quote the keys of the query_parameters in MSGraphOperator (#39207): @dabla

Provider pinecone: 2.0.0rc2

• [ ] Pinecone provider support for pinecone-client>=3 (#37307): @rawwar Linked issues:
  • [ ] Linked Issue #37158: @Taragolis
• [ ] Fix the pinecone system test (#39365): @sunank200

Provider tabular: 1.5.1rc1

• [ ] Add Iceberg provider (#39155): @romsharon98

All users involved in the PRs: @kacpermuda @e-galan @moiseenkov @Taragolis @keygrocery @o-nikolas @romsharon98 @dondaum @rawwar @dstandish @dabla @MaksYermak @CYarros10 @Kylea650 @wlinamchurch @tatiana @idantepper @tomrutter @ferruzz

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Kylea650]

#39156 tested and working. #39466 added as a result of testing (small documentation change). This has been approved but hasn’t yet been merged in.

[moiseenkov]

Hi, #38894, #39233, #39171, #39234, #39434 work as expected

[dabla]

Hello, I tested following PR and they are all working a expected:

• 38111
• 39237
• 39304
• 39207

So for me everything related to MSGraphOperator is good.

[VladaZakharova]

Hi! https://github.com/apache/airflow/pull/39206 works as expected

[wlinamchurch]

https://github.com/apache/airflow/pull/38911 works as expected

[dondaum]

#38961 is good ✅

[e-galan]

#39018 and #37693 work as expected.

System tests for #38584 are failing. CreateCustomTrainingJobOperator has a problem with training new models that are based on parent models. It seems that there is a problem with parsing and validating the parent model's resource id. The problem occurs immediately after submitting a child model's training job, when we wait for gca resource creation. It happens both for sync and deferrable modes.

Addressed in #39640

[molcay]

#39432 and #38673 works as expected

[VladaZakharova]

Hi @eladkal ! Since https://github.com/apache/airflow/pull/38584 is not working as expected, we will prepare a fix for it ASAP. Can we please discuss if we can have RC3 for google-provider in this case with that fix?

[eladkal]

Hi @eladkal ! Since #38584 is not working as expected, we will prepare a fix for it ASAP. Can we please discuss if we can have RC3 for google-provider in this case with that fix?

We can but I'd prefer to release RC2 and have the fix released in next wave (plan to cut it during the weekend) The current RC2 is massive. I am not sure if it's right to hold it's release due to the reported issue. I think releasing it and applying a quick followup hot fix is a good balance. WDYT?

[VladaZakharova]

Hi @eladkal ! Since #38584 is not working as expected, we will prepare a fix for it ASAP. Can we please discuss if we can have RC3 for google-provider in this case with that fix?

We can but due I'd prefer to release RC2 and have the fix released in next wave (plan to cut it during the weekend) The current RC2 is massive. I am not sure if it's right to hold it's release due to the reported issue. I think releasing it and applying a quick followup hot fix is a good balance. WDYT?

Sounds good to me, thank you!

[e-galan]

Hi @VladaZakharova @eladkal !

The fix for #38584 is implemented in #39640 .

[shahar1]

#39315 works as expected (issue #39127 can be closed)

[eladkal]

Thank you everyone. Providers are released. pinecone provider is excluded and will have rc3

I invite everyone to help improve providers for the next release, a list of open issues can be found here.