airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Mapped KubernetesPodOperater not rendering nested templates

Open bdsoha opened this issue 3 years ago • 3 comments

Apache Airflow version

2.3.3

What happened

Nested values, such as env_vars for the KubernetesPodOperater are not being rendered when used as a dynamically mapped operator.

Assuming the following:

op = KubernetesPodOperater.partial(
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
    # Other arguments
).expand(arguments=[[1], [2]])

The Rendered Template results for env_vars should be:

("[{'name': 'AWS_ACCESS_KEY_ID', 'value': 'some-super-secret-value', 'value_from': None}]")

Instead the actual Rendered Template results for env_vars are un-rendered:

("[{'name': 'AWS_ACCESS_KEY_ID', 'value': '{{ var.value.aws_access_key_id }}', 'value_from': None}]")

This is probably caused by the fact that MappedOperator is not calling KubernetesPodOperater._render_nested_template_fields.

What you think should happen instead

No response

How to reproduce

No response

Operating System

Ubuntu 18.04

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

bdsoha avatar Aug 08 '22 06:08 bdsoha

MappedOperator inherits AbstractOperator, which calls _render_nested_template_fields via render_template. So the analysis seems off.

uranusjr avatar Aug 08 '22 06:08 uranusjr

Does this work?

KubernetesPodOperater(
    task_id=...,
    ...,
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
)

uranusjr avatar Aug 08 '22 06:08 uranusjr

Does this work?

KubernetesPodOperater(
    task_id=...,
    ...,
    env_vars=[k8s.V1EnvVar(name='AWS_ACCESS_KEY_ID', value='{{ var.value.aws_access_key_id }}')],
)

Yes, as mentioned above the issue is only when using the operator as a mapped-operator.

bdsoha avatar Aug 08 '22 06:08 bdsoha