smtp_default connection host/port is ignored when sending email

[rogalski]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.5.1

What happened?

Setup: AWS MWAA + smtp_default connection in form of

smtp://{aws_creds_username}:{aws_creds_password}@email-smtp.{region}.amazonaws.com:587?smtp_ssl=false&smtp_starttls=true&...

Trivial EmailOperator:

        notify = EmailOperator(
            task_id="send_notify",
            to=["[email protected]"],
            subject="Some subject",
            html_content="""Some content""",
        )

From EmailOperator logs:

[2024-02-15, 10:40:01 UTC] {{base.py:73}} INFO - Using connection ID 'smtp_default' for task execution.
[2024-02-15, 10:40:01 UTC] {{email.py:268}} INFO - Email alerting: attempt 1
[2024-02-15, 10:40:01 UTC] {{taskinstance.py:1768}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/operators/email.py", line 77, in execute
    send_email(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py", line 78, in send_email
    return backend(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py", line 152, in send_email_smtp
    send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py", line 270, in send_mime_email
    smtp_conn = _get_smtp_connection(smtp_host, smtp_port, smtp_timeout, smtp_ssl)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py", line 317, in _get_smtp_connection
    else smtplib.SMTP(host=host, port=port, timeout=timeout)
  File "/usr/lib/python3.10/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib/python3.10/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python3.10/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

What you think should happen instead?

smtp_host, smtp_port, smtp_starttls, smtp_ssl, smtp_retry_limit, smtp_timeout from connection should be honored

How to reproduce

https://github.com/apache/airflow/blob/2.5.1/airflow/utils/email.py#L244-L252

    if conn_id is not None:
        try:
            from airflow.hooks.base import BaseHook

            airflow_conn = BaseHook.get_connection(conn_id)
            smtp_user = airflow_conn.login
            smtp_password = airflow_conn.password
        except AirflowException:
            pass

Please note that alternative code with same connection string works just fine:

airflow_conn = BaseHook.get_connection("smtp_default")
print(airflow_conn.debug_info())

with smtplib.SMTP(airflow_conn.host, port=airflow_conn.port) as s:
    s.starttls()
    s.login(airflow_conn.login, airflow_conn.password)

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

So far that has been deliberate choice - see https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html. SMTP for the emails that are sent by Airflow have - traditinally been sent using configuration done by Deployment Manager in Airflow configuraiton and it's been a deliberate decision to spearate "platform" configuration from "DAG" configuration. Using user/password from Connection table has been added later as a way to make security credentials kept using FERNET-encrypted DB entry mechanism we already have, but there was no intention to use all the configuration there - but if you define a way how to handle backwards compatibility, and explain it in the docs including how to change existing confifuration to the new one and also make it in the way to avoid even more confusion, then i think it's quite ok to have it.

[Taragolis]

Just wondering is it issue about EmailOperator from core or from SFTP Provider?

[potiuk]

Ah. Maybe I misunderstood :)

[Taragolis]

Maybe you rights, attached link to core modules.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions[bot]]

This issue has been closed because it has not received response from the issue author.

[bmoon4]

Hi, I'm experiencing same issue. Airflow is not picking up variables from a connection (smtp_default). It is just looking for env vars such as AIRFLOW__SMTP__xxxx or conf vars..