User does not receive an email with a magic link to sign in

[qa-wolf[bot]]

Steps to reproduce:

1. Log in as a Global Admin
2. Create a user with the "Enable two-factor authentication (email)" checkbox enabled
3. As the user, log in (fill in email and password fields)
4. Wait for magic link email to come through Expected: Magic link email should bc received by the user Actual: No magic link email is received by the user Video: https://www.loom.com/share/367889b2edb14098bbb8748b9c088c49 (https://www.loom.com/share/367889b2edb14098bbb8748b9c088c49)

[jmwatts]

I tried this workflow and it was successful for me which leads me to believe we need to check the SMTP setup for QA Wolf's environment. I am not able to see how SMTP is configured for QA Wolf's environment, all I see is this screen:

[sharon-fdm]

Thanks, @jmwatts. @xpkoala, can we check with QA-Wolf if they havb the proper configuration set?

[jmwatts]

fleetctl get config --include-server-config returns this:

  smtp_settings:
    authentication_method: authmethod_plain
    authentication_type: authtype_username_password
    configured: false
    domain: ""
    enable_smtp: false
    enable_ssl_tls: true
    enable_start_tls: true
    password: ""
    port: 587
    sender_address: ""
    server: ""
    user_name: ""
    verify_ssl_certs: true

I don't know if/how we've set up SMTP for them in the past but I'm guessing it's on us. @rfairburn or @georgekarrv do you know?

[rfairburn]

QAWolf's SMTP is configured with Amazon SES. From the UI you should not see or be able to configure email settings as a result (the is already configured message is expected).

[jmwatts]

@rfairburn Thanks! They are not receiving emails from the fleet server, how can we confirm that SMTP is set up correctly for them?

[rfairburn]

I just verified that I received am email testing the environment (I used the Invite to my fleet email). It did go to spam though.

I have verified that we create dmac,dkim, and spf records as part of the SES configuration, so everything is in place here. This is likely a content-based filter.

My recommendation is to check spam/filter folders to see if the message was blocked that way.

[jmwatts]

Added a note on #28000 as that change was made the day prior and the test began failing once that change would have been deployed to QA Wolf's environment.

[jmwatts]

@sharon-fdm and @xpkoala I am able to reproduce this using QA Wolf's instance, and in Dogfood. The email is sent to my SPAM folder when it wasn't previously.

Something has changed somewhere, uncertain of what, but it should probably be investigated otherwise customers will likely begin reporting this too.

[sharon-fdm]

@sharon-fdm and @xpkoala I am able to reproduce this using QA Wolf's instance, and in Dogfood. The email is sent to my SPAM folder when it wasn't previously.

Something has changed somewhere, uncertain of what, but it should probably be investigated otherwise customers will likely begin reporting this too.

Thanks @jmwatts. @noahtalerman, this is on product board.

[noahtalerman]

@lukeheath I think this bug qualifies as a P2. Verification emails are going to spam. I think customers will have a hard time finding them.

cc @sharon-fdm @zayhanlon

[zayhanlon]

@rfairburn related to the fix you made for customer-numa on this 2fa email thing?

[rfairburn]

Dogfood is running an older version of the SES module and can be updated, but QAWolf is running the latest version and there is nothing from the infra side that can improve this for them.

As far as I can tell, ending up in spam appears to be a content-related filter as DMARC,DKIM, and SPF rules all pass in their environment. https://github.com/fleetdm/fleet/pull/28289 will bring Dogfood to the same status. Here is what an invite to Fleet looks like when doing "Show Original" for example:

[lukeheath]

Wow, this is a cool bug for QA Wolf to report outside of our UI!

@noahtalerman Agreed we should prioritize as a P2, but I don't think it's a Fleet product bug. Spam filtering depends on the email server that is connected to Fleet, so that is an infrastructure management issue. It is strange that it was not being filtered and then suddenly started.

@rfairburn I just approved your PR. Once that's online please let @jmwatts know so she can try dogfood again and hopefully it will not get caught in spam filters. If that fixes it, please make sure all managed cloud instances are upgraded as well and we can share that PR with QA Wolf engineers to see how we fixed it.

If that doesn't fix it, we'll have to dig in further. Since this was reported by @jmwatts I'm assigning to the software team so she can assist with QA.

[rfairburn]

@lukeheath a few things:

• all managed cloud instances including QA Wolf have had this change in place already for a few weeks. The passing screenshot from above was from the premium QA Wolf instance.
• any spam filtering that is happening right now (outside of possibly dogfood) would be content or usage pattern (spam reports) related and wouldn't have an infra solution
• I don't believe this is new or "just" started happening. @allenhouchins informed me previously that he tells all prospects to check their spam folder for the invite to Fleet emails, for example.

[rfairburn]

@jmwatts the subdomain-specific dmarc records are created for dogfood and now match the rest of managed cloud (including qa wolf). These should supersede the global fleetdm.com records and provide the best experience possible for spam-free emails from Fleet. If messages are still going to spam, that is outside of my direct control.

[jmwatts]

I tested on dogfood and both the reset password and 2FA emails are still being sent to spam in my Gmail inbox. I was able to specifically add a filter to Gmail to never send messages from the do-not-reply@dogfood email to spam and that did correct the issue for my Gmail inbox.

I tried sending a 2FA email and a reset password email to a non-gmail email address and only the reset password email showed up after several minutes. The 2FA email was never delivered, not even in the spam folder. I was using my student email, and I'm guessing their built-in content filter blocked the email. I did set the do-not-reply@dogfood email as a trusted sender, but still nothing.

@lukeheath @noahtalerman I'm not sure what else to do for testing at this point, aside from maybe creating email accounts with different email providers and documenting behavior, however this should probably go back to orchestration as it's not software related. I didn't actually report this one, it was reported by QA Wolf. I just attempted to assist them when they asked questions.

[mostlikelee]

Since this issue has been downgraded from a P1 -> P2, it should go to the g-orchestration product board and get pulled in next sprint, right?

[lukeheath]

@jmwatts Thanks for the thorough testing!

@mostlikelee Yep, since it needs engineering time I'm moving over to orchestration.

@sharon-fdm Would you please deprioritize a lower priority issue to make room for this in the current sprint? Thank you!

[sharon-fdm]

3 points roughly estimated by @dantecatalfamo. There are some unknowns so the estimation could be wrong.

[dantecatalfamo]

After a discussion with @rfairburn, I'm going to add a To: field to the email headers. There's not much we can do to influence the behaviour of the spam filters on the receiving end, but it should help.

[fleet-release]

Magic link in spam, A cloud city's lost mail, Found, security's calm.

[fleet-release]

Magic link sent astray, In the cloud city, ensure, Safe passage each day.

[fleet-release]

Email link in spam, Security now seamless, Fleet users find calm.