fleet icon indicating copy to clipboard operation
fleet copied to clipboard
verified publisher icon fleetdm

TLS handshake error from fleet.server.examp #6085

Open xastherion opened this issue 2 years ago • 1 comments 

          hi, i am confronting the same problems in this thread

SERVER centos stream 9 fleet version 4.38.1

CLIENTS macOS 13 Ventura + 12 Monterey

Certificate von Let´sEncrypt renewed with Dehydrated

Browsers: Firefox 115 ESR + Chrome 117

my client repeated this logs:

W1025 15:16:03.459451 1334582912 tls_enroll.cpp:101] Failed enrollment request to https://my-fleet-server.com:8080/api/v1/osquery/enroll (Request error: certificate verify failed) retrying...

and my Server this:

Oct 25 15:18:38 my-fleet-server fleet[1062]: 2023/10/25 15:18:38 http: TLS handshake error from 129.13.171.194:50805: local error: tls: bad record MAC

Out of all Logs, my fleet client run and is showed in fleet server site, but only the hostname and serialnumber, no more. For this short time the client shine online, after go Offline an no more sucedeed.

grafik

Last fetched almost 54 years ago (that is a lot of time!)

If i turn the client "add host" command with --insecure, all run right. But the logs in server are still present.

Originally posted by @xastherion in https://github.com/fleetdm/fleet/issues/6085#issuecomment-1779298292

Sure; all target to a SSL TLS Certificate Issue, but i try with curl commands with these results:

FROM A CLIENT (macos 14)

curl -v -X POST https://myserver.mydepartament.myorganisation.edu:8080/api/v1/osquery/enroll --cacert /Users/Shared/FLEET/fleet.pem

  • Trying 100.100.100.1:8080...
  • Connected to myserver.mydepartament.myorganisation.edu (100.100.100.1) port 8080 (#0)
  • ALPN: offers h2,http/1.1
  • (304) (OUT), TLS handshake, Client hello (1):
  • CAfile: /Users/Shared/FLEET/fleet.pem
  • CApath: none
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (IN), TLS handshake, Unknown (8):
  • (304) (IN), TLS handshake, Certificate (11):
  • (304) (IN), TLS handshake, CERT verify (15):
  • (304) (IN), TLS handshake, Finished (20):
  • (304) (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
  • ALPN: server accepted h2
  • Server certificate:
  • subject: CN=myserver.mydepartament.myorganisation.edu
  • start date: Oct 25 09:19:36 2023 GMT
  • expire date: Jan 23 09:19:35 2024 GMT
  • subjectAltName: host "myserver.mydepartament.myorganisation.edu" matched cert's "myserver.mydepartament.myorganisation.edu"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • using HTTP/2
  • h2 [:method: POST]
  • h2 [:scheme: https]
  • h2 [:authority: myserver.mydepartament.myorganisation.edu:8080]
  • h2 [:path: /api/v1/osquery/enroll]
  • h2 [user-agent: curl/8.1.2]
  • h2 [accept: /]
  • Using Stream ID: 1 (easy handle 0x7fde18813e00)

POST /api/v1/osquery/enroll HTTP/2 Host: myserver.mydepartament.myorganisation.edu:8080 User-Agent: curl/8.1.2 Accept: /

< HTTP/2 400 < content-type: text/plain; charset=utf-8 < content-length: 171 < date: Fri, 27 Oct 2023 10:00:47 GMT < { "message": "Bad request", "errors": [ { "name": "base", "reason": "Expected JSON Body" } ], "uuid": "5f0c9a4e-49e8-47c7-8ae7-db23111d8555" }

  • Connection #0 to host myserver.mydepartament.myorganisation.edu left intact

IN THE FLEET SERVER (Centos Stream 9):

curl -v -X POST https://myserver.mydepartament.myorganisation.edu:8080/api/v1/osquery/enroll --cacert /etc/pki/tls/certs/cert.pem

  • Trying 100.100.100.1:8080...
  • Connected to myserver.mydepartament.myorganisation.edu (100.100.100.1) port 8080 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • CAfile: /etc/pki/tls/certs/cert.pem
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS header, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS header, Finished (20):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.2 (OUT), TLS header, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS header, Unknown (23):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=myserver.mydepartament.myorganisation.edu
  • start date: Oct 25 09:19:36 2023 GMT
  • expire date: Jan 23 09:19:35 2024 GMT
  • subjectAltName: host "myserver.mydepartament.myorganisation.edu" matched cert's "myserver.mydepartament.myorganisation.edu"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • TLSv1.2 (OUT), TLS header, Unknown (23):
  • TLSv1.2 (OUT), TLS header, Unknown (23):
  • TLSv1.2 (OUT), TLS header, Unknown (23):
  • Using Stream ID: 1 (easy handle 0x55f99352fb80)
  • TLSv1.2 (OUT), TLS header, Unknown (23):

POST /api/v1/osquery/enroll HTTP/2 Host: myserver.mydepartament.myorganisation.edu:8080 user-agent: curl/7.76.1 accept: /

  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
  • TLSv1.2 (OUT), TLS header, Unknown (23):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.2 (IN), TLS header, Unknown (23): < HTTP/2 400 < content-type: text/plain; charset=utf-8 < content-length: 171 < date: Fri, 27 Oct 2023 09:59:02 GMT < { "message": "Bad request", "errors": [ { "name": "base", "reason": "Expected JSON Body" } ], "uuid": "2770ac89-065a-4414-88f1-70bf95416d11" }
  • Connection #0 to host myserver.mydepartament.myorganisation.edu left intact

Recomendations?

xastherion avatar Oct 27 '23 12:10 xastherion

Hi @xastherion!

A few questions:

  • Do you have access to the devices where fleetd is running? Can you try running curl -v -X POST https://myserver.mydepartament.myorganisation.edu:8080/api/v1/osquery/enroll --cacert /opt/orbit/certs.pem on the macOS devices where fleetd is installed?
  • What version of fleetctl was used to generate the package?
  • Could you share what arguments you used to generate the package? (fleetctl package --type=pkg ...)

lucasmrod avatar Jul 01 '24 19:07 lucasmrod

Hi @xastherion!

We are adding documentation around these kinds of issues with certificates with intermediates here: https://github.com/fleetdm/fleet/pull/20166/files#diff-c4b70fc75882fe6bf1b38ca84d89435be72013a430ef5a4d7b448c11d5254a22R17

TL;DR: If the any of the intermediate certificates is not present in /opt/orbit/certs.pem, then the certificate presented by the Fleet server has to be the full chain.

Feel free to re-open if you have any further issues.

lucasmrod avatar Jul 08 '24 09:07 lucasmrod

TLS handshake troubles bloom, Secure connections now bloom, Fleet's trust is resumed.

fleet-release avatar Jul 08 '24 09:07 fleet-release

Hi Lucas, Thank you very much for add this on the documentation. Sure, that problem was the not presented full-chain certificate. We find a solution thank Herder over Stacks. Close the issue is then correct.

xastherion avatar Jul 08 '24 09:07 xastherion