icinga2 Communication was suddenly broken: Client TLS handshake failed [..] excessive message size

Describe the bug

Hello,

yesterday in the morning our secondary master was suddenly unable to communicate with the other master and satellites. The log contained just the following entries:

master02 - icinga2.log

[2021-12-16 03:28:29 +0100] warning/JsonRpcConnection: API client disconnected for identity 'master01'
[2021-12-16 03:28:29 +0100] warning/ApiListener: Removing API client for endpoint 'master01'. 0 API clients left.
[2021-12-16 03:28:32 +0100] critical/ApiListener: Cannot connect to host 'ip_master01' on port '5665': Connection refused
[2021-12-16 03:28:42 +0100] critical/ApiListener: Cannot connect to host 'ip_master01' on port '5665': Connection refused
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_a_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_a_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_b_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_c_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_d_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_d_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_e_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_f_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_g_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_g_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_b_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_f_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_h_sat01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master01]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_c_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_e_sat02]:5665): Operation canceled
[2021-12-16 03:31:20 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_h_sat02]:5665): Operation canceled
[2021-12-16 03:31:25 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_a_sat01]:5665): Connection reset by peer
[2021-12-16 03:31:27 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master01]:44558): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_a_sat02]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_b_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_f_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_e_sat02]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_e_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_d_sat02]:5665): Connection reset by peer
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_g_sat02]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_h_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_g_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_d_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_c_sat01]:5665): Operation canceled
[2021-12-16 03:31:30 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_h_sat02]:5665): Connection reset by peer
[2021-12-16 03:31:33 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_zone_a_sat02]:5665): Connection reset by peer
[2021-12-16 03:31:37 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master01]:44660): Connection reset by peer
[...]

master01 - icinga2.log

[2021-12-16 03:33:20 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51002): excessive message size
[2021-12-16 03:33:28 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:33:30 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51048): excessive message size
[2021-12-16 03:33:38 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:33:40 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51092): excessive message size
[2021-12-16 03:33:48 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:33:50 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51138): excessive message size
[2021-12-16 03:33:58 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:00 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51178): excessive message size
[2021-12-16 03:34:08 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:10 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51224): excessive message size
[2021-12-16 03:34:18 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:20 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51268): excessive message size
[2021-12-16 03:34:28 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:30 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51310): excessive message size
[2021-12-16 03:34:38 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:40 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51354): excessive message size
[2021-12-16 03:34:48 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:34:50 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51398): excessive message size
[2021-12-16 03:34:58 +0100] critical/ApiListener: Client TLS handshake failed (to [ip_master02]:5665): excessive message size
[2021-12-16 03:35:00 +0100] critical/ApiListener: Client TLS handshake failed (from [ip_master02]:51444): excessive message size
[...]

The problem appeared after the configmaster was reloaded (which was successful). The secondary master was unable to recover from this state. When I restarted the icinga process on master02, everything went back to normal. I found no special entries the syslog of both masters.

To Reproduce

unknown

Expected behavior

icinga nodes should communicate with each other.

Screenshots

Cluster Health of "master01"

2021-12-17_09-56-12

Disk /var/lib/icinga2/api/log of "master01"

2021-12-17_10-06-54

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.11-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Red Hat Enterprise Linux Server
  Platform version: 7.9 (Maipo)
  Kernel: Linux
  Kernel version: 3.10.0-1160.49.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-507-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Operating System and version: rhel 7.9
Enabled features (icinga2 feature list):

Disabled features: command compatlog debuglog elasticsearch gelf livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker graphite ido-mysql influxdb mainlog notification

Icinga Web 2 version and modules (System - About):

Icinga Web 2 Version
2.7.5
Git Commit
18996270b264976adf18d20da557d0c2806217c5
PHP Version
7.1.8
Git Commit Datum
2021-07-12
Copyright
© 2013-2021 Das Icinga Projekt

Config validation (icinga2 daemon -C):

[2021-12-17 10:03:49 +0100] information/cli: Icinga application loader (version: 2.11.11-1)
[2021-12-17 10:03:49 +0100] information/cli: Loading configuration file(s).
[2021-12-17 10:03:53 +0100] information/ConfigItem: Committing config item(s).
[2021-12-17 10:03:53 +0100] information/ApiListener: My API identity: dxzmicinga01
[2021-12-17 10:04:03 +0100] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 56, rate: 84.9333/s (5096/min 5096/5min 5096/15min);
[2021-12-17 10:04:03 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-12-17 10:04:03 +0100] information/WorkQueue: #6 (InfluxdbWriter, influxdb01) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-12-17 10:04:03 +0100] information/WorkQueue: #7 (InfluxdbWriter, influxdb02) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-12-17 10:04:03 +0100] information/WorkQueue: #10 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-12-17 10:04:03 +0100] information/WorkQueue: #9 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 2 InfluxdbWriters.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 User.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 6 TimePeriods.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 6230 Zones.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 146269 Services.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 69 ScheduledDowntimes.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 196172 Notifications.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 NotificationCommand.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 41 Comments.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 6238 Endpoints.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 259 HostGroups.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 24999 Hosts.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 595 Downtimes.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 362 CheckCommands.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 9 ApiUsers.
[2021-12-17 10:04:30 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2021-12-17 10:04:30 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-12-17 10:04:30 +0100] information/cli: Finished validating the configuration file(s).

Dec 17 '21 09:12 K0nne

In case you're seeing the same issue again, can you please capture the network traffic (for example using tcpdump) so that we can have a look at what's in these handshakes causing them to become too large?

Also looks like there was a similar report over in our community forum some time ago, but no real insights there so far: https://community.icinga.com/t/tls-excessive-message-size-seen-infrequently-on-icinga2-masters-after-configuration-reload/8133

Dec 17 '21 10:12 julianbrost

I just had the issue occur on one of my Satellite nodes, which restarting fixed the issue. Scanning the logs of the master/satellite nodes of my other instances I came across a Agent that was causing the error to be present in the logs.

IPs/FQDN's have been renamed to {SATELLITE_IP}/{AGENT_IP} and {SATELLITE_FQDN}/{AGENT_FQDN}: pcap.txt

I see the same sting repeat over and over, almost like the icinga2 agent process has loaded up the CA cert multiple times?

Jan 14 '22 19:01 stupiddr

@stupiddr Thanks, looks like a good hint! Which version of Icinga 2 are you running on which platform? Haven't found an obvious reason in the code why this should happen, but OpenSSL doesn't have the simplest API, so probably something very subtle, maybe even depending on the version.

Also, do you happen to have the raw pcap file and can open it in Wireshark, filter for tls.handshake.certificate and share the parsed output?

Jan 17 '22 11:01 julianbrost

Output of icinga2 --version:

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.49.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Contents of packet matched "tls.handshake.certificate" with IP/FQDN obscured:

Frame 6: 3341 bytes on wire (26728 bits), 3341 bytes captured (26728 bits)
    Encapsulation type: Linux cooked-mode capture v1 (25)
    Arrival Time: Jan 14, 2022 10:03:53.549859000 US Mountain Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1642179833.549859000 seconds
    [Time delta from previous captured frame: 0.009946000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.011677000 seconds]
    Frame Number: 6
    Frame Length: 3341 bytes (26728 bits)
    Capture Length: 3341 bytes (26728 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp:tls:x509sat:x509sat:x509ce:x509ce:x509sat:x509sat:x509ce:x509sat]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v1
    Packet type: Unicast to us (0)
    Link-layer address type: Ethernet (1)
    Link-layer address length: 6
    Source: Cisco_a0:00:02 (00:05:73:a0:00:02)
    Unused: 0000
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: XXX.XXX.XXX.XXX, Dst: XXX.XXX.XXX.XXX
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 3325
    Identification: 0x6d70 (28016)
    Flags: 0x40, Don't fragment
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 59
    Protocol: TCP (6)
    Header Checksum: 0xb6af [validation disabled]
    [Header checksum status: Unverified]
    Source Address: XXX.XXX.XXX.XXX
    Destination Address: XXX.XXX.XXX.XXX
Transmission Control Protocol, Src Port: 5665, Dst Port: 56634, Seq: 1, Ack: 186, Len: 3285
    Source Port: 5665
    Destination Port: 56634
    [Stream index: 0]
    [Conversation completeness: Complete, WITH_DATA (47)]
    [TCP Segment Len: 3285]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 3180101959
    [Next Sequence Number: 3286    (relative sequence number)]
    Acknowledgment Number: 186    (relative ack number)
    Acknowledgment number (raw): 1682844639
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window: 237
    [Calculated window size: 30336]
    [Window size scaling factor: 128]
    Checksum: 0x1bcb [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
        [Time since first frame in this TCP stream: 0.011677000 seconds]
        [Time since previous frame in this TCP stream: 0.009946000 seconds]
    [SEQ/ACK analysis]
        [iRTT: 0.001341000 seconds]
        [Bytes in flight: 3285]
        [Bytes sent since last PSH flag: 3285]
    TCP payload (3285 bytes)
Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 66
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 62
            Version: TLS 1.2 (0x0303)
            Random: ed9d90e090692f24cac64cb399f56b92f49fc3f8a7e242303b401f50fabf43cb
                GMT Unix Time: Apr 29, 2096 01:27:12.000000000 US Mountain Standard Time
                Random Bytes: 90692f24cac64cb399f56b92f49fc3f8a7e242303b401f50fabf43cb
            Session ID Length: 0
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
            Compression Method: null (0)
            Extensions Length: 22
            Extension: renegotiation_info (len=1)
                Type: renegotiation_info (65281)
                Length: 1
                Renegotiation Info extension
                    Renegotiation info extension length: 0
            Extension: ec_point_formats (len=4)
                Type: ec_point_formats (11)
                Length: 4
                EC point formats Length: 3
                Elliptic curves point formats (3)
                    EC point format: uncompressed (0)
                    EC point format: ansiX962_compressed_prime (1)
                    EC point format: ansiX962_compressed_char2 (2)
            Extension: session_ticket (len=0)
                Type: session_ticket (35)
                Length: 0
                Data (0 bytes)
            Extension: heartbeat (len=1)
                Type: heartbeat (15)
                Length: 1
                Mode: Peer allowed to send requests (1)
            [JA3S Fullstring: 771,49200,65281-11-35-15]
            [JA3S: f6e234011390444c303f74d09d87322d]
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 2540
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 2536
            Certificates Length: 2533
            Certificates (2533 bytes)
                Certificate Length: 1297
                Certificate: 3082050d308202f5a003020102021500ccfb86de88e693efde25de940afe2f2771439b74… (id-at-commonName={SATELLITE_FQDN})
                    signedCertificate
                        version: v3 (2)
                        serialNumber: 0x00ccfb86de88e693efde25de940afe2f2771439b74
                        signature (sha256WithRSAEncryption)
                            Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
                        issuer: rdnSequence (0)
                            rdnSequence: 1 item (id-at-commonName=Icinga CA)
                                RDNSequence item: 1 item (id-at-commonName=Icinga CA)
                                    RelativeDistinguishedName item (id-at-commonName=Icinga CA)
                                        Id: 2.5.4.3 (id-at-commonName)
                                        DirectoryString: uTF8String (4)
                                            uTF8String: Icinga CA
                        validity
                            notBefore: utcTime (0)
                                utcTime: 2021-11-01 20:57:56 (UTC)
                            notAfter: utcTime (0)
                                utcTime: 2036-10-28 20:57:56 (UTC)
                        subject: rdnSequence (0)
                            rdnSequence: 1 item (id-at-commonName={SATELLITE_FQDN})
                                RDNSequence item: 1 item (id-at-commonName={SATELLITE_FQDN})
                                    RelativeDistinguishedName item (id-at-commonName={SATELLITE_FQDN})
                                        Id: 2.5.4.3 (id-at-commonName)
                                        DirectoryString: uTF8String (4)
                                            uTF8String: {SATELLITE_FQDN}
                        subjectPublicKeyInfo
                            algorithm (rsaEncryption)
                                Algorithm Id: 1.2.840.113549.1.1.1 (rsaEncryption)
                            subjectPublicKey: 3082020a0282020100d2c99218e944b21ecc292e4d00baf7588ae7f33b103d312e345981…
                                modulus: 0x00d2c99218e944b21ecc292e4d00baf7588ae7f33b103d312e345981eb70218f68a68599…
                                publicExponent: 65537
                        extensions: 2 items
                            Extension (id-ce-basicConstraints)
                                Extension Id: 2.5.29.19 (id-ce-basicConstraints)
                                critical: True
                                BasicConstraintsSyntax [0 length]
                            Extension (id-ce-subjectAltName)
                                Extension Id: 2.5.29.17 (id-ce-subjectAltName)
                                GeneralNames: 1 item
                                    GeneralName: dNSName (2)
                                        dNSName: {SATELLITE_FQDN}
                    algorithmIdentifier (sha256WithRSAEncryption)
                        Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
                    Padding: 0
                    encrypted: 605a08faf106e3c2db55e37c53262116f32705d990529a3d8f6096cd97344a19dfcd0402…
                Certificate Length: 1230
                Certificate: 308204ca308202b2a003020102021500f0c7cf34180b1f83897a651ba20d8f2b2220b063… (id-at-commonName=Icinga CA)
                    signedCertificate
                        version: v3 (2)
                        serialNumber: 0x00f0c7cf34180b1f83897a651ba20d8f2b2220b063
                        signature (sha256WithRSAEncryption)
                            Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
                        issuer: rdnSequence (0)
                            rdnSequence: 1 item (id-at-commonName=Icinga CA)
                                RDNSequence item: 1 item (id-at-commonName=Icinga CA)
                                    RelativeDistinguishedName item (id-at-commonName=Icinga CA)
                                        Id: 2.5.4.3 (id-at-commonName)
                                        DirectoryString: uTF8String (4)
                                            uTF8String: Icinga CA
                        validity
                            notBefore: utcTime (0)
                                utcTime: 2019-10-31 14:14:27 (UTC)
                            notAfter: utcTime (0)
                                utcTime: 2034-10-27 14:14:27 (UTC)
                        subject: rdnSequence (0)
                            rdnSequence: 1 item (id-at-commonName=Icinga CA)
                                RDNSequence item: 1 item (id-at-commonName=Icinga CA)
                                    RelativeDistinguishedName item (id-at-commonName=Icinga CA)
                                        Id: 2.5.4.3 (id-at-commonName)
                                        DirectoryString: uTF8String (4)
                                            uTF8String: Icinga CA
                        subjectPublicKeyInfo
                            algorithm (rsaEncryption)
                                Algorithm Id: 1.2.840.113549.1.1.1 (rsaEncryption)
                            subjectPublicKey: 3082020a0282020100c0eb229480a2a7c2b723b4afc3512bd6421f076a7734f4af196e01…
                                modulus: 0x00c0eb229480a2a7c2b723b4afc3512bd6421f076a7734f4af196e01b389385368602259…
                                publicExponent: 65537
                        extensions: 1 item
                            Extension (id-ce-basicConstraints)
                                Extension Id: 2.5.29.19 (id-ce-basicConstraints)
                                critical: True
                                BasicConstraintsSyntax
                                    cA: True
                    algorithmIdentifier (sha256WithRSAEncryption)
                        Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
                    Padding: 0
                    encrypted: 28fd0b2e9616873bf5b7cba3644dd304cfd5c8f23abd6e26dc1eca8f915c1f3189925302…
    TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 589
        Handshake Protocol: Server Key Exchange
            Handshake Type: Server Key Exchange (12)
            Length: 585
            EC Diffie-Hellman Server Params
                Curve Type: named_curve (0x03)
                Named Curve: secp256r1 (0x0017)
                Pubkey Length: 65
                Pubkey: 040dd90acb8d14b4f8379da1d255e8a129c1e8b02a52379237336fcf8183decbbbd09630…
                Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                    Signature Hash Algorithm Hash: SHA512 (6)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Length: 512
                Signature: 6deb5cd22240ab4aea9dd3b6672a7b9112a976f9982de056704ea9b3f08d57e51c1933a0…
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 70
        Handshake Protocol: Certificate Request
            Handshake Type: Certificate Request (13)
            Length: 62
            Certificate types count: 3
            Certificate types (3 types)
                Certificate type: RSA Sign (1)
                Certificate type: DSS Sign (2)
                Certificate type: ECDSA Sign (64)
            Signature Hash Algorithms Length: 30
            Signature Hash Algorithms (15 algorithms)
                Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                    Signature Hash Algorithm Hash: SHA512 (6)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Algorithm: SHA512 DSA (0x0602)
                    Signature Hash Algorithm Hash: SHA512 (6)
                    Signature Hash Algorithm Signature: DSA (2)
                Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                    Signature Hash Algorithm Hash: SHA512 (6)
                    Signature Hash Algorithm Signature: ECDSA (3)
                Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                    Signature Hash Algorithm Hash: SHA384 (5)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Algorithm: SHA384 DSA (0x0502)
                    Signature Hash Algorithm Hash: SHA384 (5)
                    Signature Hash Algorithm Signature: DSA (2)
                Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                    Signature Hash Algorithm Hash: SHA384 (5)
                    Signature Hash Algorithm Signature: ECDSA (3)
                Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                    Signature Hash Algorithm Hash: SHA256 (4)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Algorithm: SHA256 DSA (0x0402)
                    Signature Hash Algorithm Hash: SHA256 (4)
                    Signature Hash Algorithm Signature: DSA (2)
                Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                    Signature Hash Algorithm Hash: SHA256 (4)
                    Signature Hash Algorithm Signature: ECDSA (3)
                Signature Algorithm: SHA224 RSA (0x0301)
                    Signature Hash Algorithm Hash: SHA224 (3)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Algorithm: SHA224 DSA (0x0302)
                    Signature Hash Algorithm Hash: SHA224 (3)
                    Signature Hash Algorithm Signature: DSA (2)
                Signature Algorithm: SHA224 ECDSA (0x0303)
                    Signature Hash Algorithm Hash: SHA224 (3)
                    Signature Hash Algorithm Signature: ECDSA (3)
                Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
                    Signature Hash Algorithm Hash: SHA1 (2)
                    Signature Hash Algorithm Signature: RSA (1)
                Signature Algorithm: SHA1 DSA (0x0202)
                    Signature Hash Algorithm Hash: SHA1 (2)
                    Signature Hash Algorithm Signature: DSA (2)
                Signature Algorithm: ecdsa_sha1 (0x0203)
                    Signature Hash Algorithm Hash: SHA1 (2)
                    Signature Hash Algorithm Signature: ECDSA (3)
            Distinguished Names Length: 24
            Distinguished Names (24 bytes)
                Distinguished Name Length: 22
                Distinguished Name: (id-at-commonName=Icinga CA)
                    RDNSequence item: 1 item (id-at-commonName=Icinga CA)
                        RelativeDistinguishedName item (id-at-commonName=Icinga CA)
                            Id: 2.5.4.3 (id-at-commonName)
                            DirectoryString: uTF8String (4)
                                uTF8String: Icinga CA
        Handshake Protocol: Server Hello Done
            Handshake Type: Server Hello Done (14)
            Length: 0

Jan 17 '22 21:01 stupiddr

Contents of packet matched "tls.handshake.certificate" with IP/FQDN obscured:

The certificates in that one look normal (satellite + CA). According to the pcap.txt you posted, there should also be larger handshake messages sent from the agent to the satellite (should be 4140 bytes in length, there seems to a slight difference in displayed length between tcpdump and Wireshark). Can you please look for one of these packets and share it as well?

Do the versions (Icinga 2.13.2 + CentOS 7) apply to both your satellite and agent?

Jan 18 '22 10:01 julianbrost

Lets ignore the above data I provided as hopefully this data should help narrow down the cause. So I have 2 Satellite nodes (Satellite-1 & Satellite-2) in a single satellite zone. Both with the exact same configuration other than name/fqdn/ips: icinga2 --version:

icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.49.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Satellite-2 begun experiencing the issue at hand, no clients or its partner satellite could connect, I gathered some data prior to restarting icinga2 which resolved the issue.

These log messages repeated for ~12 hours immediately following a deployment via director subsequent deployments didn't fix the issue. I grabbed the ones during the same time as my tcpdumps below to provide insight. Messages in /var/log/icinga2/icinga2.log on Satellite-1:

[2022-01-14 10:03:48 -0700] information/ApiListener: Reconnecting to endpoint '${SATELLITE_2_FQDN}' via host '${SATELLITE_2_IP}' and port '5665'
[2022-01-14 10:03:48 -0700] critical/ApiListener: Client TLS handshake failed (to [${SATELLITE_2_IP}]:5665): excessive message size
[2022-01-14 10:03:48 -0700] information/ApiListener: Finished reconnecting to endpoint '${SATELLITE_2_FQDN}' via host '${SATELLITE_2_IP}' and port '5665'
[2022-01-14 10:03:58 -0700] information/ApiListener: Reconnecting to endpoint '${SATELLITE_2_FQDN}' via host '${SATELLITE_2_IP}' and port '5665'
[2022-01-14 10:03:58 -0700] critical/ApiListener: Client TLS handshake failed (to [${SATELLITE_2_IP}]:5665): excessive message size
[2022-01-14 10:03:58 -0700] information/ApiListener: Finished reconnecting to endpoint '${SATELLITE_2_FQDN}' via host '${SATELLITE_2_IP}' and port '5665'

Messages in /var/log/icinga2/icinga2.log on Satellite-2:

[2022-01-14 10:03:48 -0700] critical/ApiListener: Client TLS handshake failed (from [::ffff:${SATELLITE_1_IP]:39424): Connection reset by peer
[2022-01-14 10:03:58 -0700] critical/ApiListener: Client TLS handshake failed (from [::ffff:${SATELLITE_1_IP}]:39430): Connection reset by peer

Above logs are in (-700 UTC) time. Below Wireshark snips are in (UTC) time.

Wireshark data during the same time period: We use Floating IP's that route to a private IP which is why the ending IP's are different in each pic.

From Satellite-1: Satellite-1 is the IP ending in 59 Satellite-2 is the IP ending in 80 (the one having the issue) Satellite-1-Wireshark

From Satellite-2: Satellite-1 is the IP ending in 79 Satellite-2 is the IP ending in 34 (the one having the issue) Satellite-2-Wireshark

I have the pcaps and logs saved if you'd like me to look for anything additional or provide the parsed output of any of these packets with private info removed.

Thanks!

Jan 20 '22 18:01 stupiddr

The packets of interest would be the ones sent by the TCP/TLS client (Satellite-2 in this case) after the "Server Hello" message. In my local tests, Wireshark parsed them as "Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message", not sure why it doesn't in your case. So in the screenshot from Satellite-2, can you check if Wireshark manages to parse anything meaningful in these packets (these showing up as TLSv1.2 Encrypted Handshake Message: No. 11, 14, 16, 20, 22, 24).

But something is very wrong with these packets, as there are multiple ones over 10kB in size. Usually there should be just on message around 3kB, one more message from the server and the handshake is done, excessive handshake size indeed.

Also, would you be willing to share the raw pcap files privately?

Jan 21 '22 09:01 julianbrost

Hey! Apologies for the disappearing act, had some issues come up that had me away for quite some time. I cannot provide the raw pcaps due to company policies.

Looking at the packets (No. 11, 14, 16, 20, 22, 24) This is the only thing meaningful parsed is: "Icinga CA0...191031141427Z..341027141427Z0.1.0...U....Icinga CA0"

This string just repeats over and over in the packets 8 times in Packet No. 11 12 times in Packet No. 14 9 times in Packet No. 16 12 times in Packet No. 20 11 times in Packet No. 22 13 times in Packet No. 24

Numbers in the string as linux timestamps (if thats what they are) we get: 191031141427Z = 46 years ago 341027141427Z = 42 years ago

Apr 22 '22 18:04 stupiddr

This is the only thing meaningful parsed is: "Icinga CA0...191031141427Z..341027141427Z0.1.0...U....Icinga CA0"

What I meant by parsed isn't just the ASCII dump (which misses information as half the characters are replaced with .) but rather opening the file in Wireshark and let it analyze the structure of the packet.

Numbers in the string as linux timestamps (if thats what they are) we get: 191031141427Z = 46 years ago 341027141427Z = 42 years ago

They aren't. Add 20 in front and the numbers start to make sense. It's 2019-10-31 14:14:27Z and 2034-10-27 14:14:27Z which sounds like plausible values for the validity period of your Icinga CA certificate.

Apr 25 '22 07:04 julianbrost

Hopefully, these are more useful.

Packet No. 11

Frame 11: 10276 bytes on wire (82208 bits), 10276 bytes captured (82208 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.553522000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.553522000 seconds
  [Time delta from previous captured frame: 0.000043000 seconds]
  [Time delta from previous displayed frame: 0.000043000 seconds]
  [Time since reference or first frame: 0.015340000 seconds]
  Frame Number: 11
  Frame Length: 10276 bytes (82208 bits)
  Capture Length: 10276 bytes (82208 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 10260
  Identification: 0xd448 (54344)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x2fc0 [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 14786, Ack: 3286, Len: 10220
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 10220]
  Sequence Number: 14786    (relative sequence number)
  Sequence Number (raw): 1682859239
  [Next Sequence Number: 25006    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x36e2 [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.015340000 seconds]
      [Time since previous frame in this TCP stream: 0.000043000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 17520]
      [Bytes sent since last PSH flag: 24820]
  TCP payload (10220 bytes)
  TCP segment data (1789 bytes)
  [Reassembled PDU in frame: 14]
  TCP segment data (8431 bytes)
[3 Reassembled TCP Segments (16389 bytes): #8(7300), #9(7300), #11(1789)]
  [Frame: 8, payload: 0-7299 (7300 bytes)]
  [Frame: 9, payload: 7300-14599 (7300 bytes)]
  [Frame: 11, payload: 14600-16388 (1789 bytes)]
  [Segment count: 3]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 16030340000b01e6bb01e6b80005113082050d308202f5a003020102021500da52b67687…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Packet No. 14

Frame 14: 14656 bytes on wire (117248 bits), 14656 bytes captured (117248 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.553571000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.553571000 seconds
  [Time delta from previous captured frame: 0.000010000 seconds]
  [Time delta from previous displayed frame: 0.000010000 seconds]
  [Time since reference or first frame: 0.015389000 seconds]
  Frame Number: 14
  Frame Length: 14656 bytes (117248 bits)
  Capture Length: 14656 bytes (117248 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 14640
  Identification: 0xd452 (54354)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x1e9a [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 29386, Ack: 3286, Len: 14600
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 14600]
  Sequence Number: 29386    (relative sequence number)
  Sequence Number (raw): 1682873839
  [Next Sequence Number: 43986    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x47fe [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.015389000 seconds]
      [Time since previous frame in this TCP stream: 0.000010000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 29200]
      [Bytes sent since last PSH flag: 43800]
  TCP payload (14600 bytes)
  TCP segment data (3578 bytes)
  [Reassembled PDU in frame: 16]
  TCP segment data (11022 bytes)
[3 Reassembled TCP Segments (16389 bytes): #11(8431), #12(4380), #14(3578)]
  [Frame: 11, payload: 0-8430 (8431 bytes)]
  [Frame: 12, payload: 8431-12810 (4380 bytes)]
  [Frame: 14, payload: 12811-16388 (3578 bytes)]
  [Segment count: 3]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 16030340005264c6c62d9c45a0a25f5a0dc2113c0e379d799d19cc3f9eb371bdfff3c362…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Packet No. 16

Frame 16: 11736 bytes on wire (93888 bits), 11736 bytes captured (93888 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.553832000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.553832000 seconds
  [Time delta from previous captured frame: 0.000017000 seconds]
  [Time delta from previous displayed frame: 0.000017000 seconds]
  [Time since reference or first frame: 0.015650000 seconds]
  Frame Number: 16
  Frame Length: 11736 bytes (93888 bits)
  Capture Length: 11736 bytes (93888 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 11720
  Identification: 0xd45c (54364)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x29f8 [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 43986, Ack: 3286, Len: 11680
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 11680]
  Sequence Number: 43986    (relative sequence number)
  Sequence Number (raw): 1682888439
  [Next Sequence Number: 55666    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x3c96 [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.015650000 seconds]
      [Time since previous frame in this TCP stream: 0.000017000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 30660]
      [Bytes sent since last PSH flag: 55480]
  TCP payload (11680 bytes)
  TCP segment data (5367 bytes)
  [Reassembled PDU in frame: 20]
  TCP segment data (6313 bytes)
[2 Reassembled TCP Segments (16389 bytes): #14(11022), #16(5367)]
  [Frame: 14, payload: 0-11021 (11022 bytes)]
  [Frame: 16, payload: 11022-16388 (5367 bytes)]
  [Segment count: 2]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 1603034000364a353ecb5686587a77d0a058081ca17de5a57008916fc2d014c4ba4669c3…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Packet No. 20

Frame 20: 14656 bytes on wire (117248 bits), 14656 bytes captured (117248 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.553886000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.553886000 seconds
  [Time delta from previous captured frame: 0.000010000 seconds]
  [Time delta from previous displayed frame: 0.000010000 seconds]
  [Time since reference or first frame: 0.015704000 seconds]
  Frame Number: 20
  Frame Length: 14656 bytes (117248 bits)
  Capture Length: 14656 bytes (117248 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 14640
  Identification: 0xd466 (54374)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x1e86 [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 58586, Ack: 3286, Len: 14600
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 14600]
  Sequence Number: 58586    (relative sequence number)
  Sequence Number (raw): 1682903039
  [Next Sequence Number: 73186    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x47fe [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.015704000 seconds]
      [Time since previous frame in this TCP stream: 0.000010000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 29200]
      [Bytes sent since last PSH flag: 73000]
  TCP payload (14600 bytes)
  TCP segment data (7156 bytes)
  [Reassembled PDU in frame: 22]
  TCP segment data (7444 bytes)
[3 Reassembled TCP Segments (16389 bytes): #16(6313), #18(2920), #20(7156)]
  [Frame: 16, payload: 0-6312 (6313 bytes)]
  [Frame: 18, payload: 6313-9232 (2920 bytes)]
  [Frame: 20, payload: 9233-16388 (7156 bytes)]
  [Segment count: 3]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 1603034000a9702d614fc578d553805455d9a4266d089ee960e2b07fe2817f7c80ce6b0a…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Packet No. 22

Frame 22: 14656 bytes on wire (117248 bits), 14656 bytes captured (117248 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.554792000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.554792000 seconds
  [Time delta from previous captured frame: 0.000017000 seconds]
  [Time delta from previous displayed frame: 0.000017000 seconds]
  [Time since reference or first frame: 0.016610000 seconds]
  Frame Number: 22
  Frame Length: 14656 bytes (117248 bits)
  Capture Length: 14656 bytes (117248 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 14640
  Identification: 0xd470 (54384)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x1e7c [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 73186, Ack: 3286, Len: 14600
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 14600]
  Sequence Number: 73186    (relative sequence number)
  Sequence Number (raw): 1682917639
  [Next Sequence Number: 87786    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x47fe [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.016610000 seconds]
      [Time since previous frame in this TCP stream: 0.000017000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 29200]
      [Bytes sent since last PSH flag: 87600]
  TCP payload (14600 bytes)
  TCP segment data (8945 bytes)
  [Reassembled PDU in frame: 24]
  TCP segment data (5655 bytes)
[2 Reassembled TCP Segments (16389 bytes): #20(7444), #22(8945)]
  [Frame: 20, payload: 0-7443 (7444 bytes)]
  [Frame: 22, payload: 7444-16388 (8945 bytes)]
  [Segment count: 2]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 16030340001006035504030c094963696e676120434130820222300d06092a864886f70d…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Packet No. 24

Frame 24: 16116 bytes on wire (128928 bits), 16116 bytes captured (128928 bits)
  Encapsulation type: Linux cooked-mode capture v1 (25)
  Arrival Time: Jan 14, 2022 10:03:53.554812000 US Mountain Standard Time
  [Time shift for this packet: 0.000000000 seconds]
  Epoch Time: 1642179833.554812000 seconds
  [Time delta from previous captured frame: 0.000006000 seconds]
  [Time delta from previous displayed frame: 0.000006000 seconds]
  [Time since reference or first frame: 0.016630000 seconds]
  Frame Number: 24
  Frame Length: 16116 bytes (128928 bits)
  Capture Length: 16116 bytes (128928 bits)
  [Frame is marked: False]
  [Frame is ignored: False]
  [Protocols in frame: sll:ethertype:ip:tcp:tls]
  [Coloring Rule Name: TCP]
  [Coloring Rule String: tcp]
Linux cooked capture v1
  Packet type: Sent by us (4)
  Link-layer address type: Ethernet (1)
  Link-layer address length: 6
  Source: fa:16:3e:9f:be:66 (fa:16:3e:9f:be:66)
  Unused: 0000
  Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: Satellite-2, Dst: Satellite-1
  0100 .... = Version: 4
  .... 0101 = Header Length: 20 bytes (5)
  Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
      0000 00.. = Differentiated Services Codepoint: Default (0)
      .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
  Total Length: 16100
  Identification: 0xd47a (54394)
  Flags: 0x40, Don't fragment
      0... .... = Reserved bit: Not set
      .1.. .... = Don't fragment: Set
      ..0. .... = More fragments: Not set
  ...0 0000 0000 0000 = Fragment Offset: 0
  Time to Live: 64
  Protocol: TCP (6)
  Header Checksum: 0x18be [validation disabled]
  [Header checksum status: Unverified]
  Source Address: Satellite-2
  Destination Address: Satellite-1
Transmission Control Protocol, Src Port: 56634, Dst Port: 5665, Seq: 87786, Ack: 3286, Len: 16060
  Source Port: 56634
  Destination Port: 5665
  [Stream index: 0]
  [Conversation completeness: Complete, WITH_DATA (47)]
  [TCP Segment Len: 16060]
  Sequence Number: 87786    (relative sequence number)
  Sequence Number (raw): 1682932239
  [Next Sequence Number: 103846    (relative sequence number)]
  Acknowledgment Number: 3286    (relative ack number)
  Acknowledgment number (raw): 3180105244
  0101 .... = Header Length: 20 bytes (5)
  Flags: 0x010 (ACK)
      000. .... .... = Reserved: Not set
      ...0 .... .... = Nonce: Not set
      .... 0... .... = Congestion Window Reduced (CWR): Not set
      .... .0.. .... = ECN-Echo: Not set
      .... ..0. .... = Urgent: Not set
      .... ...1 .... = Acknowledgment: Set
      .... .... 0... = Push: Not set
      .... .... .0.. = Reset: Not set
      .... .... ..0. = Syn: Not set
      .... .... ...0 = Fin: Not set
      [TCP Flags: ·······A····]
  Window: 280
  [Calculated window size: 35840]
  [Window size scaling factor: 128]
  Checksum: 0x4db2 [unverified]
  [Checksum Status: Unverified]
  Urgent Pointer: 0
  [Timestamps]
      [Time since first frame in this TCP stream: 0.016630000 seconds]
      [Time since previous frame in this TCP stream: 0.000006000 seconds]
  [SEQ/ACK analysis]
      [iRTT: 0.001341000 seconds]
      [Bytes in flight: 30660]
      [Bytes sent since last PSH flag: 103660]
  TCP payload (16060 bytes)
  TCP segment data (10734 bytes)
  TCP segment data (5326 bytes)
[2 Reassembled TCP Segments (16389 bytes): #22(5655), #24(10734)]
  [Frame: 22, payload: 0-5654 (5655 bytes)]
  [Frame: 24, payload: 5655-16388 (10734 bytes)]
  [Segment count: 2]
  [Reassembled TCP length: 16389]
  [Reassembled TCP Data: 160303400060955b5e4440c01c48a53fb66adace7557f0cab309642d32125fa8c18859de…]
Transport Layer Security
  TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
      Content Type: Handshake (22)
      Version: TLS 1.2 (0x0303)
      Length: 16384
      Handshake Protocol: Encrypted Handshake Message

Apr 26 '22 15:04 stupiddr

Length: 16384 Handshake Protocol: Encrypted Handshake Message

Not what I was hoping for unfortunately :(

But given that all these share about the same timestamp, I think they might all be part of the same handshake message and this makes Wireshark fail to parse it. Wireshark can export the TCP stream, but I don't know a good tool to parse a TLS handshake from that.

Apr 26 '22 15:04 julianbrost

The best tool I know is called Wireshark 🙈

May 03 '22 16:05 Al2Klimov

I am closing this. The problem has never appeared again.

May 23 '23 17:05 K0nne