besu icon indicating copy to clipboard operation
besu copied to clipboard
verified publisher icon hyperledger

Losing 25 useful peers within 11 seconds

Open pinges opened this issue 3 years ago • 3 comments

This node was running fine on Goerli when it lost 25 useful peers within 11 seconds.

"Useful" peers can be identified by a reputation > 100 (see logs) . The reputation starts at 100 and with every useful response is incremented by 1 (is decremented for other behaviour).

Grepped from the log:

~ % grep -i termin goodPeersDisconnecting.log
2022-07-28 17:25:57.899+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 1400996420, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:25:57.899+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 1400996420, reason 0x0b TIMEOUT
2022-07-28 17:25:57.906+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 61613170, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:25:57.906+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 61613170, reason 0x0b TIMEOUT
2022-07-28 17:25:57.938+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Terminating connection 977750439, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:25:57.938+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Terminating connection 977750439, reason 0x0b TIMEOUT
2022-07-28 17:25:57.960+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 2076069203, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:25:57.960+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 2076069203, reason 0x0b TIMEOUT
2022-07-28 17:25:58.133+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 1125325009, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:25:58.133+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 1125325009, reason 0x0b TIMEOUT
2022-07-28 17:26:07.282+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Terminating connection 508336088, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.283+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Terminating connection 1515310701, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.468+00:00 | nioEventLoopGroup-3-9 | DEBUG | AbstractPeerConnection | Terminating connection 735312525, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.478+00:00 | nioEventLoopGroup-3-9 | DEBUG | AbstractPeerConnection | Terminating connection 891115807, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.486+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 726437042, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.682+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 777430832, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.684+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 2137147640, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.697+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 164782903, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.698+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 833094358, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.700+00:00 | nioEventLoopGroup-3-4 | DEBUG | AbstractPeerConnection | Terminating connection 1352598947, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.830+00:00 | nioEventLoopGroup-3-4 | DEBUG | AbstractPeerConnection | Terminating connection 271808806, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.831+00:00 | nioEventLoopGroup-3-4 | DEBUG | AbstractPeerConnection | Terminating connection 1522644462, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.957+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 231308390, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:07.958+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 2023919271, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.042+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Terminating connection 1644912518, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.054+00:00 | nioEventLoopGroup-3-2 | DEBUG | AbstractPeerConnection | Terminating connection 456315532, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.056+00:00 | nioEventLoopGroup-3-2 | DEBUG | AbstractPeerConnection | Terminating connection 911257545, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.134+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 585961575, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.135+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 1508837149, reason 0x01 TCP_SUBSYSTEM_ERROR
2022-07-28 17:26:08.135+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 1726789671, reason 0x01 TCP_SUBSYSTEM_ERROR

The attached file called "goodPeersDisconnecting.log" contains the log for the 11 seconds cleaned by grepping for the entries concerning the connections that are disconnecting. The attached file called "goodPeersDisconnectingFull.log" is the full log contents for that time.

[Uploading... goodPeersDisconnecting.log]

[Uploading... goodPeersDisconnectingFull.log]

pinges avatar Aug 01 '22 05:08 pinges

I have the original log file available, but it is 11GB in size.

pinges avatar Aug 01 '22 05:08 pinges

Disconnects with reason TIMEOUT from full log file: 2022-07-28 04:47:42.502+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 685122028, reason 0x0b TIMEOUT 2022-07-28 06:07:55.932+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Disconnecting connection 404130061, reason 0x0b TIMEOUT 2022-07-28 06:35:48.213+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 50639526, reason 0x0b TIMEOUT 2022-07-28 06:49:03.096+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Disconnecting connection 2090135662, reason 0x0b TIMEOUT 2022-07-28 06:49:27.866+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 289528100, reason 0x0b TIMEOUT 2022-07-28 08:37:02.775+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 420949187, reason 0x0b TIMEOUT 2022-07-28 08:37:03.993+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 835522595, reason 0x0b TIMEOUT 2022-07-28 10:24:27.664+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Disconnecting connection 1815480295, reason 0x0b TIMEOUT 2022-07-28 12:19:00.531+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 1354296997, reason 0x0b TIMEOUT 2022-07-28 13:14:57.655+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 629978772, reason 0x0b TIMEOUT 2022-07-28 14:21:40.102+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Disconnecting connection 655890139, reason 0x0b TIMEOUT 2022-07-28 15:09:04.853+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 9111664, reason 0x0b TIMEOUT 2022-07-28 17:25:57.899+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Terminating connection 1400996420, reason 0x0b TIMEOUT 2022-07-28 17:25:57.906+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Terminating connection 61613170, reason 0x0b TIMEOUT 2022-07-28 17:25:57.938+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Terminating connection 977750439, reason 0x0b TIMEOUT 2022-07-28 17:25:57.960+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 2076069203, reason 0x0b TIMEOUT 2022-07-28 17:25:58.133+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 1125325009, reason 0x0b TIMEOUT 2022-07-28 17:26:03.048+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 1352598947, reason 0x0b TIMEOUT 2022-07-28 17:40:37.321+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Disconnecting connection 1876310112, reason 0x0b TIMEOUT 2022-07-28 17:40:39.254+00:00 | nioEventLoopGroup-3-1 | DEBUG | AbstractPeerConnection | Disconnecting connection 1841326411, reason 0x0b TIMEOUT 2022-07-28 18:07:33.795+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Disconnecting connection 1569828653, reason 0x0b TIMEOUT 2022-07-28 18:12:29.341+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 1947995979, reason 0x0b TIMEOUT 2022-07-28 19:05:10.558+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Terminating connection 137329395, reason 0x0b TIMEOUT 2022-07-28 19:16:42.005+00:00 | nioEventLoopGroup-3-7 | DEBUG | AbstractPeerConnection | Disconnecting connection 1212902293, reason 0x0b TIMEOUT 2022-07-28 19:21:26.823+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Terminating connection 894913976, reason 0x0b TIMEOUT 2022-07-28 19:21:56.257+00:00 | nioEventLoopGroup-3-2 | DEBUG | AbstractPeerConnection | Disconnecting connection 653435822, reason 0x0b TIMEOUT 2022-07-28 20:14:48.697+00:00 | nioEventLoopGroup-3-10 | DEBUG | AbstractPeerConnection | Disconnecting connection 1006879460, reason 0x0b TIMEOUT 2022-07-28 20:57:45.814+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Disconnecting connection 1184500912, reason 0x0b TIMEOUT 2022-07-28 22:05:49.083+00:00 | nioEventLoopGroup-3-1 | DEBUG | AbstractPeerConnection | Disconnecting connection 105772993, reason 0x0b TIMEOUT 2022-07-28 23:17:05.785+00:00 | nioEventLoopGroup-3-8 | DEBUG | AbstractPeerConnection | Disconnecting connection 781079243, reason 0x0b TIMEOUT 2022-07-28 23:17:53.213+00:00 | nioEventLoopGroup-3-1 | DEBUG | AbstractPeerConnection | Terminating connection 1241719049, reason 0x0b TIMEOUT 2022-07-29 00:04:14.408+00:00 | nioEventLoopGroup-3-9 | DEBUG | AbstractPeerConnection | Disconnecting connection 1440722720, reason 0x0b TIMEOUT 2022-07-29 00:04:36.947+00:00 | nioEventLoopGroup-3-4 | DEBUG | AbstractPeerConnection | Disconnecting connection 781407646, reason 0x0b TIMEOUT 2022-07-29 00:09:24.452+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 910848200, reason 0x0b TIMEOUT 2022-07-29 00:33:35.128+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Disconnecting connection 54715734, reason 0x0b TIMEOUT 2022-07-29 02:17:56.437+00:00 | nioEventLoopGroup-3-1 | DEBUG | AbstractPeerConnection | Disconnecting connection 1943305489, reason 0x0b TIMEOUT 2022-07-29 02:38:28.371+00:00 | nioEventLoopGroup-3-4 | DEBUG | AbstractPeerConnection | Disconnecting connection 800061439, reason 0x0b TIMEOUT 2022-07-29 03:49:03.657+00:00 | EthScheduler-Timer-0 | DEBUG | AbstractPeerConnection | Disconnecting connection 2058173025, reason 0x0b TIMEOUT 2022-07-29 04:15:52.526+00:00 | nioEventLoopGroup-3-3 | DEBUG | AbstractPeerConnection | Disconnecting connection 1022040584, reason 0x0b TIMEOUT 2022-07-29 04:18:12.889+00:00 | nioEventLoopGroup-3-6 | DEBUG | AbstractPeerConnection | Terminating connection 1339215602, reason 0x0b TIMEOUT 2022-07-29 04:28:09.416+00:00 | nioEventLoopGroup-3-5 | DEBUG | AbstractPeerConnection | Terminating connection 744263814, reason 0x0b TIMEOUT

pinges avatar Aug 01 '22 05:08 pinges

https://app.zenhub.com/files/206414745/c4c7dae8-c7e2-4b14-8bfb-8d8845c262d3/download

pinges avatar Aug 09 '22 02:08 pinges

Closing for tacking in other issues

non-fungible-nelson avatar Jan 26 '23 12:01 non-fungible-nelson