harmony icon indicating copy to clipboard operation
harmony copied to clipboard
verified publisher icon harmony-one

Mainnet S0 node got stuck

Open sophoah opened this issue 4 years ago • 2 comments

Describe the bug It's the second time in two days we have in mainnet an RPC explorer node being stuck. The fix is to simply restart the node.

To Reproduce N/A it just happens in the network for now

Expected behavior It shouldn't get stuck

Screenshots See logs of an recent RPC node that got stuck :

tail -f latest/zero*.log | grep -v 'transaction'
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19707296,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:39.176479178Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19707296,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:39.254560632Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2 ","blockHeight":19707296,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:39.310893548Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19707296,"lastHeight":19701843,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-23T10:51:39.310937124Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1 ","blockHeight":19707299,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:46.954560637Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19707299,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:47.032565158Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2 ","blockHeight":19707299,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-23T10:51:47.089218806Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19707299,"lastHeight":19701843,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-23T10:51:47.089256842Z","message":"[SYNC] Checking sync status"}

Node was stuck at block 19701843

RPC node is running the current mainnet node version v7174-v4.3.0-0-g15f9b2d1

Environment (please complete the following information):

  • OS: [Linux, MacOS]
  • Go environment [ go env ]

Additional context one DNS node has a PR #3938 currently on it for testing s0-min-sync-3 however a grep DNS calculate response doesn't show any hit

sophoah avatar Nov 23 '21 10:11 sophoah

adding into this issue as another instance of internal RPC node stuck. this node stuck had some other logs messages. node was stuck at block 19824640 new log messages from previous capture :

{"level":"error","msgBlock":19828401,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:55.86039816Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:55.860426048Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828402,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:58.37962429Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:58.379649851Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:53","time":"2021-11-27T05:30:01.049577726Z","message":"[Explorer] not have enough signature power"}

and I don't think those are actually relevant

here is the entire log

tail -f latest/zero*.log | grep -v 'transaction'
{"level":"error","msgBlock":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:39.958982919Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:39.95902115Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.222960528Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.300731696Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.357198299Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828395,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:29:41.357241803Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.394922266Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.473011332Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.52937196Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828395,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:29:41.52942909Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.569654248Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.647612921Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828395,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:41.703921115Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828395,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:29:41.703944532Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828396,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:42.637411094Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:42.637435142Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828397,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:45.214588129Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:45.214616277Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828398,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:47.879390101Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:47.879414821Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828399,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:50.594589434Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:50.594617744Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828400,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:53.227659551Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:53.227872433Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828399,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:54.056660747Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828399,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:54.134622815Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828399,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:29:54.190857473Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828399,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:29:54.190876389Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828401,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:55.86039816Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:55.860426048Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828402,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:29:58.37962429Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:29:58.379649851Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:53","time":"2021-11-27T05:30:01.049577726Z","message":"[Explorer] not have enough signature power"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:02.083665018Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:02.161397014Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:02.217596218Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828403,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:02.217615808Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:03.231452288Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:03.309341712Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828403,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:03.365700639Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828403,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:03.365756498Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828404,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:03.549014289Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:03.549040277Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828405,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:06.330722257Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:06.330745437Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:08.943570625Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:08.943595934Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.252868552Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.330858653Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.387240101Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828406,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:11.387303582Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.416858524Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.49483466Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"error","msgBlock":19828407,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:11.539439752Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:11.539464622Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.551257725Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828406,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:11.551307888Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.599214232Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.676995932Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828406,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:11.733428545Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828406,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:11.733447034Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828408,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:14.172497058Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:14.172523481Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828409,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:16.933781588Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:16.933808705Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828410,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:19.776456151Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:19.77650272Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","blockHeight":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:124","time":"2021-11-27T05:30:22.391424449Z","message":"[Explorer] Adding new block for explorer node"}
{"level":"error","caller":"/home/runner/work/harmony/harmony/harmony/core/blockchain.go:1650","time":"2021-11-27T05:30:22.391854124Z","message":"\n########## BAD BLOCK #########\nChain config: {ChainID: 1 EthCompatibleChainID: 1666600000 EIP155: 28 CrossTx: 28 Staking: 186 CrossLink: 186 ReceiptLog: 101 SHA3Epoch:725}\n\nNumber: 19828411\nEpoch: 778\nNumTxn: 95\nNumStkTxn: 0\nHash: 0x4f35ec6b65ca5f622dfd14ce677a20022528d4d00f4fe862d19f80bd45b0ea13\n\n\nError: unknown ancestor\n##############################\n"}
{"level":"error","error":"unknown ancestor","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:149","time":"2021-11-27T05:30:22.391868885Z","message":"[Explorer] Error when adding new block for explorer node"}
{"level":"info","blockNum":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:159","time":"2021-11-27T05:30:22.39188105Z","message":"[Explorer] Committing block into explorer DB"}
{"level":"info","module":"explorer storage","block number":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/explorer/storage.go:135","time":"2021-11-27T05:30:22.497417057Z","message":"writing explorer DB"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:24.086050568Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:24.163946008Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828411,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:24.220243474Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828411,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:24.220262915Z","message":"[SYNC] Checking sync status"}
{"level":"info","blockHeight":19828412,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:124","time":"2021-11-27T05:30:24.983672792Z","message":"[Explorer] Adding new block for explorer node"}
{"level":"error","caller":"/home/runner/work/harmony/harmony/harmony/core/blockchain.go:1650","time":"2021-11-27T05:30:24.984060978Z","message":"\n########## BAD BLOCK #########\nChain config: {ChainID: 1 EthCompatibleChainID: 1666600000 EIP155: 28 CrossTx: 28 Staking: 186 CrossLink: 186 ReceiptLog: 101 SHA3Epoch:725}\n\nNumber: 19828412\nEpoch: 778\nNumTxn: 87\nNumStkTxn: 0\nHash: 0x0671253d9c4420f0859e5e94973a2bc289003db8a34205e2f2a2faa7ef34a76c\n\n\nError: unknown ancestor\n##############################\n"}
{"level":"error","error":"unknown ancestor","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:149","time":"2021-11-27T05:30:24.984076082Z","message":"[Explorer] Error when adding new block for explorer node"}
{"level":"info","blockNum":19828412,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:159","time":"2021-11-27T05:30:24.984082767Z","message":"[Explorer] Committing block into explorer DB"}
{"level":"info","module":"explorer storage","block number":19828412,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/explorer/storage.go:135","time":"2021-11-27T05:30:25.080323463Z","message":"writing explorer DB"}
{"level":"error","msgBlock":19828413,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:27.683685953Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:27.683716458Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:30.355120163Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:30.355147444Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:32.091477038Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:32.169326274Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:32.225769835Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828414,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:32.225813779Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:33.246903655Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:33.324804936Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828414,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:33.381138341Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828414,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:33.38115921Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828415,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:34.336515848Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:34.336542514Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:37.913518582Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:37.913544828Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828417,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:41.019768289Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:41.019794443Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.283231269Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.361087595Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.417549057Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828416,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:41.417612078Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.429426196Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.507355515Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.563609492Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828416,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:41.563651894Z","message":"[SYNC] Checking sync status"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.606838724Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.684772503Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828416,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:41.741111951Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828416,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:41.741137863Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828418,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:43.740093102Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:43.740121457Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828419,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:46.400804285Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:46.400830602Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828420,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:49.173492362Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:49.173516481Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828421,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:52.700826582Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:52.700854081Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828421,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:54.114683292Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828421,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:54.192684977Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-2","blockHeight":19828421,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:30:54.248990679Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","OtherHeight":19828421,"lastHeight":19824640,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1080","time":"2021-11-27T05:30:54.249011221Z","message":"[SYNC] Checking sync status"}
{"level":"error","msgBlock":19828422,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:55.514095699Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:55.514123697Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828423,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:30:58.125114208Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:30:58.125141946Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"error","msgBlock":19828424,"caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:73","time":"2021-11-27T05:31:00.801013525Z","message":"[Explorer] Failed to verify the multi signature for commit phase"}
{"level":"info","item":null,"caller":"/home/runner/work/harmony/harmony/harmony/node/node.go:916","time":"2021-11-27T05:31:00.801040738Z","message":"[p2p]: issue while handling incoming p2p message: explorer failed to verify the multi signature for commit phase"}
{"level":"info","peerIP":"s0-min-sync-1","blockHeight":19828424,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:31:02.120732864Z","message":"[SYNC] getMaxPeerHeight"}
{"level":"info","peerIP":"s0-min-sync-3","blockHeight":19828424,"caller":"/home/runner/work/harmony/harmony/harmony/api/service/legacysync/syncing.go:1035","time":"2021-11-27T05:31:02.19862816Z","message":"[SYNC] getMaxPeerHeight"}
^C

sophoah avatar Nov 27 '21 05:11 sophoah

Any updates on this issue? My nodes have been getting worse performance with recent updates

kingli-crypto avatar Dec 20 '21 14:12 kingli-crypto