node-binary Can not get node working after update to 0.8.2

Hi

Our node got stuck with Wrong.Block.Header.AppHash error. We've performed state-recovery to 149170000 (the last but one snapshot), but after that the node gets stuсk at 1491700020, throwing the following error:

bnbchaind: panic: Orphan expires before it comes alive. 149170017 > 149170000
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: goroutine 640170 [running]:
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*nodeDB).saveOrphan(0xc000ae3c50, 0xc0362a8860, 0x20, 0x20, 0x8e42761,
0x8e42750)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/nodedb.go:166
 +0x253
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*nodeDB).SaveOrphans(0xc000ae3c50, 0x8e42765, 0xc031a16090)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/nodedb.go:159
 +0x1b2
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*MutableTree).SaveVersion(0xc004fb1e60, 0xc00006e6b0, 0xc00006e6b0, 0x3
00000001aa7201, 0xffffffffffffffff, 0x1, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/mutable_tree.
go:407 +0x63b
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.(*IavlStore).Commit(0xc000b49380, 0xc00099e510, 0x169c042, 0x3,
0xc0004955b8)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/store/iavlstore.g
o:80 +0x32
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.commitStores(0x8e42765, 0xc00099f290, 0x7020105, 0x0, 0xffffffff
ffffffff, 0xc010e428d8)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/store/rootmultist
ore.go:489 +0x1e5
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.(*rootMultiStore).Commit(0xc00016c4d0, 0x0, 0x0, 0x0, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/store/rootmultist
ore.go:209 +0x68
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000ae0a50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/baseapp/baseapp.g
o:957 +0xff
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/binance-chain/node/app.(*BinanceChain).Commit(0xc0009d5200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0
, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /server/temp/node/app/app.go:866 +0x50
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).CommitSync(0xc026d5a630, 0x0, 0x
0, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/server/concurrent
/async_local_client.go:377 +0x1fe
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc026d5c360, 0x0, 0x0, 0xc03
71b43c0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/proxy/app_conn.go:
119 +0x33
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc03ab8c150, 0xa, 0x0, 0xc03751a740
, 0x6, 0xc03d0b81a0, 0x14, 0x8e42765, 0xe4e8c67, 0xc0370071c0, ...)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/state/execution.go
:209 +0x224
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc03ab8c150, 0xa, 0x0, 0xc03751
a740, 0x6, 0xc03d0b81a0, 0x14, 0x8e42765, 0xe4e8c67, 0xc0370071c0, ...)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/state/execution.go
:163 +0x6b6
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0188441a0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reac
tor.go:401 +0x983
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reac
tor.go:127 +0x84
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 12 09:21:19 ip-172-31-29-224 systemd: Unit binance.service entered failed state.
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service failed.
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service holdoff time over, scheduling restart.

bnc.log shows that the last Executed block is 1491700021. It would be great if you could help us. Thanks

Mar 12 '21 09:03 dzmitrykliapkou

Please try revert to 1491690000.

and double check the open file limit. It cannot be less than 1024.

Mar 17 '21 09:03 chainwhisper

I've tried to revert to 1491690000, but after that binance service refused to start with new error:

bnbchaind: failed to get rootMultistore: no data

Also I've checked open file limit and it is 655350 per process

Mar 18 '21 11:03 dzmitrykliapkou

I've tried to revert to 1491690000, but after that binance service refused to start with new error:
bnbchaind: failed to get rootMultistore: no data
Also I've checked open file limit and it is 655350 per process

please try the latest snapshot: https://docs.binance.org/guides/node/snapshot.html

Mar 18 '21 12:03 chainwhisper

please try the latest snapshot: https://docs.binance.org/guides/node/snapshot.html

I've downloaded the latest snapshot and started the service, but after ~1hr binance service failed with:

Mar 26 12:56:23 ip-172-31-29-224 bnbchaind: ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected A5B794D9B4D96
CDA0F60BFA1BE660BD49B5DEF0A0262A3A3FC066E5965742575, got 5AEDFAE1C95C3D21C27EF0091DB2045535B53E0D92A18B4FCE40751666E81881
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service: main process exited, code=exited, status=1/FAILURE
Mar 26 12:56:23 ip-172-31-29-224 systemd: Unit binance.service entered failed state.
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service failed.
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service holdoff time over, scheduling restart.

Mar 26 '21 13:03 dzmitrykliapkou

@huangsuyu same issue Did you solve the issue?

Apr 14 '21 12:04 DevMinJong

please try these steps: @DevMinJong @dzmitrykliapkou

stop the bnbchaind
delete the data folder
extract the snapshot to the new data folder
update the state_sync_reactor to false
start the bnbchaind

Apr 14 '21 13:04 chainwhisper

Same issue here. Using the April snapshot, and all the steps above.

Binance Chain Release: v0.8.2;Binance Chain Commit: 88ce4872; Cosmos Release: v0.25.0-binance.24; Tendermint Release: v0.32.3-binance.5;

log:

I[2021-06-11|07:07:33.695] replayed all tx. Starting match module=dex height=153975665 I[2021-06-11|07:07:33.695] No order comes in for the block module=dexkeeper I[2021-06-11|07:07:37.742] ABCI Handshake App Info module=consensus height=153975665 hash=E8090A2A1D20258FDB3C2EEE151C4B1A21D57BA8346A9BB84216CB6F592E3051 software-version= protocol-version=0 I[2021-06-11|07:07:37.742] ABCI Replay Blocks module=consensus appHeight=153975665 storeHeight=153975666 stateHeight=153975665 I[2021-06-11|07:07:37.742] Replay last block using real app module=consensus ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected E8090A2A1D20258FDB3C2EEE151C4B1A21D57BA8346A9BB84216CB6F592E3051, got 952153E05A393E5395B9638CDA3B018DC4FF7EA1F197E4523C38A551DF048803

Jun 11 '21 07:06 eloc3147

See: https://github.com/cosmos/cosmos-sdk/issues/10281

Nov 13 '21 15:11 kenorb