[Error] In uptime service, POST of uptime data was interrupted
Preliminary Checks
- [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
- [X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions
Description
Oct 28 07:03:00 ip-172-31-80-92 mina[5038]: 2021-10-28 07:03:00 UTC [Error] In uptime service, POST of uptime data was interrupted
with 10 lines of context, before/after
Oct 28 07:00:18 ip-172-31-80-92 mina[5038]: 2021-10-28 07:00:18 UTC [Warn] Not rebroadcasting block $state_hash because it was received "1 slots too late"
Oct 28 07:00:18 ip-172-31-80-92 mina[5038]: state_hash: "3NKa6bxozVtPgmCV7De2Q7G2LhBafego4roK2hNn7T4BKwmnSFMj"
Oct 28 07:00:18 ip-172-31-80-92 mina[5038]: 2021-10-28 07:00:18 UTC [Info] Saw block with state hash $state_hash
Oct 28 07:00:18 ip-172-31-80-92 mina[5038]: state_hash: "3NKa6bxozVtPgmCV7De2Q7G2LhBafego4roK2hNn7T4BKwmnSFMj"
Oct 28 07:02:12 ip-172-31-80-92 mina[5038]: 2021-10-28 07:02:12 UTC [Info] verifier successfully got sigkill
Oct 28 07:02:12 ip-172-31-80-92 mina[5038]: 2021-10-28 07:02:12 UTC [Info] verifier successfully stopped
Oct 28 07:02:12 ip-172-31-80-92 mina[5038]: 2021-10-28 07:02:12 UTC [Info] Starting a new verifier process
Oct 28 07:02:41 ip-172-31-80-92 mina[5038]: 2021-10-28 07:02:41 UTC [Info] Daemon started process of kind "Verifier" with pid 6520
Oct 28 07:03:00 ip-172-31-80-92 mina[5038]: 2021-10-28 07:03:00 UTC [Info] Uptime service will attempt to send a block and SNARK work
Oct 28 07:03:00 ip-172-31-80-92 mina[5038]: 2021-10-28 07:03:00 UTC [Error] In uptime service, POST of uptime data was interrupted
Oct 28 07:04:17 ip-172-31-80-92 mina[5038]: 2021-10-28 07:04:16 UTC [Error] Error when sending block with state hash $state_hash to uptime service at URL "http://34.134.227.208/v1/submit"
Oct 28 07:04:17 ip-172-31-80-92 mina[5038]: state_hash: "3NLeFvH8RqNM4ZqhJfYdm6r2Eh2M123zpdRnECHZYxjRitm9mqzm"
Oct 28 07:04:33 ip-172-31-80-92 mina[5038]: 2021-10-28 07:04:33 UTC [Warn] validation callback timed out before we could respond
Oct 28 07:04:33 ip-172-31-80-92 mina[5038]: 2021-10-28 07:04:33 UTC [Error] validation timed out :(
Oct 28 07:04:37 ip-172-31-80-92 mina[5038]: 2021-10-28 07:04:37 UTC [Warn] validation callback timed out before we could respond
Oct 28 07:04:37 ip-172-31-80-92 mina[5038]: 2021-10-28 07:04:37 UTC [Error] validation timed out :(
Oct 28 07:06:57 ip-172-31-80-92 mina[5038]: 2021-10-28 07:06:57 UTC [Info] Received a block from $sender
Oct 28 07:06:57 ip-172-31-80-92 mina[5038]: sender: {
Oct 28 07:06:57 ip-172-31-80-92 mina[5038]: "Remote": {
Oct 28 07:06:57 ip-172-31-80-92 mina[5038]: "host": "142.93.100.73",
Oct 28 07:06:57 ip-172-31-80-92 mina[5038]: "peer_id": "12D3KooWPmjJGp9HEk9uyEJ9RyrLpgGQE2QrQj23gRGdU1EQYdDk",
log export in https://s3.amazonaws.com/mina.crowbar.systems/mainnet/export/2021-10-28_07-18-20.tar.gz
Steps to Reproduce
- Running 1.2.0 on Ubuntu
- Uptime service set to http://34.134.227.208/v1/submit
Expected Result
Not having this kind of entry in the log :)
Actual Result
Having this kind of entry in the log
How frequently do you see this issue?
Sometimes
What is the impact of this issue on your ability to run a node?
Blocker
Status
Mina daemon status
-----------------------------------
Global number of accounts: 68555
Block height: 75165
Max observed block height: 75165
Max observed unvalidated block height: 75165
Local uptime: 6h6m9s
Ledger Merkle root: jwJnd2gh8nj2pr9xtZAz3Ty7zWFTrappGuzcAaDKe2Mwh3LagZD
Protocol state hash: 3NLfVau9AYWhJ3qmLJQYcPu5TFZUvMXS73AozLzfV3Ey9YxSR1vR
Chain id: 5f704cc0c82e0ed70e873f0893d7e06f148524e3f0bdae2afb02e7819a0c24d1
Git SHA-1: fe51f1ef5cfbcb4d5f559af39080963cf28ff1af
Configuration directory: /home/ubuntu/.mina-config
Peers: 54
User_commands sent: 0
SNARK worker: None
SNARK work fee: 100000000
Sync status: Synced
Catchup status:
To build breadcrumb: 0
To initial validate: 0
Finished: 304
To verify: 0
To download: 0
Waiting for parent to finish: 0
Block producers running: 1 (B62qrcQDeAfg2qSnZhdF8wQ78G9QmKNDqpESVt2SL43F97kF3Cyv32a)
Coinbase receiver: Block producer
Best tip consensus time: epoch=15, slot=1046
Best tip global slot (across all hard-forks): 108146
Next block will be produced in: in 13.674h for slot: 108420 slot-since-genesis: 108420 (Generated from consensus at slot: 108064 slot-since-genesis: 108064)
Consensus time now: epoch=15, slot=1046
Consensus mechanism: proof_of_stake
Consensus configuration:
Delta: 0
k: 290
Slots per epoch: 7140
Slot duration: 3m
Epoch duration: 14d21h
Chain start timestamp: 2021-03-17 00:00:00.000000Z
Acceptable network delay: 3m
Addresses and ports:
External IP: 3.84.79.225
Bind IP: 0.0.0.0
Libp2p PeerID: 12D3KooWQBiq7tzFZ2uScLDrXvsk2ecTzWp3Kj7pFvQ5YKLXxPhT
Libp2p port: 8302
Client port: 8301
Additional information
You can get in touch with me on Discord.
Just adding a comment since no activity here - want to register that I also see these daily -- 2 - 4 errors every day.
@hsq125 @jrwashburn DO you still experience same issue?
Closing this issue because of being inactive for long time. Please feel free to open another ticket should you experience any further issues.