erigon
erigon copied to clipboard
erigontech
Crashing during snapshot download
System information
Erigon version: erigon/v2022.99.99-dev-348be135
OS & Version: Linux/Ubuntu
Commit hash :
Actual behaviour
Erigon crashing during the snapshot download using the latest devel docker image
It starts to download just fine
[INFO] [08-11|20:02:58.063] Starting Erigon on Ethereum mainnet...
[INFO] [08-11|20:02:58.064] Maximum peer count ETH=100 total=100
[INFO] [08-11|20:02:58.064] starting HTTP APIs APIs=eth,erigon,engine
[INFO] [08-11|20:02:58.064] torrent verbosity level=WRN
[INFO] [08-11|20:03:00.173] Set global gas cap cap=50000000
[INFO] [08-11|20:03:00.219] Opening Database label=chaindata path=/home/erigon/.local/share/erigon/chaindata
[INFO] [08-11|20:03:00.226] Re-Opening DB in exclusive mode to apply migrations
[INFO] [08-11|20:03:00.232] Apply migration name=db_schema_version5
[INFO] [08-11|20:03:00.233] Applied migration name=db_schema_version5
[INFO] [08-11|20:03:00.233] Apply migration name=txs_begin_end
[INFO] [08-11|20:03:00.234] Applied migration name=txs_begin_end
[INFO] [08-11|20:03:00.234] Apply migration name=reset_blocks_3
[INFO] [08-11|20:03:00.234] Applied migration name=reset_blocks_3
[INFO] [08-11|20:03:00.235] Updated DB schema to version=6.0.0
[INFO] [08-11|20:03:00.413] Writing custom genesis block hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [08-11|20:03:00.423] Initialised chain configuration config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [08-11|20:03:00.423] Effective prune_flags= snapshot_flags="--snapshots=true"
[INFO] [08-11|20:03:00.432] Initialising Ethereum protocol network=1
[INFO] [08-11|20:03:00.432] Disk storage enabled for ethash DAGs dir=/home/erigon/.local/share/erigon/ethash-dags count=2
[INFO] [08-11|20:03:00.440] Starting private RPC server on=0.0.0.0:9090
[INFO] [08-11|20:03:00.440] new subscription to logs established
[INFO] [08-11|20:03:00.440] rpc filters: subscribing to Erigon events
[INFO] [08-11|20:03:00.440] new subscription to newHeaders established
[INFO] [08-11|20:03:00.441] New txs subscriber joined
[INFO] [08-11|20:03:00.441] Reading JWT secret path=/home/erigon/.local/share/erigon/jwt.hex
[INFO] [08-11|20:03:00.441] Generated JWT secret path=/home/erigon/.local/share/erigon/jwt.hex
[INFO] [08-11|20:03:00.442] HTTP endpoint opened for Engine API url=localhost:8551 ws=true ws.compression=true
[INFO] [08-11|20:03:00.442] HTTP endpoint opened url=localhost:8545 ws=false ws.compression=true grpc=false
[INFO] [08-11|20:03:00.451] Started P2P networking version=66 self=enode://71d66c0260b1dad3173b8452fc17ac8382b977375d785e46d1c84de9deb98228f3a442b2fd97cf9860deb25229c5a4763155752a9662921f733f9a1cca553459@127.0.0.1:29234 name=erigon/v2022.99.99-dev-348be135/linux-amd64/go1.18.5
[INFO] [08-11|20:03:00.451] [txpool] Started
[INFO] [08-11|20:03:00.451] [Snapshots] Fetching torrent files metadata
[INFO] [08-11|20:03:20.465] [Snapshots] Waiting for torrents metadata: 23/90
[INFO] [08-11|20:03:40.496] [Snapshots] Waiting for torrents metadata: 52/90
[INFO] [08-11|20:04:00.442] [txpool] stat block=0 pending=0 baseFee=0 queued=4 alloc=134.9MB sys=352.4MB
[INFO] [08-11|20:04:00.475] [Snapshots] Waiting for torrents metadata: 72/90
[INFO] [08-11|20:04:20.458] [Snapshots] Waiting for torrents metadata: 84/90
[INFO] [08-11|20:04:40.459] [Snapshots] download progress="0.32% 740.4MB/226.2GB" download=28.2MB/s upload=0B/s peers=49 connections=187 files=90 alloc=208.5MB sys=468.2MB
[INFO] [08-11|20:05:00.432] [p2p] GoodPeers eth66=2
[INFO] [08-11|20:05:00.441] [txpool] stat block=0 pending=0 baseFee=0 queued=4 alloc=320.3MB sys=483.2MB
[INFO] [08-11|20:05:00.459] [Snapshots] download progress="0.80% 1.8GB/226.2GB" download=55.2MB/s upload=0B/s peers=55 connections=201 files=90 alloc=321.2MB sys=483.2MB
[INFO] [08-11|20:05:20.458] [Snapshots] download progress="1.32% 3.0GB/226.2GB" download=60.1MB/s upload=0B/s peers=55 connections=178 files=90 alloc=205.7MB sys=486.0MB
[INFO] [08-11|20:05:40.459] [Snapshots] download progress="1.86% 4.2GB/226.2GB" download=62.7MB/s upload=0B/s peers=33 connections=149 files=90 alloc=330.4MB sys=487.7MB
[INFO] [08-11|20:06:00.442] [txpool] stat block=0 pending=0 baseFee=0 queued=4 alloc=291.3MB sys=488.3MB
[INFO] [08-11|20:06:00.458] [Snapshots] download progress="2.42% 5.5GB/226.2GB" download=64.9MB/s upload=5.6KB/s peers=34 connections=115 files=90 alloc=291.5MB sys=488.3MB
[INFO] [08-11|20:06:20.459] [Snapshots] download progress="3.00% 6.8GB/226.2GB" download=67.0MB/s upload=366.4KB/s peers=33 connections=82 files=90 alloc=322.2MB sys=488.5MB
[INFO] [08-11|20:06:40.459] [Snapshots] download progress="3.60% 8.2GB/226.2GB" download=70.4MB/s upload=789.6KB/s peers=34 connections=54 files=90 alloc=251.6MB sys=492.5MB
[INFO] [08-11|20:07:00.432] [p2p] GoodPeers eth66=4
[INFO] [08-11|20:07:00.524] [txpool] stat block=0 pending=0 baseFee=0 queued=4 alloc=189.0MB sys=496.5MB
[INFO] [08-11|20:07:00.534] [Snapshots] download progress="4.25% 9.6GB/226.2GB" download=74.8MB/s upload=583.2KB/s peers=33 connections=36 files=90 alloc=189.1MB sys=496.5MB
Around 20% it crashes with the following errors
Backtrace
goroutine 662193 [chan receive]:
github.com/anacrolix/go-libutp.(*Conn).waitForConnect.func1()
github.com/anacrolix/[email protected]/conn.go:70 +0x32
created by github.com/anacrolix/go-libutp.(*Conn).waitForConnect
github.com/anacrolix/[email protected]/conn.go:69 +0xba
goroutine 663217 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00e368320, 0x2)
runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x1ffaf88?)
sync/cond.go:56 +0x8c
github.com/anacrolix/go-libutp.(*Conn).waitForConnect(0xc00e368300, {0x1ffaf88?, 0xc01446e040?})
github.com/anacrolix/[email protected]/conn.go:86 +0xd9
github.com/anacrolix/go-libutp.(*Socket).DialContext(0xc000f0cc60, {0x1ffaf88, 0xc01446e040}, {0xc0022e2438?, 0x0?}, {0xc007044000?, 0x0?})
github.com/anacrolix/[email protected]/socket.go:442 +0x2a5
github.com/anacrolix/torrent.utpSocketSocket.Dial({{0x2006c70?, 0xc000f0cc60?}, {0xc0022e2438?, 0x0?}}, {0x1ffaf88, 0xc01446e040}, {0xc007044000, 0x12})
github.com/anacrolix/[email protected]/socket.go:124 +0x108
github.com/anacrolix/torrent.dialFromSocket({0x1ffaf88?, 0xc01446e040?}, {0x7f7195ab4990?, 0xc000ff6720?}, {0xc007044000?, 0x2d21a40?})
github.com/anacrolix/[email protected]/client.go:667 +0x46
github.com/anacrolix/torrent.DialFirst.func2()
github.com/anacrolix/[email protected]/client.go:639 +0x4b
created by github.com/anacrolix/torrent.DialFirst
github.com/anacrolix/[email protected]/client.go:637 +0x44e
goroutine 662046 [select]:
golang.org/x/time/rate.(*Limiter).wait(0xc0022a8780, {0x1ffafc0, 0xc00033a000}, 0x1, {0x14?, 0xc013e51df8?, 0x2d21140?}, 0x1dde6f0)
golang.org/x/[email protected]/rate/rate.go:270 +0x3f9
golang.org/x/time/rate.(*Limiter).WaitN(0xc013e51e40?, {0x1ffafc0, 0xc00033a000}, 0xc013e51e28?)
golang.org/x/[email protected]/rate/rate.go:233 +0x56
golang.org/x/time/rate.(*Limiter).Wait(...)
golang.org/x/[email protected]/rate/rate.go:218
github.com/anacrolix/torrent.(*Client).outgoingConnection(0xc0002fed80, 0x464e65?, {0x1fecb00?, 0xc01409a060}, {0x197556f, 0x2}, 0x0)
github.com/anacrolix/[email protected]/client.go:778 +0x9f
created by github.com/anacrolix/torrent.(*Torrent).initiateConn
github.com/anacrolix/[email protected]/torrent.go:2269 +0x25d
goroutine 662334 [chan receive]:
github.com/anacrolix/go-libutp.(*Conn).waitForConnect.func1()
github.com/anacrolix/[email protected]/conn.go:70 +0x32
created by github.com/anacrolix/go-libutp.(*Conn).waitForConnect
github.com/anacrolix/[email protected]/conn.go:69 +0xba
goroutine 662760 [select]:
golang.org/x/time/rate.(*Limiter).wait(0xc0022a8780, {0x1ffafc0, 0xc00033a000}, 0x1, {0xc01a6925f8?, 0xc01a6925f8?, 0x2d21140?}, 0x1dde6f0)
golang.org/x/[email protected]/rate/rate.go:270 +0x3f9
golang.org/x/time/rate.(*Limiter).WaitN(0x7771949e8fc0?, {0x1ffafc0, 0xc00033a000}, 0x48c6ce?)
golang.org/x/[email protected]/rate/rate.go:233 +0x56
golang.org/x/time/rate.(*Limiter).Wait(...)
golang.org/x/[email protected]/rate/rate.go:218
github.com/anacrolix/torrent.(*Client).outgoingConnection(0xc0002fed80, 0x464e65?, {0x1fecb00?, 0xc011b7a640}, {0x197556f, 0x2}, 0x0)
github.com/anacrolix/[email protected]/client.go:778 +0x9f
created by github.com/anacrolix/torrent.(*Torrent).initiateConn
github.com/anacrolix/[email protected]/torrent.go:2269 +0x25d
[signal SIGBUS: bus error code=0x2 addr=0x71444aaaa020 pc=0x48fb6e]
goroutine 554966 [running]:
runtime.throw({0x1977b3d?, 0x19?})
runtime/panic.go:992 +0x71 fp=0xc0030c6c78 sp=0xc0030c6c48 pc=0x45b4b1
runtime.sigpanic()
runtime/signal_unix.go:815 +0x125 fp=0xc0030c6cc8 sp=0xc0030c6c78 pc=0x471645
runtime.memmove()
runtime/memmove_amd64.s:372 +0x42e fp=0xc0030c6cd0 sp=0xc0030c6cc8 pc=0x48fb6e
github.com/anacrolix/torrent/mmap_span.copyBytes(...)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:74
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).locateCopy.func1(0x4718eeecf848da92?, {0x477888?, 0x2331fcb?})
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:81 +0xae fp=0xc0030c6d50 sp=0xc0030c6cd0 pc=0xae116e
github.com/anacrolix/torrent/segments.Index.Locate.func2(0xc0030c6de0?, {0x479dfe?, 0xc0030c6db8?})
github.com/anacrolix/[email protected]/segments/index.go:43 +0x24 fp=0xc0030c6d78 sp=0xc0030c6d50 pc=0xadfc84
github.com/anacrolix/torrent/segments.Scan(0xc0030c6de8, {0xc0030c6e28?, 0x2982c10?}, 0xc0030c6e10)
github.com/anacrolix/[email protected]/segments/segments.go:43 +0xe9 fp=0xc0030c6dc8 sp=0xc0030c6d78 pc=0xadfea9
github.com/anacrolix/torrent/segments.Index.Locate({{0xc007f6f700, 0x1, 0x1}}, {0x27598000, 0x4000}, 0xc0030c6e88)
github.com/anacrolix/[email protected]/segments/index.go:42 +0x190 fp=0xc0030c6e50 sp=0xc0030c6dc8 pc=0xadfbf0
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).locateCopy(0xc014b6a780?, 0xc0030c6fb8?, {0xc001adc000, 0x4000, 0x4000}, 0xae3cf4?)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:78 +0x99 fp=0xc0030c6ec0 sp=0xc0030c6e50 pc=0xae1059
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).WriteAt(0x0?, {0xc001adc000?, 0x4000, 0xc01217a000?}, 0xc01831b6e0?)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:97 +0xe8 fp=0xc0030c6f40 sp=0xc0030c6ec0 pc=0xae13c8
github.com/anacrolix/missinggo/v2.(*SectionWriter).WriteAt(0x6?, {0xc001adc000?, 0xc01831b6e0?, 0x16b6160?}, 0xc01831b6e8?)
github.com/anacrolix/missinggo/[email protected]/section_writer.go:22 +0x56 fp=0xc0030c6f78 sp=0xc0030c6f40 pc=0xacf4f6
github.com/anacrolix/torrent/storage.(*mmapStoragePiece).WriteAt(0x16b6160?, {0xc001adc000?, 0xc0030c6fe8?, 0xc00048ef80?}, 0x13a?)
<autogenerated>:1 +0x30 fp=0xc0030c6fb0 sp=0xc0030c6f78 pc=0xae6db0
github.com/anacrolix/torrent/storage.Piece.WriteAt({{0x1ffeea0?, 0xc014b6a780?}, {0xc00048ef80?, 0x13a?}}, {0xc001adc000, 0x4000, 0x4000}, 0x198000)
github.com/anacrolix/[email protected]/storage/wrappers.go:67 +0x1ec fp=0xc0030c7078 sp=0xc0030c6fb0 pc=0xae560c
github.com/anacrolix/torrent.(*Torrent).writeChunk(0xc0010e8000, 0x13a, 0xc0030c7148?, {0xc001adc000, 0x4000, 0x4000})
github.com/anacrolix/[email protected]/torrent.go:917 +0xeb fp=0xc0030c7120 sp=0xc0030c7078 pc=0xd3498b
github.com/anacrolix/torrent.(*Peer).receiveChunk.func5(0xc0002fed80, 0xc010311650?, 0xc0167b5b90)
github.com/anacrolix/[email protected]/peerconn.go:1520 +0xee fp=0xc0030c71b0 sp=0xc0030c7120 pc=0xd1e58e
github.com/anacrolix/torrent.(*Peer).receiveChunk(0xc01628c000, 0xc0167b5b90)
github.com/anacrolix/[email protected]/peerconn.go:1521 +0x732 fp=0xc0030c74d0 sp=0xc0030c71b0 pc=0xd1df52
github.com/anacrolix/torrent.(*PeerConn).mainReadLoop(0xc01628c000)
github.com/anacrolix/[email protected]/peerconn.go:1255 +0x945 fp=0xc0030c79b8 sp=0xc0030c74d0 pc=0xd1baa5
github.com/anacrolix/torrent.(*Client).runHandshookConn(0xc0002fed80, 0xc01628c000, 0xc0010e8000)
github.com/anacrolix/[email protected]/client.go:994 +0x498 fp=0xc0030c7b60 sp=0xc0030c79b8 pc=0xd0a7d8
github.com/anacrolix/torrent.(*Torrent).logRunHandshookConn(0xc0010e8000, 0xc0010e8000?, 0x0, {0x15?})
github.com/anacrolix/[email protected]/torrent.go:1615 +0x52 fp=0xc0030c7e08 sp=0xc0030c7b60 pc=0xd39052
github.com/anacrolix/torrent.(*Torrent).runHandshookConnLoggingErr(...)
github.com/anacrolix/[email protected]/torrent.go:1622
github.com/anacrolix/torrent.(*Client).outgoingConnection(0xc0002fed80, 0xcfd0a0?, {0x1fecb00?, 0xc0044a91e0}, {0x197556f, 0x2}, 0x0)
github.com/anacrolix/[email protected]/client.go:797 +0x3bb fp=0xc0030c7f98 sp=0xc0030c7e08 pc=0xd08b3b
github.com/anacrolix/torrent.(*Torrent).initiateConn.func1()
github.com/anacrolix/[email protected]/torrent.go:2269 +0x3f fp=0xc0030c7fe0 sp=0xc0030c7f98 pc=0xd3f27f
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1 fp=0xc0030c7fe8 sp=0xc0030c7fe0 pc=0x48e8a1
created by github.com/anacrolix/torrent.(*Torrent).initiateConn
github.com/anacrolix/[email protected]/torrent.go:2269 +0x25d
unfortunately error reason and where it happend: is in the beginning of stack traces, but in issue shown only end of stack traces.
That still seems to be missing the original issue that caused the panic, but it might be an out-of-memory issue, I've seen something similar with snapshots on a computer with 8GB RAM, and I worked around it by lowering amount of workers that are downloading in parallel. For example in eth/stagedsync/stage_headers.go: if cfg.snapshots.Cfg().Produce { if !cfg.snapshots.SegmentsReady() { return fmt.Errorf("not all snapshot segments are available") }
// wait for Downloader service to download all expected snapshots
if cfg.snapshots.IndicesMax() < cfg.snapshots.SegmentsMax() {
chainID, _ := uint256.FromBig(cfg.chainConfig.ChainID)
workers := cmp.InRange(1, 2, runtime.GOMAXPROCS(-1)-1)
if err := snapshotsync.BuildMissedIndices(ctx, cfg.snapshots.Dir(), *chainID, cfg.tmpdir, workers, log.LvlInfo); err != nil {
return fmt.Errorf("BuildMissedIndices: %w", err)
}
}
Try changing workers variable to 1. If the issue goes away, you're running out of memory.
The issue is I am having a hard time capturing the error from the beginning because the log is being overwritten once the issue starts. Th is an extremely powerful server with over a TB a mem and the docker container currently is not set with a limit on resources.
[INFO] [08-12|06:20:53.783] Build info git_branch=alpha git_tag=v2021.10.03-1610-g7dabdc326-dirty git_commit=7dabdc3269f1e3b098fee5ab4d96ec843823a6fd
[INFO] [08-12|06:20:53.784] Starting Erigon on Ethereum mainnet...
[INFO] [08-12|06:20:53.785] Maximum peer count ETH=100 total=100
[INFO] [08-12|06:20:53.785] starting HTTP APIs APIs=eth,erigon,engine
[INFO] [08-12|06:20:53.785] torrent verbosity level=WRN
[INFO] [08-12|06:20:55.887] Set global gas cap cap=50000000
[INFO] [08-12|06:20:55.936] Opening Database label=chaindata path=/home/erigon/.local/share/erigon/chaindata
[INFO] [08-12|06:20:55.943] Re-Opening DB in exclusive mode to apply migrations
[INFO] [08-12|06:20:55.948] Apply migration name=db_schema_version5
[INFO] [08-12|06:20:55.949] Applied migration name=db_schema_version5
[INFO] [08-12|06:20:55.949] Apply migration name=txs_begin_end
[INFO] [08-12|06:20:55.950] Applied migration name=txs_begin_end
[INFO] [08-12|06:20:55.950] Apply migration name=reset_blocks_3
[INFO] [08-12|06:20:55.951] Applied migration name=reset_blocks_3
[INFO] [08-12|06:20:55.951] Updated DB schema to version=6.0.0
[INFO] [08-12|06:20:56.125] Writing custom genesis block hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [08-12|06:20:56.136] Initialised chain configuration config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [08-12|06:20:56.139] Effective prune_flags= snapshot_flags="--snapshots=true"
[INFO] [08-12|06:20:56.140] Using snapshots on=true
[INFO] [08-12|06:20:56.148] Initialising Ethereum protocol network=1
[INFO] [08-12|06:20:56.148] Disk storage enabled for ethash DAGs dir=/home/erigon/.local/share/erigon/ethash-dags count=2
[INFO] [08-12|06:20:56.156] Starting private RPC server on=0.0.0.0:9090
[INFO] [08-12|06:20:56.156] new subscription to logs established
[INFO] [08-12|06:20:56.156] rpc filters: subscribing to Erigon events
[INFO] [08-12|06:20:56.156] New txs subscriber joined
[INFO] [08-12|06:20:56.157] new subscription to newHeaders established
[INFO] [08-12|06:20:56.157] Reading JWT secret path=/home/erigon/.local/share/erigon/jwt.hex
[INFO] [08-12|06:20:56.157] Generated JWT secret path=/home/erigon/.local/share/erigon/jwt.hex
[INFO] [08-12|06:20:56.159] HTTP endpoint opened for Engine API url=localhost:8551 ws=true ws.compression=true
[INFO] [08-12|06:20:56.159] HTTP endpoint opened url=localhost:8545 ws=false ws.compression=true grpc=false
[INFO] [08-12|06:20:56.167] [txpool] Started
[INFO] [08-12|06:20:56.167] [Snapshots] Fetching torrent files metadata
[INFO] [08-12|06:20:56.170] Started P2P networking version=66 self=enode://8a04daf9effc29fce7886d5cdc4220d832fe1e41bfb519a41b7891a775da1e32f9da4275c5945f3c6bdc721da666f3fdddb6b44cee7ac14fe8a0fe6f807f2ec3@127.0.0.1:29234 name=erigon/v2022.08.1-alpha-7dabdc32/linux-amd64/go1.18.5
[INFO] [08-12|06:21:16.176] [Snapshots] Waiting for torrents metadata: 21/90
[INFO] [08-12|06:21:36.180] [Snapshots] Waiting for torrents metadata: 50/90
[INFO] [08-12|06:21:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=247.2MB sys=407.2MB
[INFO] [08-12|06:21:56.297] [Snapshots] Waiting for torrents metadata: 73/90
[INFO] [08-12|06:22:16.233] [Snapshots] Waiting for torrents metadata: 85/90
[INFO] [08-12|06:22:36.398] [Snapshots] Waiting for torrents metadata: 89/90
[INFO] [08-12|06:22:56.148] [p2p] GoodPeers eth66=3
[INFO] [08-12|06:22:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=358.5MB sys=533.9MB
[INFO] [08-12|06:22:56.176] [Snapshots] download progress="0.67% 1.5GB/226.2GB" download=44.8MB/s upload=0B/s peers=56 connections=198 files=90 alloc=359.8MB sys=533.9MB
[INFO] [08-12|06:23:16.177] [Snapshots] download progress="1.23% 2.8GB/226.2GB" download=65.4MB/s upload=0B/s peers=58 connections=215 files=90 alloc=289.8MB sys=565.5MB
[INFO] [08-12|06:23:36.179] [Snapshots] download progress="1.87% 4.2GB/226.2GB" download=73.6MB/s upload=0B/s peers=39 connections=168 files=90 alloc=346.9MB sys=568.6MB
[INFO] [08-12|06:23:56.188] [Snapshots] download progress="2.58% 5.8GB/226.2GB" download=83.1MB/s upload=0B/s peers=44 connections=142 files=90 alloc=387.7MB sys=573.5MB
[INFO] [08-12|06:23:56.161] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=387.3MB sys=573.5MB
[INFO] [08-12|06:24:16.177] [Snapshots] download progress="3.29% 7.4GB/226.2GB" download=81.5MB/s upload=0B/s peers=48 connections=112 files=90 alloc=276.4MB sys=574.3MB
[INFO] [08-12|06:24:36.179] [Snapshots] download progress="3.94% 8.9GB/226.2GB" download=75.4MB/s upload=0B/s peers=52 connections=83 files=90 alloc=348.7MB sys=578.5MB
[INFO] [08-12|06:24:56.148] [p2p] GoodPeers eth66=6
[INFO] [08-12|06:24:56.158] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=270.0MB sys=578.8MB
[INFO] [08-12|06:24:56.176] [Snapshots] download progress="4.66% 10.6GB/226.2GB" download=84.1MB/s upload=0B/s peers=45 connections=54 files=90 alloc=270.4MB sys=578.8MB
[INFO] [08-12|06:25:16.176] [Snapshots] download progress="5.44% 12.3GB/226.2GB" download=90.4MB/s upload=0B/s peers=43 connections=52 files=90 alloc=284.0MB sys=579.0MB
[INFO] [08-12|06:25:36.177] [Snapshots] download progress="6.18% 14.0GB/226.2GB" download=85.1MB/s upload=0B/s peers=44 connections=51 files=90 alloc=288.4MB sys=583.0MB
[INFO] [08-12|06:25:56.170] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=220.2MB sys=583.0MB
[INFO] [08-12|06:25:56.176] [Snapshots] download progress="6.82% 15.4GB/226.2GB" download=74.5MB/s upload=0B/s peers=44 connections=51 files=90 alloc=220.4MB sys=583.0MB
[INFO] [08-12|06:26:16.176] [Snapshots] download progress="7.56% 17.1GB/226.2GB" download=85.0MB/s upload=0B/s peers=43 connections=51 files=90 alloc=193.0MB sys=583.3MB
[WARN] [08-12|06:26:34.016] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2279 after failed piece check
[WARN] [08-12|06:26:34.656] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2275 after failed piece check
[WARN] [08-12|06:26:34.750] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2281 after failed piece check
[WARN] [08-12|06:26:35.110] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2284 after failed piece check
[WARN] [08-12|06:26:35.578] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2285 after failed piece check
[WARN] [08-12|06:26:35.978] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2282 after failed piece check
[INFO] [08-12|06:26:36.177] [Snapshots] download progress="8.33% 18.9GB/226.2GB" download=90.7MB/s upload=0B/s peers=39 connections=51 files=90 alloc=325.9MB sys=583.3MB
[WARN] [08-12|06:26:36.217] "v1-014000-014500-transactions.seg": would have banned 87.149.247.43 for touching piece 2297 after failed piece check
[INFO] [08-12|06:26:56.148] [p2p] GoodPeers eth66=10
[INFO] [08-12|06:26:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=309.6MB sys=583.5MB
[INFO] [08-12|06:26:56.176] [Snapshots] download progress="9.10% 20.6GB/226.2GB" download=88.3MB/s upload=0B/s peers=38 connections=52 files=90 alloc=310.1MB sys=583.5MB
[INFO] [08-12|06:27:16.177] [Snapshots] download progress="9.87% 22.3GB/226.2GB" download=89.8MB/s upload=0B/s peers=42 connections=53 files=90 alloc=312.5MB sys=583.5MB
[INFO] [08-12|06:27:36.180] [Snapshots] download progress="10.66% 24.1GB/226.2GB" download=91.6MB/s upload=0B/s peers=47 connections=54 files=90 alloc=299.1MB sys=583.5MB
[INFO] [08-12|06:27:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=236.4MB sys=584.3MB
[INFO] [08-12|06:27:56.176] [Snapshots] download progress="11.44% 25.9GB/226.2GB" download=90.1MB/s upload=0B/s peers=48 connections=54 files=90 alloc=237.1MB sys=584.3MB
[INFO] [08-12|06:28:16.177] [Snapshots] download progress="12.22% 27.7GB/226.2GB" download=90.9MB/s upload=0B/s peers=40 connections=43 files=90 alloc=198.2MB sys=584.3MB
[INFO] [08-12|06:28:36.184] [Snapshots] download progress="12.88% 29.1GB/226.2GB" download=75.5MB/s upload=0B/s peers=33 connections=35 files=90 alloc=275.0MB sys=584.3MB
[INFO] [08-12|06:28:56.148] [p2p] GoodPeers eth66=13
[INFO] [08-12|06:28:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=326.4MB sys=584.5MB
[INFO] [08-12|06:28:56.176] [Snapshots] download progress="13.50% 30.5GB/226.2GB" download=72.0MB/s upload=0B/s peers=32 connections=34 files=90 alloc=326.7MB sys=584.5MB
[INFO] [08-12|06:29:16.177] [Snapshots] download progress="14.15% 32.0GB/226.2GB" download=75.8MB/s upload=0B/s peers=34 connections=35 files=90 alloc=215.5MB sys=584.5MB
[INFO] [08-12|06:29:36.176] [Snapshots] download progress="14.77% 33.4GB/226.2GB" download=72.2MB/s upload=0B/s peers=33 connections=35 files=90 alloc=263.5MB sys=585.0MB
[INFO] [08-12|06:29:56.159] [txpool] stat block=0 pending=0 baseFee=0 queued=0 alloc=268.4MB sys=585.0MB
[INFO] [08-12|06:29:56.197] [Snapshots] download progress="15.40% 34.8GB/226.2GB" download=72.6MB/s upload=0B/s peers=34 connections=37 files=90 alloc=268.7MB sys=585.0MB
[INFO] [08-12|06:30:16.177] [Snapshots] download progress="15.95% 36.1GB/226.2GB" download=63.5MB/s upload=0B/s peers=26 connections=27 files=90 alloc=229.9MB sys=585.0MB
[INFO] [08-12|06:30:36.176] [Snapshots] download progress="16.50% 37.3GB/226.2GB" download=63.8MB/s upload=0B/s peers=27 connections=27 files=90 alloc=188.8MB sys=585.0MB
[INFO] [08-12|06:30:56.148] [p2p] GoodPeers eth66=20
[INFO] [08-12|06:30:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=6 alloc=305.3MB sys=607.1MB
[INFO] [08-12|06:30:56.176] [Snapshots] download progress="17.17% 38.8GB/226.2GB" download=77.7MB/s upload=0B/s peers=29 connections=30 files=90 alloc=306.0MB sys=607.1MB
[INFO] [08-12|06:31:16.176] [Snapshots] download progress="17.89% 40.5GB/226.2GB" download=82.9MB/s upload=0B/s peers=31 connections=32 files=90 alloc=267.4MB sys=607.1MB
[INFO] [08-12|06:31:36.176] [Snapshots] download progress="18.61% 42.1GB/226.2GB" download=83.8MB/s upload=0B/s peers=27 connections=28 files=90 alloc=304.9MB sys=607.1MB
[INFO] [08-12|06:31:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=6 alloc=223.7MB sys=607.1MB
[INFO] [08-12|06:31:56.177] [Snapshots] download progress="19.32% 43.7GB/226.2GB" download=82.4MB/s upload=0B/s peers=29 connections=30 files=90 alloc=224.0MB sys=607.1MB
[INFO] [08-12|06:32:16.176] [Snapshots] download progress="20.02% 45.3GB/226.2GB" download=80.8MB/s upload=0B/s peers=29 connections=31 files=90 alloc=303.5MB sys=607.1MB
[INFO] [08-12|06:32:36.176] [Snapshots] download progress="20.69% 46.8GB/226.2GB" download=77.7MB/s upload=0B/s peers=28 connections=30 files=90 alloc=309.4MB sys=607.4MB
[INFO] [08-12|06:32:56.153] [p2p] GoodPeers eth66=22
[INFO] [08-12|06:32:56.157] [txpool] stat block=0 pending=0 baseFee=0 queued=6 alloc=237.2MB sys=607.4MB
[INFO] [08-12|06:32:56.176] [Snapshots] download progress="21.34% 48.3GB/226.2GB" download=75.7MB/s upload=0B/s peers=30 connections=34 files=90 alloc=237.7MB sys=607.4MB
[INFO] [08-12|06:33:16.176] [Snapshots] download progress="22.00% 49.8GB/226.2GB" download=76.0MB/s upload=0B/s peers=34 connections=37 files=90 alloc=336.1MB sys=607.4MB
unexpected fault address 0x70feecf95000
fatal error: fault
unexpected fault address 0x70f78434d020
fatal error: fault
unexpected fault address 0x70fb965cd020
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x70feecf95000 pc=0x48fb7c]
goroutine 598632 [running]:
runtime.throw({0x193292a?, 0xfa6e81e3bdd1b550?})
runtime/panic.go:992 +0x71 fp=0xc00545ac78 sp=0xc00545ac48 pc=0x45b4b1
runtime.sigpanic()
runtime/signal_unix.go:815 +0x125 fp=0xc00545acc8 sp=0xc00545ac78 pc=0x471645
runtime.memmove()
runtime/memmove_amd64.s:375 +0x43c fp=0xc00545acd0 sp=0xc00545acc8 pc=0x48fb7c
github.com/anacrolix/torrent/mmap_span.copyBytes(...)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:74
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).locateCopy.func1(0x235942e272c3a4e5?, {0x477888?, 0x22dc7c2?})
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:81 +0xae fp=0xc00545ad50 sp=0xc00545acd0 pc=0xae08ae
github.com/anacrolix/torrent/segments.Index.Locate.func2(0xc00545ade0?, {0x479dfe?, 0xc00545adb8?})
github.com/anacrolix/[email protected]/segments/index.go:43 +0x24 fp=0xc00545ad78 sp=0xc00545ad50 pc=0xadf3c4
github.com/anacrolix/torrent/segments.Scan(0xc00545ade8, {0xc00545ae28?, 0x2922e68?}, 0xc00545ae10)
github.com/anacrolix/[email protected]/segments/segments.go:43 +0xe9 fp=0xc00545adc8 sp=0xc00545ad78 pc=0xadf5e9
github.com/anacrolix/torrent/segments.Index.Locate({{0xc007ae7150, 0x1, 0x1}}, {0xaea3c000, 0x4000}, 0xc00545ae88)
github.com/anacrolix/[email protected]/segments/index.go:42 +0x190 fp=0xc00545ae50 sp=0xc00545adc8 pc=0xadf330
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).locateCopy(0xc0090e5560?, 0xc00545afb8?, {0xc018a2a000, 0x4000, 0x4000}, 0xae3434?)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:78 +0x99 fp=0xc00545aec0 sp=0xc00545ae50 pc=0xae0799
github.com/anacrolix/torrent/mmap_span.(*MMapSpan).WriteAt(0x0?, {0xc018a2a000?, 0x4000, 0xc000301800?}, 0xc0133a94e0?)
github.com/anacrolix/[email protected]/mmap_span/mmap_span.go:97 +0xe8 fp=0xc00545af40 sp=0xc00545aec0 pc=0xae0b08
github.com/anacrolix/missinggo/v2.(*SectionWriter).WriteAt(0x6?, {0xc018a2a000?, 0xc0133a94e0?, 0x1679ce0?}, 0xc0133a94e8?)
github.com/anacrolix/missinggo/[email protected]/section_writer.go:22 +0x56 fp=0xc00545af78 sp=0xc00545af40 pc=0xacec36
github.com/anacrolix/torrent/storage.(*mmapStoragePiece).WriteAt(0x1679ce0?, {0xc018a2a000?, 0xc00545afe8?, 0xc00113b080?}, 0x575?)
<autogenerated>:1 +0x30 fp=0xc00545afb0 sp=0xc00545af78 pc=0xae64f0
github.com/anacrolix/torrent/storage.Piece.WriteAt({{0x1fb2320?, 0xc0090e5560?}, {0xc00113b080?, 0x575?}}, {0xc018a2a000, 0x4000, 0x4000}, 0x3c000)
github.com/anacrolix/[email protected]/storage/wrappers.go:67 +0x1ec fp=0xc00545b078 sp=0xc00545afb0 pc=0xae4d4c
github.com/anacrolix/torrent.(*Torrent).writeChunk(0xc000fbf500, 0x575, 0xc00545b148?, {0xc018a2a000, 0x4000, 0x4000})
github.com/anacrolix/[email protected]/torrent.go:917 +0xeb fp=0xc00545b120 sp=0xc00545b078 pc=0xd340cb
github.com/anacrolix/torrent.(*Peer).receiveChunk.func5(0xc00054ed80, 0xc0090271d0?, 0xc017993960)
github.com/anacrolix/[email protected]/peerconn.go:1520 +0xee fp=0xc00545b1b0 sp=0xc00545b120 pc=0xd1dcce
github.com/anacrolix/torrent.(*Peer).receiveChunk(0xc0149da900, 0xc017993960)
github.com/anacrolix/[email protected]/peerconn.go:1521 +0x732 fp=0xc00545b4d0 sp=0xc00545b1b0 pc=0xd1d692
github.com/anacrolix/torrent.(*PeerConn).mainReadLoop(0xc0149da900)
github.com/anacrolix/[email protected]/peerconn.go:1255 +0x945 fp=0xc00545b9b8 sp=0xc00545b4d0 pc=0xd1b1e5
github.com/anacrolix/torrent.(*Client).runHandshookConn(0xc00054ed80, 0xc0149da900, 0xc000fbf500)
github.com/anacrolix/[email protected]/client.go:994 +0x498 fp=0xc00545bb60 sp=0xc00545b9b8 pc=0xd09f18
github.com/anacrolix/torrent.(*Torrent).logRunHandshookConn(0xc000fbf500, 0xc000fbf500?, 0x0, {0x13?})
github.com/anacrolix/[email protected]/torrent.go:1615 +0x52 fp=0xc00545be08 sp=0xc00545bb60 pc=0xd38792
github.com/anacrolix/torrent.(*Torrent).runHandshookConnLoggingErr(...)
github.com/anacrolix/[email protected]/torrent.go:1622
github.com/anacrolix/torrent.(*Client).outgoingConnection(0xc00054ed80, 0xcfc7e0?, {0x1fa0580?, 0xc001f45f80}, {0x1930370, 0x2}, 0x0)
github.com/anacrolix/[email protected]/client.go:797 +0x3bb fp=0xc00545bf98 sp=0xc00545be08 pc=0xd0827b
github.com/anacrolix/torrent.(*Torrent).initiateConn.func1()
github.com/anacrolix/[email protected]/torrent.go:2269 +0x3f fp=0xc00545bfe0 sp=0xc00545bf98 pc=0xd3e9bf
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1 fp=0xc00545bfe8 sp=0xc00545bfe0 pc=0x48e8a1
The same thing happens with the devel branch as well. Just using alpha right now to see if it makes a difference but clearly didn't.
Ah, interesting, that's a different issue than I've seen--I never had a SIGBUS. I thought I'd share my experience just in case it was a similar issue, but it sounds like it's not. Unfortunately, then, I can't provide any further assistance. Hopefully one of the core devs will be able to help, I'm just a fellow Erigon user :) Cheers!
Try start with —downloader.verify
Can you please elaborate on the usage? I don't see it anywhere in erigon --help
Is it --download.verify or —downloader.verify?
erigon --help | grep verify
Try start with —downloader.verify
Can you please elaborate on the usage? I don't see it anywhere in
erigon --helpIs it
--download.verifyor—downloader.verify?
- all flags are using double dash. but sometime people reply you from mobile phone and it replacing things.
erigon --help | grep verify
--miner.noverify Disable remote sealing verification
--downloader.verify verify snapshots on startup. it will not report founded problems but just re-download broken pieces
Yup I found it now. I did that already and it is still crashing. Perhaps I can share my docker file and you can tell me if you see anything out of the ordinary. Also can I disable the snapshot download all together.
version: '3.4'
services:
erigon:
image: thorax/erigon:devel
container_name: eth-mainnet
user: "${UID:-1000}:${GID:-1000}"
ports:
- 30303:30303
- 30303:30303/udp
- 30303:30303
- 42069:42069/udp
volumes:
- ./:/home/erigon/.local/share/erigon'
command:
- erigon
- --port=30303
- --private.api.addr=0.0.0.0:9090
- --torrent.port=42069
- --torrent.download.rate=100mb
- --torrent.download.slots=6
- --downloader.verify
ulimits:
nofile:
soft: 500000
hard: 500000
rpcdaemon:
image: thorax/erigon:devel
container_name: eth-mainnet_rpcdaemon
user: "${UID:-1000}:${GID:-1000}"
ports:
- '8545:8545'
volumes:
- ./:/home/erigon/.local/share/erigon
command:
- rpcdaemon
- --http.port=8545
- --ws
- --http.api="eth,debug,net,trace,web3"
- --http.addr=0.0.0.0
- --http.vhosts="*"
- --private.api.addr="eth-mainnet:9090"
30303:30303 - exposed twice
Yeah in my config it's commented out. I removed the commented while posting when I should have removed the line completely. I just deleted everything and starting again from scratch. I don't know what it could be but starting over sometime helps😏🤞🏽
You didn’t expose tcp 42069:42069
Can disable it by —snapshots=false (need re-create node), but likely sync will be slower.
Also i don’t know your host-machine info - maybe you have weird server or limits.
“I did that already and it is still crashing” - also better show logs instead of describing them - because maybe you just didn’t notice difference.
This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 7 days with no activity.