node icon indicating copy to clipboard operation
node copied to clipboard
verified publisher icon akash-network

Akash v0.12.1 Fails to Start with: unexpected fault address 0xd6f656627f092fec

Open nhandler opened this issue 5 years ago • 1 comments

Issue type

Bug

Akash version

v0.12.1

Akash token balance

N/A

What happened

I attempted to upgrade my validator by following https://docs.akash.network/guides/akashnet-2-software-upgrade , but the upgraded binary fails to start with the following output:

7:14PM INF starting ABCI with Tendermint
unexpected fault address 0xd6f656627f092fec
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xd6f656627f092fec pc=0xb6ea80]

goroutine 1 [running]:
runtime.throw(0x248f2b1, 0x5)
    runtime/panic.go:1117 +0x54 fp=0x4001b18100 sp=0x4001b180d0 pc=0x434c14
runtime.sigpanic()
    runtime/signal_unix.go:741 +0x230 fp=0x4001b18140 sp=0x4001b18100 pc=0x44c7c0
github.com/golang/snappy.encodeBlock(0x400051d002, 0x16e1, 0x16e1, 0x4001b10000, 0x1006, 0x1cda, 0x4001b20278)
    github.com/golang/[email protected]/encode_arm64.s:666 +0x360 fp=0x4001b201e0 sp=0x4001b18150 pc=0xb6ea80
github.com/golang/snappy.Encode(0x400051d000, 0x16e3, 0x16e3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4001b202f8)
    github.com/golang/[email protected]/encode.go:39 +0x17c fp=0x4001b20230 sp=0x4001b201e0 pc=0xb6dfbc
github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0x40005206c0, 0x4000520718, 0x2, 0x0, 0x28, 0x40, 0x4000057140)
    github.com/syndtr/[email protected]/leveldb/table/writer.go:171 +0xb0 fp=0x4001b202b0 sp=0x4001b20230 pc=0xb767a0
github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0x40005206c0, 0x4001e04850, 0x31)
    github.com/syndtr/[email protected]/leveldb/table/writer.go:222 +0x4c fp=0x4001b20300 sp=0x4001b202b0 pc=0xb76c3c
github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0x40005206c0, 0x4001e04850, 0x31, 0x17b7b0, 0x4001e04881, 0x0, 0x17b77f, 0x100, 0x1)
    github.com/syndtr/[email protected]/leveldb/table/writer.go:255 +0x1e4 fp=0x4001b20370 sp=0x4001b20300 pc=0xb76ea4
github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0x400057afc0, 0x4001e04850, 0x31, 0x17b7b0, 0x4001e04881, 0x0, 0x17b77f, 0x0, 0x0)
    github.com/syndtr/[email protected]/leveldb/table.go:559 +0xbc fp=0x4001b203e0 sp=0x4001b20370 pc=0xb98a0c
github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0x400053d620, 0x2a86458, 0x4001092000, 0x0, 0x0, 0x0, 0x0)
    github.com/syndtr/[email protected]/leveldb/table.go:397 +0x11c fp=0x4001b20480 sp=0x4001b203e0 pc=0xb97d8c
github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0x4000d56f00, 0x40010ca280, 0x40000e61c0, 0x0, 0x0, 0x0, 0x0)
    github.com/syndtr/[email protected]/leveldb/session_compaction.go:35 +0xa4 fp=0x4001b205e0 sp=0x4001b20480 pc=0xb90004
github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0x40000e81c0, 0x40000e81c0, 0x400052a900)
    github.com/syndtr/[email protected]/leveldb/db.go:612 +0xa80 fp=0x4001b20800 sp=0x4001b205e0 pc=0xb7bcd0
github.com/syndtr/goleveldb/leveldb.openDB(0x4000d56f00, 0x0, 0x0, 0x4000d56f00)
    github.com/syndtr/[email protected]/leveldb/db.go:131 +0x488 fp=0x4001b208f0 sp=0x4001b20800 pc=0xb7ad18
github.com/syndtr/goleveldb/leveldb.Open(0x2a82fa0, 0x4000fae230, 0x0, 0x0, 0x0, 0x0)
    github.com/syndtr/[email protected]/leveldb/db.go:203 +0xd0 fp=0x4001b20960 sp=0x4001b208f0 pc=0xb7af70
github.com/syndtr/goleveldb/leveldb.OpenFile(0x400120b3e0, 0x27, 0x0, 0x400120b3e0, 0x27, 0x4000b87f10)
    github.com/syndtr/[email protected]/leveldb/db.go:225 +0x7c fp=0x4001b209e0 sp=0x4001b20960 pc=0xb7b16c
github.com/tendermint/tm-db.NewGoLevelDBWithOpts(0x24985cf, 0xb, 0x40011736e0, 0x18, 0x0, 0xe959bf00, 0x400013e8f0, 0xadf3c8)
    github.com/tendermint/[email protected]/goleveldb.go:32 +0xa4 fp=0x4001b20a50 sp=0x4001b209e0 pc=0xba7ee4
github.com/tendermint/tm-db.NewGoLevelDB(...)
    github.com/tendermint/[email protected]/goleveldb.go:27
github.com/tendermint/tm-db.init.0.func1(0x24985cf, 0xb, 0x40011736e0, 0x18, 0x400013e978, 0x4001137801, 0x0, 0xadd75c)
    github.com/tendermint/[email protected]/goleveldb.go:15 +0x44 fp=0x4001b20aa0 sp=0x4001b20a50 pc=0xbae574
github.com/tendermint/tm-db.NewDB(0x24985cf, 0xb, 0x24956d0, 0x9, 0x40011736e0, 0x18, 0x40011736e0, 0x18, 0x40011736e0, 0x18)
    github.com/tendermint/[email protected]/db.go:64 +0x2b4 fp=0x4001b20bd0 sp=0x4001b20aa0 pc=0xba7cb4
github.com/cosmos/cosmos-sdk/types.NewLevelDB(0x24985cf, 0xb, 0x40011736e0, 0x18, 0x0, 0x0, 0x0, 0x0)
    github.com/cosmos/[email protected]/types/utils.go:95 +0x90 fp=0x4001b20c50 sp=0x4001b20bd0 pc=0xc617e0
github.com/cosmos/cosmos-sdk/server.openDB(0x40000eb5f0, 0x13, 0xb, 0x0, 0x0, 0x2097b80)
    github.com/cosmos/[email protected]/server/util.go:365 +0x88 fp=0x4001b20cc0 sp=0x4001b20c50 pc=0xf7d7b8
github.com/cosmos/cosmos-sdk/server.startInProcess(0x4000fff680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2a701f0, 0x4000ef9d60, ...)
    github.com/cosmos/[email protected]/server/start.go:226 +0xec fp=0x4001b218e0 sp=0x4001b20cc0 pc=0xf7a76c
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0x40010c42c0, 0x3d25540, 0x0, 0x0, 0x0, 0x0)
    github.com/cosmos/[email protected]/server/start.go:120 +0x144 fp=0x4001b21cf0 sp=0x4001b218e0 pc=0xf7e2e4
github.com/spf13/cobra.(*Command).execute(0x40010c42c0, 0x3d25540, 0x0, 0x0, 0x40010c42c0, 0x3d25540)
    github.com/spf13/[email protected]/command.go:850 +0x320 fp=0x4001b21db0 sp=0x4001b21cf0 pc=0x59c730
github.com/spf13/cobra.(*Command).ExecuteC(0x4000e4fb80, 0x248eb04, 0x5, 0x40000eb5f0)
    github.com/spf13/[email protected]/command.go:958 +0x268 fp=0x4001b21e90 sp=0x4001b21db0 pc=0x59cfc8
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/[email protected]/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(...)
    github.com/spf13/[email protected]/command.go:888
github.com/ovrclk/akash/cmd/akash/cmd.Execute(0x4000e4fb80, 0x2a759e8, 0x4000f43428)
    github.com/ovrclk/akash/cmd/akash/cmd/root.go:117 +0x230 fp=0x4001b21f20 sp=0x4001b21e90 pc=0x1ed4c90
main.main()
    github.com/ovrclk/akash/cmd/akash/main.go:14 +0x24 fp=0x4001b21f70 sp=0x4001b21f20 pc=0x1ed79d4
runtime.main()
    runtime/proc.go:225 +0x274 fp=0x4001b21fd0 sp=0x4001b21f70 pc=0x437164
runtime.goexit()
    runtime/asm_arm64.s:1130 +0x4 fp=0x4001b21fd0 sp=0x4001b21fd0 pc=0x46c0c4

goroutine 35 [IO wait]:
internal/poll.runtime_pollWait(0xffff7d5f4e88, 0x72, 0x1000)
    runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x4000028098, 0x72, 0x0, 0x10, 0x4000c94020)
    internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
    internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadMsg(0x4000028080, 0x40000277c0, 0x10, 0x10, 0x4000c94020, 0x1000, 0x1000, 0x0, 0x0, 0x0, ...)
    internal/poll/fd_unix.go:249 +0x19c
net.(*netFD).readMsg(0x4000028080, 0x40000277c0, 0x10, 0x10, 0x4000c94020, 0x1000, 0x1000, 0x2097b80, 0x40000b9c08, 0x2790308, ...)
    net/fd_posix.go:67 +0x5c
net.(*UnixConn).readMsg(0x4000b880c8, 0x40000277c0, 0x10, 0x10, 0x4000c94020, 0x1000, 0x1000, 0x0, 0x40000b9d48, 0x40cb00, ...)
    net/unixsock_posix.go:115 +0x5c
net.(*UnixConn).ReadMsgUnix(0x4000b880c8, 0x40000277c0, 0x10, 0x10, 0x4000c94020, 0x1000, 0x1000, 0x18, 0x18, 0x2049860, ...)
    net/unixsock.go:143 +0x60
github.com/godbus/dbus.(*oobReader).Read(0x4000c94000, 0x40000277c0, 0x10, 0x10, 0x40000b9de8, 0xc999ec, 0x18)
    github.com/godbus/[email protected]/transport_unix.go:21 +0x5c
io.ReadAtLeast(0x29ff808, 0x4000c94000, 0x40000277c0, 0x10, 0x10, 0x10, 0xc, 0x21, 0x40000b9e58)
    io/io.go:328 +0x78
io.ReadFull(...)
    io/io.go:347
github.com/godbus/dbus.(*unixTransport).ReadMessage(0x4000b92090, 0x4000c90750, 0x4, 0x4000ba8b01)
    github.com/godbus/[email protected]/transport_unix.go:91 +0xd8
github.com/godbus/dbus.(*Conn).inWorker(0x4000ba62d0)
    github.com/godbus/[email protected]/conn.go:294 +0x38
created by github.com/godbus/dbus.(*Conn).Auth
    github.com/godbus/[email protected]/auth.go:118 +0x4b4

goroutine 10 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x3cefa20)
    k8s.io/klog/[email protected]/klog.go:1131 +0x70
created by k8s.io/klog/v2.init.0
    k8s.io/klog/[email protected]/klog.go:416 +0x118

goroutine 22 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x40000e60e0)
    github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:209 +0xcc
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:240 +0x154

goroutine 23 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0x4000d56f00)
    github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x3b0
created by github.com/syndtr/goleveldb/leveldb.newSession
    github.com/syndtr/[email protected]/leveldb/session.go:93 +0x22c

What did you expect to happen

I expected akash to start back up on the upgraded version.

How to reproduce it

Environment

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

5.4.0-1041-aws #43-Ubuntu SMP Fri Mar 19 22:06:27 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Anything else we need to know?

This issue is happening with both the pre-built binary and when building from source.

Next steps

You may join developer chat for community support.

nhandler avatar Apr 10 '21 19:04 nhandler

@nhandler Do you have any additional information on your environment? Were you running your validator set to prune everything?

hydrogen18 avatar Jul 22 '21 14:07 hydrogen18