tikv
Write meta data to tikv failed from juicefs
Bug Report
What did you do?
I installed tikv by binary manually according to the community documentation.
When I test using juicefs+tikv+oss, the metadata is written from juicefs to tikv. Writing to tikv fails when benchmark's threads are 200. The relevant error is
What did you expect to see?
I want to know what causes the error，how should i fix it.
What did you see instead?
What version of PD are you using (pd-server -V)?
tidb-v5.0.1-linux-amd64
write error
0\000\000\000\000\000I" end_key:"0904\375A\254Z\377\000\000\000\000\000\000C\000\377\000\000\002\000\000\000\000\000\372" region_epoch:<conf_ver:5 version:27 > peers:<id:21 store_id:1 > peers:<id:22 store_id:4 > peers:<id:23 store_id:5 > , invalid marker byte, group bytes "0904\xfdAa|\x00" goroutine 922 [running, locked to thread]: runtime/debug.Stack() /home/work/go/src/runtime/debug/stack.go:24 +0x65 github.com/juicedata/juicefs/pkg/meta.errno({0x7fdc5dd722a0, 0xc001b98210}) /home/work/juicefs/pkg/meta/utils.go:122 +0xad github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doLookup(0xc00031ee80, {0xc00236c330?, 0x7c61?}, 0xc005407320?, {0xc006ae2a39, 0x4}, 0xc002b23100, 0xc007170190) /home/work/juicefs/pkg/meta/tkv.go:839 +0x3a8 github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Lookup(0xc002400000, {0x7fdc5dda4f48, 0xc0059d5bc0}, 0x7c61, {0xc006ae2a39, 0x4}, 0xc002b23100, 0xc007170190, 0x0) /home/work/juicefs/pkg/meta/base.go:1114 +0x4b7 github.com/juicedata/juicefs/pkg/fs.(*FileSystem).lookup(0xc001a3a000, {0x7fdc5dda4f48, 0xc0059d5bc0}, 0x1?, {0xc006ae2a39, 0x4}, 0xc002b23100, 0xc007170190) /home/work/juicefs/pkg/fs/fs.go:660 +0x4db github.com/juicedata/juicefs/pkg/fs.(*FileSystem).doResolve(0xc001a3a000, {0x7fdc5dda4f48, 0xc0059d5bc0}, {0xc006ae2a20, 0x21}, 0x1, 0x0) /home/work/juicefs/pkg/fs/fs.go:734 +0x40d github.com/juicedata/juicefs/pkg/fs.(*FileSystem).resolve(...) /home/work/juicefs/pkg/fs/fs.go:684 github.com/juicedata/juicefs/pkg/fs.(*FileSystem).Create(0xc001a3a000, {0x7fdc5dda4f48?, 0xc0059d5bc0}, {0xc006ae2a20, 0x2a}, 0x1b6, 0x3e3?) /home/work/juicefs/pkg/fs/fs.go:786 +0x2e7 main.jfs_create(0x7fdc5a86f7f8?, 0xc000068060?, 0x0?, 0xd501?, 0x1a7?) /home/work/juicefs/sdk/java/libjfs/main.go:741 +0xb0 [utils.go:122]
read error
2023-09-05 19:13:07,337 ERROR juicefs.JuiceFileSystemImpl: 2023/09/05 19:13:07.337504 juicefs[154644] <ERROR>: error: failed to decode region range key, key: "30393034FD41010000000000000044", err: invalid marker byte, group bytes "0904\xfdAa|\x00" goroutine 17 [running, locked to thread]: runtime/debug.Stack() /home/work/go/src/runtime/debug/stack.go:24 +0x65 github.com/juicedata/juicefs/pkg/meta.errno({0x7fa3874f02a0, 0xc001b9a678}) /home/work/juicefs/pkg/meta/utils.go:122 +0xad github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doReaddir(0xc002590060, {0x7fa387522f48?, 0xc001544680?}, 0x59?, 0x1, 0xc0010b4558, 0x7fa386edda20?) /home/work/juicefs/pkg/meta/tkv.go:1781 +0x29c github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Readdir(0xc0025aa000, {0x7fa387522f48, 0xc001544680}, 0x1, 0x1, 0xc0010b4558) /home/work/juicefs/pkg/meta/base.go:1759 +0x484 github.com/juicedata/juicefs/pkg/fs.(*File).ReaddirPlus(0xc000f02b40, {0x7fa387522f48?, 0xc001544680}, 0x0) /home/work/juicefs/pkg/fs/fs.go:1105 +0x289 main.jfs_listdir(0xc000943e28?, 0xc000943e28?, 0x7fa383fed7f8?, 0x0, 0x7fa3fdb48880, 0x8000) /home/work/juicefs/sdk/java/libjfs/main.go:1054 +0x1f4 [utils.go:122]
FYI, TiDB writes to TiKV with encode, maybe this is an entry point :) https://github.com/tikv/client-go/blob/master/internal/locate/pd_codec.go#L56
I'm going to change the label to question type. If there is no more feedback, this issue will be closed in the next few days.
