[BUG]: Panic from a lot of DELs and SETs with very large keys

[nathanhack]

What version of Badger are you using?

v4.2.0 But it also happened with v3.2103.2.

What version of Go are you using?

1.21.1

Have you tried reproducing the issue with the latest release?

None

What is the hardware spec (RAM, CPU, OS)?

128GiB, AMD Ryzen™ 9 5950X × 32, Fedora Linux 38 (Workstation Edition)

What steps will reproduce the bug?

running a bunch (several 100k) of DELs followed by SETs with large very large keys.

Expected behavior and actual result.

Expected: no panic.

Actual: a panic

panic: cannot grow buffer beyond 2 gigabytes

goroutine 1424649 [running]:
github.com/google/flatbuffers/go.(*Builder).growByteBuffer(...)
	/homeDIR/go/pkg/mod/github.com/google/[email protected]+incompatible/go/builder.go:216
github.com/google/flatbuffers/go.(*Builder).Prep(0xc152a58c80, 0x4, 0x5b48)
	/homeDIR/go/pkg/mod/github.com/google/[email protected]+incompatible/go/builder.go:270 +0x2b8
github.com/google/flatbuffers/go.(*Builder).CreateByteVector(0xc152a58c80, {0xc13207a000, 0x5b48, 0xc152a58b60?})
	/homeDIR/go/pkg/mod/github.com/google/[email protected]+incompatible/go/builder.go:390 +0x45
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffset(0xc07bb0e000?, 0xc152a58c80, 0xc0202ec410, 0x47765e50?)
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/builder.go:591 +0x3d
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffsets(0xc001a15b90, 0x43f38e?)
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/builder.go:579 +0xc5
github.com/dgraph-io/badger/v4/table.(*Builder).buildIndex(0xc001a15b90, {0xc001574000, 0xfbe5, 0x1156a})
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/builder.go:540 +0x106
github.com/dgraph-io/badger/v4/table.(*Builder).Done(0xc001a15b90)
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/builder.go:450 +0x145
github.com/dgraph-io/badger/v4/table.CreateTable({0xc0791fa030, 0x16}, 0xc001a15b90)
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:258 +0x3d
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func4(0xc001a15b90, 0xc001a14090?)
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:846 +0x109
created by github.com/dgraph-io/badger/v4.(*levelsController).subcompact in goroutine 1418647
	/homeDIR/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:836 +0x68f

Additional information

No response