foundationdb
foundationdb copied to clipboard
apple
Simulated correctness crash in Redwood
A recent correctness test turned up a crash in Redwood that may also be related to change feeds. The test is reproduced on 999097efb193d8f5f9fc72a1c77ab2a47f3953db with the following parameters:
bin/fdbserver -r simulation -f tests/slow/BlobGranuleVerifyBalance.toml -s 474988633 -b on --crash --trace_format json
It produces a crash immediately after processing a private mutation on the storage server that logs DestroyingChangeFeed. The crash gives the following backtrace:
ObjectCache<unsigned int, DWALPager::PageCacheEntry>::get(unsigned int const&, int, bool) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:2025
Future<Reference<ArenaPage> >::isValid() const at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:796
(inlined by) DWALPager::PageCacheEntry::initialized() const at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:2100
(inlined by) DWALPager::readPage(PagerEventReasons, unsigned int, unsigned int, int, bool, bool) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:3119
DWALPager::readPageAtVersion(PagerEventReasons, unsigned int, unsigned int, int, long, bool, bool) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:3220
(inlined by) DWALPagerSnapshot::getPhysicalPage(PagerEventReasons, unsigned int, unsigned int, int, bool, bool) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:4051
VersionedBTree::ReadPageActorState<VersionedBTree::ReadPageActor>::a_body1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:6004
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:872
(inlined by) VersionedBTree::readPage(VersionedBTree* const&, PagerEventReasons const&, unsigned int const&, IPagerSnapshot* const&, VectorRef<unsigned int, (VecSerStrategy)0> const&, int const&, bool const&, bool const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:5988
VersionedBTree::BTreeCursor::pushPage(DeltaTree2<RedwoodRecordRef, RedwoodRecordRef::Delta>::Cursor const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7411
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:807
(inlined by) StrictFuture at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:893
(inlined by) VersionedBTree::BTreeCursor::Seek_implActorState<VersionedBTree::BTreeCursor::Seek_implActor>::a_body1loopBody1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7497
VersionedBTree::BTreeCursor::Seek_implActorState<VersionedBTree::BTreeCursor::Seek_implActor>::a_body1loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:24752
(inlined by) VersionedBTree::BTreeCursor::Seek_implActorState<VersionedBTree::BTreeCursor::Seek_implActor>::a_body1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:24731
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:872
(inlined by) VersionedBTree::BTreeCursor::seek_impl(VersionedBTree::BTreeCursor* const&, RedwoodRecordRef const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7472
(inlined by) VersionedBTree::BTreeCursor::seek(RedwoodRecordRef) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7507
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:807
(inlined by) StrictFuture at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:893
(inlined by) VersionedBTree::BTreeCursor::SeekGTE_implActorState<VersionedBTree::BTreeCursor::SeekGTE_implActor>::a_body1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7511
Future<Void>::operator=(Future<Void>&&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:839
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont1(Void const&, int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7821
KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1(int) at VersionedBTree.actor.g.cpp:?
ReadRange_implActor at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27829
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:872
(inlined by) KeyValueStoreRedwood::readRange_impl(KeyValueStoreRedwood* const&, Standalone<KeyRangeRef> const&, int const&, int const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7801
(inlined by) KeyValueStoreRedwood::readRange(KeyRangeRef, int, int, IKeyValueStore::ReadType) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/VersionedBTree.actor.cpp:7798
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:807
(inlined by) StrictFuture at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:893
(inlined by) StorageServerDisk::ReadFirstKeyActorState<StorageServerDisk::ReadFirstKeyActor>::a_body1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:325
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:872
(inlined by) StorageServerDisk::readFirstKey(IKeyValueStore* const&, KeyRangeRef const&, IKeyValueStore::ReadType const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:324
(inlined by) StorageServerDisk::readNextKeyInclusive(StringRef, IKeyValueStore::ReadType) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:277
std::__1::vector<Future<Standalone<StringRef> >, std::__1::allocator<Future<Standalone<StringRef> > > >::operator[](unsigned long) at /usr/local/bin/../include/c++/v1/vector:1572
(inlined by) (anonymous namespace)::DoEagerReadsActorState<(anonymous namespace)::DoEagerReadsActor>::a_body1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:4207
(inlined by) DoEagerReadsActor at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:17797
(inlined by) doEagerReads(StorageServer* const&, UpdateEagerReadInfo* const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:4201
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:807
(inlined by) StrictFuture at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:893
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont1loopBody1(int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:6832
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont1loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27186
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10when1(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27081
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont9break1(int) at storageserver.actor.g.cpp:?
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont9break1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:26922
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont9loopHead1(int) at storageserver.actor.g.cpp:?
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont9loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:26899
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont5loopHead1(int) at storageserver.actor.g.cpp:?
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont5loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:26734
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont2(int) at storageserver.actor.g.cpp:?
Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:872
(inlined by) update(StorageServer* const&, bool* const&) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:6674
Future<Void>::operator=(Future<Void>&&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:839
(inlined by) (anonymous namespace)::StorageServerCoreActorState<(anonymous namespace)::StorageServerCoreActor>::a_body1loopBody1when6(Void const&, int) at /mnt/ephemeral/abeamon/foundationdb/fdbserver/storageserver.actor.cpp:8652
ActorCallback<(anonymous namespace)::StorageServerCoreActor, 5, Void>::fire(Void const&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:1319
SAV<Void>::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont10(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:28097
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont24(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:28104
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont9(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27793
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont6break1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27846
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont6loopBody1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27816
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont6loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27806
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont4loopHead1(int) at storageserver.actor.g.cpp:?
(anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont1loopBody1when1(Void const&, int) at storageserver.actor.g.cpp:?
(inlined by) (anonymous namespace)::UpdateActorState<(anonymous namespace)::UpdateActor>::a_body1cont10cont1loopBody1when1(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:27351
ActorCallback<(anonymous namespace)::UpdateActor, 6, Void>::fire(Void const&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:1319
SAV<Void>::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) (anonymous namespace)::DoEagerReadsActorState<(anonymous namespace)::DoEagerReadsActor>::a_body1cont6(std::__1::vector<Optional<Standalone<StringRef> >, std::__1::allocator<Optional<Standalone<StringRef> > > > const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:17666
(inlined by) (anonymous namespace)::DoEagerReadsActorState<(anonymous namespace)::DoEagerReadsActor>::a_body1cont1when1(std::__1::vector<Optional<Standalone<StringRef> >, std::__1::allocator<Optional<Standalone<StringRef> > > > const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:17700
SAV<std::__1::vector<Optional<Standalone<StringRef> >, std::__1::allocator<Optional<Standalone<StringRef> > > > >::delFutureRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:740
(inlined by) ~Future at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:829
(inlined by) (anonymous namespace)::DoEagerReadsActorState<(anonymous namespace)::DoEagerReadsActor>::a_body1cont1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:17561
ActorCallback<(anonymous namespace)::DoEagerReadsActor, 0, std::__1::vector<Standalone<StringRef>, std::__1::allocator<Standalone<StringRef> > > >::fire(std::__1::vector<Standalone<StringRef>, std::__1::allocator<Standalone<StringRef> > > const&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:1319
SAV<std::__1::vector<Standalone<StringRef>, std::__1::allocator<Standalone<StringRef> > > >::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) (anonymous namespace)::GetAllActorState<Standalone<StringRef>, (anonymous namespace)::GetAllActor<Standalone<StringRef> > >::a_body1cont1(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/flow/genericactors.actor.g.h:11110
(inlined by) (anonymous namespace)::GetAllActorState<Standalone<StringRef>, (anonymous namespace)::GetAllActor<Standalone<StringRef> > >::a_body1when1(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/flow/genericactors.actor.g.h:11139
ActorCallback<(anonymous namespace)::GetAllActor<Standalone<StringRef> >, 0, Void>::fire(Void const&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:1319
SAV<Void>::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) void SAV<Void>::sendAndDelPromiseRef<Void>(Void&&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:687
QuorumCallback<Standalone<StringRef> >::fire(Standalone<StringRef> const&) at /mnt/ephemeral/abeamon/foundationdb/flow/genericactors.actor.h:925
SAV<Standalone<StringRef> >::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) StorageServerDisk::ReadFirstKeyActorState<StorageServerDisk::ReadFirstKeyActor>::a_body1cont1(Standalone<RangeResultRef> const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:394
StorageServerDisk::ReadFirstKeyActorState<StorageServerDisk::ReadFirstKeyActor>::a_callback_fire(ActorCallback<StorageServerDisk::ReadFirstKeyActor, 0, Standalone<RangeResultRef> >*, Standalone<RangeResultRef> const&) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/storageserver.actor.g.cpp:469
SAV<Standalone<RangeResultRef> >::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) (anonymous namespace)::ForwardErrorActorState<Standalone<RangeResultRef>, (anonymous namespace)::ForwardErrorActor<Standalone<RangeResultRef> > >::a_body1cont2(Standalone<RangeResultRef> const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:5722
(inlined by) (anonymous namespace)::ForwardErrorActorState<Standalone<RangeResultRef>, (anonymous namespace)::ForwardErrorActor<Standalone<RangeResultRef> > >::a_body1when1(Standalone<RangeResultRef> const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:5741
ActorCallback<(anonymous namespace)::ForwardErrorActor<Standalone<RangeResultRef> >, 0, Standalone<RangeResultRef> >::fire(Standalone<RangeResultRef> const&) at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:1319
SAV<Standalone<RangeResultRef> >::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont2(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27052
KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont4break1(int) at VersionedBTree.actor.g.cpp:?
KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont4loopBody1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27311
KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont4loopHead1(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27253
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont4(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27070
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont7(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27171
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_body1cont6when1(Void const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27183
(inlined by) KeyValueStoreRedwood::ReadRange_implActorState<KeyValueStoreRedwood::ReadRange_implActor>::a_callback_fire(ActorCallback<KeyValueStoreRedwood::ReadRange_implActor, 2, Void>*, Void const&) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:27204
SAV<Void>::finishSendAndDelPromiseRef() at /mnt/ephemeral/abeamon/foundationdb/flow/flow.h:693
(inlined by) VersionedBTree::BTreeCursor::SeekGTE_implActorState<VersionedBTree::BTreeCursor::SeekGTE_implActor>::a_body1cont2(int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:25130
(inlined by) VersionedBTree::BTreeCursor::SeekGTE_implActorState<VersionedBTree::BTreeCursor::SeekGTE_implActor>::a_body1cont1(int const&, int) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:25030
VersionedBTree::BTreeCursor::SeekGTE_implActorState<VersionedBTree::BTreeCursor::SeekGTE_implActor>::a_callback_fire(ActorCallback<VersionedBTree::BTreeCursor::SeekGTE_implActor, 0, int>*, int const&) at /mnt/ephemeral/abeamon/build/foundationdb.linux.clang.x86_64/fdbserver/VersionedBTree.actor.g.cpp:25092
The bug is that somehow the Evictor list contains a pointer to an invalid or freed or uninitialized entry.
The bug is that somehow the Evictor list contains a pointer to an invalid or freed or uninitialized entry.
Sounds the same as https://github.com/apple/foundationdb/issues/7372#issuecomment-1167954910, but that one looks mysterious to me.
