feast Materialization into Datastore leads to contention error

This issue was reported directly to us by a user. Documenting the failure here for posterity:

Materializing 1 feature views from 2021-04-29 21:19:00-07:00 to 2021-04-29 21:19:05-07:00 into the datastore online store.
my_fv:
  8%|████▋                                                    | 2720/33120 [00:07<01:18, 388.10it/s]

[SNIP]

google.api_core.exceptions.Aborted: 409 too much contention on these datastore entities. please try again. entity groups:

Example of the data used for ingestion https://pastebin.com/raw/GCJ2CLSb (this is just a sample, the actual ingestion contains many more rows)

May 22 '21 00:05 woop

This issue seems to only happen in the old Datastore (not Firestore in Datastore mode)

May 24 '21 18:05 woop

I'm facing this issue still 🤔

Dec 14 '21 18:12 judahrand

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Apr 14 '22 14:04 stale[bot]

Consolidating in the linked issue

Apr 21 '22 21:04 adchia

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Dec 20 '22 23:12 stale[bot]

think that my (not very well researched) recommendation here generally probably is to stick with bigtable for larger scale use cases since it's both cheaper, doesn't suffer from these afaict, and now auto scales.

Dec 21 '22 08:12 adchia