go-ipld-cbor icon indicating copy to clipboard operation
go-ipld-cbor copied to clipboard

feat: store: BatchOpStore (`GetMany`/`PutMany` methods)

Open i-norden opened this issue 2 years ago • 4 comments

This PR introduces a BatchOpIpldStore that exposes GetMany and PutMany methods. This is for use in parallel ForEach traversal methods for go-hamt-ipld and go-amt-ipld. Actually only the GetMany method is being used there, but a PutMany method was added to compliment it. This PR uses the GetMany blockstore method introduced in https://github.com/ipfs/boxo/pull/492 which, in turn, relies on the TxnDatastore wrappers introduced in https://github.com/ipfs/go-datastore/pull/210.

Parallel go-amt-ipld traversal work that uses this: https://github.com/filecoin-project/go-amt-ipld/pull/84 Parallel go-hamt-ipld traversal work that uses this: https://github.com/filecoin-project/go-hamt-ipld/pull/115

TODO: Replace replace directives if/when dependencies are merged and released

i-norden avatar Oct 17 '23 22:10 i-norden

@i-norden are you still working on this set of changes? sorry nobody's been able to review your changes, if these are still in the path of your work we can prioritise them .. just not this week for me sorry but I'll keep it in my queue, it looks quite useful.

rvagg avatar Apr 19 '24 05:04 rvagg

Hey @rvagg sorry for abandoning this, we are no longer using this work but if there is still general interest in these changes and/or the linked PRs it is dependent on I can rebase and clean things up this week for review!

i-norden avatar Apr 22 '24 16:04 i-norden

Converted to draft for someone raises a production need for this.

gammazero avatar Aug 20 '24 16:08 gammazero

@gammazero fwiw this is used in production by @aschmahmann here: https://github.com/aschmahmann/filexp/blob/main/go.mod#L5-L7

Having either https://github.com/filecoin-project/go-hamt-ipld/pull/103 or https://github.com/filecoin-project/go-hamt-ipld/pull/115 merge would be really great.

cc @rvagg

ribasushi avatar Aug 24 '24 12:08 ribasushi