Possible and useful to implement Apache Arrow in IPLD?

Open timcash opened this issue 5 years ago • 2 comments

Apache Arrow is popular in the data science space as it reduces time to load into memory, would it be possible to integrate the format into the Dag? Any tips on where to get started with IPLD to take a shot at it? https://arrow.apache.org/overview/

Jul 29 '20 01:07 timcash

That'd probably depend on how you imagine using it with IPLD, could you explain a workflow that might connect Arrow and IPLD @timcash?

Jul 29 '20 01:07 rvagg

Good question, still learning about IPLD but I was thinking pulling fields out of files in Arrow format. E.g. "/CID/session_id/0" would get the first "session_id" out of the file with that CID.

Jul 30 '20 05:07 timcash