go-ds-flatfs
go-ds-flatfs copied to clipboard
ipfs
A datastore implementation using sharded directories and flat files to store data
Current implementation of putMany seems to open as many FDs as there are entries to put, with batches few K in size this will cause `too many files open` errors....
The README is misleading https://github.com/ipfs/go-ds-flatfs/blob/f0750204617b5b0cfb414c38ce486bf9f01963bb/readme.go#L3-L33 FlatFS does not know anything about CIDs, blocks or IPLD and actually is being used by different projects in the ecosystem in different ways. In...
Bumps [github.com/ipfs/go-datastore](https://github.com/ipfs/go-datastore) from 0.4.4 to 0.4.6. Commits b5dee91 Merge pull request #175 from ipfs/web3-bot/sync 4f52b50 test: completely disable delay in delaystore test cf976ab chore: tidy submodule 1029008 add .github/workflows/go-check.yml 86129bc...
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot] avatar"){kind=avatar}
_Dependabot Preview will be shut down on August 3rd, 2021. In order to keep getting Dependabot updates, please merge this PR and migrate to GitHub-native Dependabot before then._ Dependabot has...
Bumps [github.com/ipfs/go-log](https://github.com/ipfs/go-log) from 1.0.3 to 1.0.5. Release notes Sourced from github.com/ipfs/go-log's releases. Release v1.0.5 Update go-log v2 dep to v2.1.3. Commits c39ee80 Merge pull request #108 from ipfs/chore/update-maint-deps f58978d ci:...
Currently, we use a slice and sync one dir per put. We should use a map and sync each directory once.
ref: https://github.com/ipfs/go-ipfs/issues/7115#event-3219629851 Current attempt towards mitigation is to open files on windows with the SHARE_DELETE flag per https://github.com/golang/go/issues/32088
Part of the work to unblock: https://github.com/ipfs/go-log/pull/65
See: https://github.com/ipfs/go-datastore/pull/103. Either that or we need to document the semantics of batch (and say that it first puts, then deletes).
``` Error: context canceled panic: send on closed channel goroutine 72 [running]: gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs.(*Datastore).checkpointDiskUsage(0xc420514230) ~/go/src/gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs/flatfs.go:820 +0x39 gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs.(*Datastore).updateDiskUsage(0xc420514230, 0xc420ddecc0, 0x1a, 0x1) ~/go/src/gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs/flatfs.go:814 +0x65 gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs.(*Datastore).makeDirNoSync(0xc420514230, 0xc420ddecc0, 0x1a, 0xc420ddecc0, 0x1a) ~/go/src/gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs/flatfs.go:321 +0xc8 gx/ipfs/Qmc1ExJkrEUesbvRNHq6g3bJ2km7K4XffcC2bvwdpfGYDx/go-ds-flatfs.(*Datastore).putMany(0xc420514230, 0xc4206fb290,...
