MemPool.jl
MemPool.jl copied to clipboard
JuliaData
High-performance parallel and distributed datastore for Julia
https://github.com/JuliaData/MemPool.jl/blob/8508088b32caff629ce6905cb263f78cdc068cd6/src/storage.jl#L963 The call to `reverse` above assumes `to_delete` is in ascending order, which isn't necessarily the case. I didn't thoroughly test, so I don't know what combinations of `to_mem` and...
Appears sometimes when process exits ``` IOError: unlink("/home/krynju/.mempool/sess-utvz1V-1/h2x1LD/jl_N2bctMjqbi"): no such file or directory (ENOENT) Stacktrace: [1] uv_error @ ./libuv.jl:97 [inlined] [2] unlink(p::String) @ Base.Filesystem ./file.jl:972 [3] rm(path::String; force::Bool, recursive::Bool) @...
At the moment, our storage layer (in `StorageState`) is non-transactional, and so it's not possible to ensure that a certain set of operations occur without other (incompatible or performance-degrading) operations...
The API that `poolget` provides, while simple, is unfortunately harmful to swap-to-disk and related use cases, as it does not indicate when the returned data is no longer in use....
At the moment, creating a new `DRef` requires the associated data to be in memory. This makes it annoying to persist datasets and be able to easily re-load them in...
Leaving this here as it's the first time I see this - it continued computing the result afterwards, but on one worker only Super rare ```julia @@@ STARTED: innerjoin_r_unique :...
When benchmarking parallel application which uses Dagger, it seems like `MemPool.approx_size` is the bottleneck due to it falling back to `Base.summarysize`. Here is a quick MWE: ```julia julia> using BenchmarkTools,...
https://github.com/JuliaComputing/MemPool.jl/pull/22 should make MemPool work for `Vector{Missing.T}`. At least in some cases. However, it will be using the slow fallback serialization so eventually, we should add faster serialization of `Vector{Missing,T}`...
Example situation: - A 1GB table with 10 columns is created - An operation creates a different object with the same 10 columns (e.g. `rows(t)`), MemPool thinks that this operation...
