Dagger.jl
Dagger.jl copied to clipboard
JuliaParallel
A framework for out-of-core and parallel execution
This pull request adds `DArray` documentation.
Main problem is that adding more workers to the pool is ineffective since there are no unassigned tasks. I suppose this might be done to reduce scheduling latency. If it...
We could provide an option to let users define an error handler function which will handle all kinds of task failures (which may necessitate propagating more richly-annotated information on error...
We should propagate the RNG state (just like Base tasks do right now) across tasks. We should also implement a scheduler option to set an RNG that we'll serialize and...
- Logging and rendering - Cleanup actions
@DrChainsaw @shashi we should add FileTrees reverse CI to Dagger to prevent me from breaking things :smile:
We currently ignore the costs of transferring non-`Chunk` task inputs, but that's silly; they have a real transfer cost, but it's incurred immediately at task transfer to the worker. We...
``` function init_eager() EAGER_INIT[] && return EAGER_INIT[] = true ctx = eager_context() @async try sopts = SchedulerOptions(;allow_errors=true) topts = ThunkOptions(;single=1) Dagger.compute(ctx, Dagger.delayed(eager_thunk;options=topts)(); options=sopts) catch err iob = IOContext(IOBuffer(), :color=>true) println(iob,...
Another one found when benchmarking, also pretty rare. For later, will probably be useful when looking at stability ```julia @@@ STARTING CONFIG: julia -J /tmp/jlsysimage/sysimage.so -p3 -t4 scripts/dtable_innerjoin_unique.jl 500000000 10000000...
This may or may not make sense at the Dagger level, but for consideration - as an example copying the [prefect ](https://docs.prefect.io/core/tutorial/04-handling-failure.html#if-at-first-you-don-t-succeed)keywords below ```julia import Dates import Dagger function flaky_function()...
Metadata
Owner
Metadata
A framework for out-of-core and parallel execution