Issue with OffsetArrays
I was getting nervous about the communication costs of being imprecise with communications outside of the localpart of the array, so I figured I'd build a halo exchange system by combining OffsetArrays and DistributedArrays.
I intended for that to look like the following, where each array has a 4x4 block of purely local data and a surrounding 1-cell halo to cache remote data pulled using DistributedArrays.
@everywhere using DistributedArrays
@everywhere using OffsetArrays
r1 = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
r2 = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
r3 = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
r4 = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
ras = [r1 r2; r3 r4]
D = DArray(ras)
Unfortunately, when I try to run this, I get:
julia> D = DArray(ras)
ERROR: MethodError: no method matching OffsetArray{Float64,2,Array{Float64,2}}(::Array{Float64,2})
Closest candidates are:
OffsetArray{Float64,2,Array{Float64,2}}(::Any, ::Any) where {T, N, AA<:AbstractArray} at /home/rick/.julia/packages/OffsetArrays/ICRTb/src/OffsetArrays.jl:10
Stacktrace:
[1] empty_localpart(::Type, ::Int64, ::Type) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:66
[2] macro expansion at ./task.jl:264 [inlined]
[3] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:84 [inlined]
[4] macro expansion at ./task.jl:244 [inlined]
[5] DArray(::Tuple{Int64,Int64}, ::Array{Future,2}, ::Tuple{Int64,Int64}, ::Array{Int64,2}, ::Array{Tuple{UnitRange{Int64},UnitRange{Int64}},2}, ::Array{Array{Int64,1},1}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:82
[6] macro expansion at ./task.jl:266 [inlined]
[7] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:194 [inlined]
[8] macro expansion at ./task.jl:244 [inlined]
[9] DArray(::Array{Future,2}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:192
[10] top-level scope at none:0
which doesn't seem like an issue with OffsetArrays because using SparseArrays in an analogous way produces no problems:
@everywhere using DistributedArrays
@everywhere using SparseArrays
r1 = @spawnat 2 sparse(zeros(6,6))
r2 = @spawnat 3 sparse(zeros(6,6))
r3 = @spawnat 4 sparse(zeros(6,6))
r4 = @spawnat 5 sparse(zeros(6,6))
ras = [r1 r2; r3 r4]
D = DArray(ras)
Sadly, I'm not familiar enough with Julia to know why the interaction of OffsetArrays and DistributedArrays is problematic.
I've got:
[aaf54ef3] DistributedArrays v0.5.1
[6fe1bfb0] OffsetArrays v0.8.1
Should hopefully be fixed by #175, but what is the end goal? Would be lovely to have a larger test-case, I suspect the OffsetArrays will shake out a couple of assumption we have about axes vs indices
I have a problem that I'd usually use C++ and MPI for which involves many nodes loading data and holding that data in memory for the duration of the program. The data collectively represents a large domain. Local computation is performed and halos (the edge cells of the local matrices) are exchanged to keep things synchronized.
Julia doesn't seem well-suited to this paradigm from a library perspective, though the language itself seems fine. So I've been experimenting with ways to achieve this behaviour.
Yes that sounds like a usecase I would love to eventually support. We have a project right now that will require doing something similar.
Note that with the conversion method now being defined in OffsetArrays.jl, the construction of the DArray works:
julia> @everywhere using DistributedArrays
julia> @everywhere using OffsetArrays
julia> r1 = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(2, 1, 93, nothing)
julia> r2 = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(3, 1, 94, nothing)
julia> r3 = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(4, 1, 95, nothing)
julia> r4 = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(5, 1, 96, nothing)
julia> ras = [r1 r2; r3 r4]
2×2 Matrix{Future}:
Future(2, 1, 93, nothing) Future(3, 1, 94, nothing)
Future(4, 1, 95, nothing) Future(5, 1, 96, nothing)
julia> D = DArray(ras);
julia> D |> typeof
DArray{Float64, 2, OffsetMatrix{Float64, Matrix{Float64}}}
However the indices are not correct as DArray explicitly assumes 1-based indexing.
julia> D |>axes
(Base.OneTo(12), Base.OneTo(12))