DistributedArrays.jl icon indicating copy to clipboard operation
DistributedArrays.jl copied to clipboard

Issue with OffsetArrays

Open r-barnes opened this issue 7 years ago • 4 comments

I was getting nervous about the communication costs of being imprecise with communications outside of the localpart of the array, so I figured I'd build a halo exchange system by combining OffsetArrays and DistributedArrays.

I intended for that to look like the following, where each array has a 4x4 block of purely local data and a surrounding 1-cell halo to cache remote data pulled using DistributedArrays.

@everywhere using DistributedArrays
@everywhere using OffsetArrays

r1  = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
r2  = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
r3  = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
r4  = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
ras = [r1 r2; r3 r4]
D   = DArray(ras)

Unfortunately, when I try to run this, I get:

julia> D   = DArray(ras)
ERROR: MethodError: no method matching OffsetArray{Float64,2,Array{Float64,2}}(::Array{Float64,2})
Closest candidates are:
  OffsetArray{Float64,2,Array{Float64,2}}(::Any, ::Any) where {T, N, AA<:AbstractArray} at /home/rick/.julia/packages/OffsetArrays/ICRTb/src/OffsetArrays.jl:10
Stacktrace:
 [1] empty_localpart(::Type, ::Int64, ::Type) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:66
 [2] macro expansion at ./task.jl:264 [inlined]
 [3] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:84 [inlined]
 [4] macro expansion at ./task.jl:244 [inlined]
 [5] DArray(::Tuple{Int64,Int64}, ::Array{Future,2}, ::Tuple{Int64,Int64}, ::Array{Int64,2}, ::Array{Tuple{UnitRange{Int64},UnitRange{Int64}},2}, ::Array{Array{Int64,1},1}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:82
 [6] macro expansion at ./task.jl:266 [inlined]
 [7] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:194 [inlined]
 [8] macro expansion at ./task.jl:244 [inlined]
 [9] DArray(::Array{Future,2}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:192
 [10] top-level scope at none:0

which doesn't seem like an issue with OffsetArrays because using SparseArrays in an analogous way produces no problems:

@everywhere using DistributedArrays
@everywhere using SparseArrays

r1  = @spawnat 2 sparse(zeros(6,6))
r2  = @spawnat 3 sparse(zeros(6,6))
r3  = @spawnat 4 sparse(zeros(6,6))
r4  = @spawnat 5 sparse(zeros(6,6))
ras = [r1 r2; r3 r4]
D   = DArray(ras)

Sadly, I'm not familiar enough with Julia to know why the interaction of OffsetArrays and DistributedArrays is problematic.

I've got:

  [aaf54ef3] DistributedArrays v0.5.1
  [6fe1bfb0] OffsetArrays v0.8.1

r-barnes avatar Oct 02 '18 08:10 r-barnes

Should hopefully be fixed by #175, but what is the end goal? Would be lovely to have a larger test-case, I suspect the OffsetArrays will shake out a couple of assumption we have about axes vs indices

vchuravy avatar Oct 02 '18 13:10 vchuravy

I have a problem that I'd usually use C++ and MPI for which involves many nodes loading data and holding that data in memory for the duration of the program. The data collectively represents a large domain. Local computation is performed and halos (the edge cells of the local matrices) are exchanged to keep things synchronized.

Julia doesn't seem well-suited to this paradigm from a library perspective, though the language itself seems fine. So I've been experimenting with ways to achieve this behaviour.

r-barnes avatar Oct 11 '18 21:10 r-barnes

Yes that sounds like a usecase I would love to eventually support. We have a project right now that will require doing something similar.

vchuravy avatar Oct 15 '18 14:10 vchuravy

Note that with the conversion method now being defined in OffsetArrays.jl, the construction of the DArray works:

julia> @everywhere using DistributedArrays

julia> @everywhere using OffsetArrays

julia> r1  = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(2, 1, 93, nothing)

julia> r2  = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(3, 1, 94, nothing)

julia> r3  = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(4, 1, 95, nothing)

julia> r4  = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(5, 1, 96, nothing)

julia> ras = [r1 r2; r3 r4]
2×2 Matrix{Future}:
 Future(2, 1, 93, nothing)  Future(3, 1, 94, nothing)
 Future(4, 1, 95, nothing)  Future(5, 1, 96, nothing)

julia> D   = DArray(ras);

julia> D |> typeof
DArray{Float64, 2, OffsetMatrix{Float64, Matrix{Float64}}}

However the indices are not correct as DArray explicitly assumes 1-based indexing.

julia> D |>axes
(Base.OneTo(12), Base.OneTo(12))

jishnub avatar May 02 '21 09:05 jishnub