Parallel gradient, Cannot determine ordering of Dual tags

[aaowens]

I don't have a MWE, but I hit this error on 0.6.2, not master.

ERROR: Cannot determine ordering of Dual tags ForwardDiff.Tag{#nLLp#38,Float64} and ForwardDiff.Tag{Base.Serializer.__deserialized_types__.#nLLp#38,Float64}
Stacktrace:
 [1] extract_gradient_chunk!(::Type{ForwardDiff.Tag{#nLLp#38,Float64}}, ::Array{Float64,1}, ::ForwardDiff.Dual{ForwardDiff.Tag{Base.Serializer.__deserialized_types__.#nLLp#38,Float64},Float64,8}, ::Int64, ::Int64) at /home/andrew/.julia/v0.6/ForwardDiff/src/gradient.jl:81

This happened after I tried to use pmap in an optimization routine.

I know it's possible to opt out of the tag system, but I don't believe NLSolversBase exposes that option when constructing a OnceDifferentiable type.

[MikeInnes]

Going through the serializer is probably always going to be tricky, although there are also some more basic cases where the tag system fails (not silently though, thankfully).

D(x -> x+D(y -> x, 1), 1)
ERROR: Cannot determine ordering of Dual tags ...

[MasonProtter]

I'm also hitting this issue with ForwardDiff and NLsolve.

[mattwigway]

I ran into this error as well; passing arguments to my function as vectors rather than NTuples solved it as then there were no user-defined types that were being serialized. I'm not sure the tagging system will be 100% correct in a distributed environment at this time, as tagcount() in config.jl is based on an atomic int that is per-process.

[FelixBenning]

Here is a MWE I think:

using KernelFunctions: Kernel, KernelFunctions as KF
using OneHotArrays: OneHotVector
import ForwardDiff as FD

struct Pt{Dim}
	pos::AbstractArray
	partial
end

Pt(x;partial=()) = Pt{length(x)}(x, partial)

for T in subtypes(Kernel)
	(k::T)(x::Pt{Dim}, y::Pt{Dim}) where {Dim} = evaluate_(k, x, y)
	(k::T)(x::Pt{Dim}, y) where {Dim} = evaluate_(k, x, Pt(y))
	(k::T)(x, y::Pt{Dim}) where {Dim} = evaluate_(k, Pt(x), y)
end

function evaluate_(k::Kernel, x::Pt{Dim}, y::Pt{Dim}) where {Dim}
	if !isnothing(local next = iterate(x.partial))
		ii, state = next # take partial derivative in direction ii
		return FD.derivative(0) do dx
			evaluate_( # recursion
				k,  
				Pt(
					x.pos .+ dx * OneHotVector(ii, Dim), # directional variation
					partial=Base.rest(x.partial, state) # remaining partial derivatives
				),
				y
			)
		end
	end
	if !isnothing(local next = iterate(y.partial))
		jj, state = next # take partial derivative in direction jj
		return FD.derivative(0) do dy
			evaluate_( # recursion
				k,
				x,
				Pt(
					y.pos .+ dy * OneHotVector(jj, Dim), # directional variation
					partial=Base.rest(y.partial, state) # remaining partial derivatives
				)
			)
		end
	end
	k(x.pos, y.pos)
end

k = KF.MaternKernel()
k(Pt([1], partial=(1)), Pt([2], partial=1))

resulting in

ERROR: Cannot determine ordering of Dual tags ForwardDiff.Tag{var"#7#9"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64} and ForwardDiff.Tag{var"#6#8"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64}
Stacktrace:
 [1] partials
   @ ~/.julia/packages/ForwardDiff/vXysl/src/dual.jl:111 [inlined]
 [2] extract_derivative(#unused#::Type{ForwardDiff.Tag{var"#7#9"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64}}, y::ForwardDiff.Dual{ForwardDiff.Tag{var"#6#8"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64}, ForwardDiff.Dual{ForwardDiff.Tag{var"#7#9"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64}, Float64, 1}, 1})
   @ ForwardDiff ~/.julia/packages/ForwardDiff/vXysl/src/derivative.jl:84
 [3] derivative(f::var"#7#9"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, x::Int64)
   @ ForwardDiff ~/.julia/packages/ForwardDiff/vXysl/src/derivative.jl:14
 [4] evaluate_(k::KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, x::Pt{1}, y::Pt{1})
   @ Main ~/code/incrementalGP/scripts/experiments.jl:34
 [5] (::var"#6#8"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64})(dx::ForwardDiff.Dual{ForwardDiff.Tag{var"#6#8"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, Int64}, Int64, 1})
   @ Main ~/code/incrementalGP/scripts/experiments.jl:22
 [6] derivative(f::var"#6#8"{1, KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, Pt{1}, Pt{1}, Int64}, x::Int64)
   @ ForwardDiff ~/.julia/packages/ForwardDiff/vXysl/src/derivative.jl:14
 [7] evaluate_(k::KernelFunctions.MaternKernel{Float64, Distances.Euclidean}, x::Pt{1}, y::Pt{1})
   @ Main ~/code/incrementalGP/scripts/experiments.jl:21
 [8] (::KernelFunctions.MaternKernel{Float64, Distances.Euclidean})(x::Pt{1}, y::Pt{1})
   @ Main ~/code/incrementalGP/scripts/experiments.jl:13
 [9] top-level scope
   @ ~/code/incrementalGP/scripts/experiments.jl:53