Support ADAM for parameters of generative functions

[alex-lew]

Currently, users can configure an ADAM parameter update, but can only use it on Torch or TensorFlow parameters. We should implement a version of ADAM for the static and dynamic modeling languages.

In general, it seems like it would be quite straightforward to allow for any of these optimizers, provided by Flux, to be used directly: https://github.com/FluxML/Flux.jl/blob/master/src/optimise/optimisers.jl

They each follow essentially the same pattern that Gen does :)

[alex-lew]

OK, prototyped some support for this at https://github.com/probcomp/GenFluxOptimizers.jl.

One question, though, is how to handle Gen's ADAM update configuration. It is not supported by the built-in modeling languages, only by TF and Torch functions. Should GenFluxOptimizers.jl provide implementations of Gen.apply! and Gen.initialize_update_state for Gen.ADAM configurations? Or is there a better code organization?

[SamWitty]

Any chance GenFluxOptizers.jl can be merged with Gen.jl? Either that or a standalone Julia package? I'm running into a use-case that would benefit from ADAM.

Thanks, and it looks great!

[fsaad]

Any chance GenFluxOptizers.jl can be merged with Gen.jl?

@SamWitty You could install GenFluxOptimizers.jl into your local installation by writing the following into the Julia REPL

$ julia
julia> using Pkg
julia> Pkg.add(Pkg.PackageSpec(url="https://github.com/probcomp/GenFluxOptimizers.jl", rev="master"))

[alex-lew]

Yep -- @SamWitty in case it wasn't clear from my messages, GenFluxOptimizers.jl is not a fork of Gen, but rather a package meant to be used alongside it. Just add it to your project, and then run using GenFluxOptimizers (in addition to using Gen) in the file where you want to use it.

[SamWitty]

Got it, thanks for the clarification @fsaad and @alex-lew.