conduit icon indicating copy to clipboard operation
conduit copied to clipboard

JavaScript transforms: Provide easy way to work with structured data

Open hariso opened this issue 3 years ago • 2 comments

Feature description

Currently, it's not possible for JS transforms to return records with changed structured data. This issue has two parts:

  1. There's no way in a JS script to create a new structured payload. The helper method Conduit currently provides make it possible to create raw payloads.
  2. There's no easy way to modify a structured payload (e.g. replacing a key, adding a new one etc.). The struct used to represent structured data, record.StructuredData has the underlying type map[string]interface{}, however, due to the explanation given here, it's not possible to use it as a map.

hariso avatar Jun 16 '22 12:06 hariso

Given the feedback in the mentioned issue in goja, using an external getter/setter should be good, as in this gist.

This works on the latest version of goja, but not the version Conduit is using. Upgrading the goja version though breaks some tests, so this needs a bit more work than just adding the function and upgrading the library.

hariso avatar Jun 16 '22 12:06 hariso

We're going to come back to this after we get through OpenCDC and performance benchmarks.

neovintage avatar Jul 18 '22 16:07 neovintage