conduit
conduit copied to clipboard
JavaScript transforms: Provide easy way to work with structured data
Feature description
Currently, it's not possible for JS transforms to return records with changed structured data. This issue has two parts:
- There's no way in a JS script to create a new structured payload. The helper method Conduit currently provides make it possible to create raw payloads.
- There's no easy way to modify a structured payload (e.g. replacing a key, adding a new one etc.). The struct used to represent structured data,
record.StructuredDatahas the underlying typemap[string]interface{}, however, due to the explanation given here, it's not possible to use it as a map.
Given the feedback in the mentioned issue in goja, using an external getter/setter should be good, as in this gist.
This works on the latest version of goja, but not the version Conduit is using. Upgrading the goja version though breaks some tests, so this needs a bit more work than just adding the function and upgrading the library.
We're going to come back to this after we get through OpenCDC and performance benchmarks.