Transformed CE type can lead to unexpected trigger filters when using `--source`

[jmcx]

If I use a JSON transformation to customise the value of the CE type using stored variables, as such:

context:
- operation: add
  paths:
  - key: type
    value: $region-$category
data:
- operation: store
  paths:
  - key: $region
    value: region
  - key: $category
    value: category

And create this component, where orders-source is a kafka source (not relevant):

tmctl create transformation --name transform-extract-region-category -f orders-add-region-category.yaml --source orders-source

This creates a component which in tmctl describe shows eventType as $region-$category which is actually cool because I know this is going to be a dynamic event type:

transform-extract-region-category $region-$category online(http://localhost:56877)

BUT if I then use this transformation as a source, e.g. in a new transformation:

tmctl create transformation -f orders-eu-formatting.yaml --source transform-extract-region-category

Then this uses the variable expression as part of the filter in the Trigger, e.g. as shown here in the tmctl describe output and in the Trigger CRD:

order-router-trigger-e8079841 order-router-transformation type is $region-$category

filters:
        - exact:
            type: $region-$category

This filter does not do what I'm expecting it to

[jmcx]

Some rough solution ideas

• using the actual source metadata attribute could help (again coming back to my metadata issue :sweat_smile: )
• the tmctl create command could fail if it sees you're using --source with a component that has variables in its output event type, and could recommend explicitly creating a Trigger instead

[tzununbekov]

Well, I'll elaborate on my slack message here.

You have following transformation:

context:
- operation: add
  paths:
  - key: type
    value: $region-$category
  - key: extensions.region
    value: $region
  - key: extensions.category
    value: $category
data:
- operation: store
  paths:
  - key: $region
    value: region
  - key: $category
    value: category

With this input event payload:

{"hello":"Jonathan"}

Resulting event type (after transformation) will be $region-$category which plays well with the generated trigger filter.

If incoming event payload would look like this:

{"region":"eu"}

The output event type will be eu-$category which, although does not "fit" the trigger filter expectation, also does not make it "not valid".

So there are two components that make this issue solution impossible:

1. Event types (or any attributes) defined in the transformation with the variables depend on the incoming events data, hence they are entirely unpredictable,
2. There is no way to figure out if the event type exported by the transformation ($region-$category) consists of variables or if it is just a plain type string (even having $ is not a requirement for the variable).

So, we cannot just disable the filter generation for transformations because we don't have the ability to define if the type is dynamic or not, and we cannot put the expected filter string because the result type depends on the incoming data. The describe output with the dynamic transformation type ($region-$category) should be taken as a insight of what can be produced by the component, not the eventual result.