bytedance
[BitSail][Improve]Support transform operator in bitsail.
Is your feature request related to a problem? Please describe
Now, BitSail only support two type of operators
- Source Operator
- Sink Operator
In the original plan, we expect bitsail become a ELT pipeline engine in the data integration scope, but recently we got more and more requirments from the community. Users want bitsail to support lightweight action in the bitsail with follow requirements:
- data value filter: for some dirty data or anyother data don't care.
- data value transform:
- groupby/keyby:
- ...
I think the requirement has mant scenes to be used and will help bitsail become more and more powerful, so i suppose we need support this one.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
is that possable to use the apache beam as the transform engin
It's only a transform operator inside BitSail. Introducing another engine like Beam would be too heavy
