Add support for EXTRACT syntax and converts it to appropriate Pinot expression
Description
This PR adds support EXTRACT syntax and converts it to its Pinot expression.
This PR will solve the following issue -- https://github.com/apache/pinot/issues/9075
Testing
Verified the desired behavior locally by running CalciteSqlCompilerTest
thanks, @Jackie-Jiang for providing context. Very helpful.
I am trying to understand how ExtractTransformFunction as you suggested will be used in the overall flow (write path and read path). Few follow-up ques from my investigation:
- Do we store the transformed data (after applying transformations as defined here on raw data) into segments or only raw data into segments and apply the transformations on the fly during Pinot query execution?
ExtractTransformFunction will only be applied at query time. If we want to support ingestion transform, we need to add a ScalarFunction for the extract (you may take a look at DateTimeFunctions` class).
To support this feature, we need 2 parts of the changes:
- Calcite parser change to support
extractsyntax and parse the query intoextract(field, expression) - Add
ExtractTransformFunctionorScalarFunctionto support query/ingestion time transform
Hey @Jackie-Jiang , I have completed part 1 of the task.
For Part2: Can you please help me point to where ‘DateTimeTransformFunction’ (or other query time transformers defined here) are getting applied? I was able to locate TransformFunctionFactory but couldn’t locate where transformations are actually getting applied.
PTAL @Jackie-Jiang when you get a chance. Thanks!