piped-processing-language icon indicating copy to clipboard operation
piped-processing-language copied to clipboard

[FEATURE]Move PPL AST (SQL) and consolidate

Open YANG-DB opened this issue 1 year ago • 3 comments

Is your feature request related to a problem?

This second step of consolidating the PPL spec into a single repository which will be deployed as an independent artifact.

What solution would you like?

Second step:

  • Move PPL AST visitors into this repo
  • consolidate different divergence between SQL's & Spark's PPL AST
  • add as a jar artifact publishing to Maven

#23 - Main RFC

Do you have any additional context?

  • https://github.com/opensearch-project/sql/tree/main/ppl/src/main/antlr
  • https://github.com/opensearch-project/sql/tree/main/ppl/src/main/java/org/opensearch/sql/ppl
  • https://github.com/opensearch-project/opensearch-spark/tree/main/ppl-spark-integration/src/main/java/org/opensearch/sql/ast
  • https://github.com/opensearch-project/opensearch-spark/tree/main/ppl-spark-integration/src/main/antlr4

YANG-DB avatar Nov 27 '24 21:11 YANG-DB

We should have some guidelines ahead of the implementation, regarding the part of consolidate different divergence between SQL's & Spark's PPL AST Some divergence I foresee would be:

  • If the PPL command on both sides take the same amount of arguments but in different order, the order from SQL Plugin should be honoured.

  • If arguments are being read differently in terms of type hierarchy, cast up should happen. (Number v.s Integer, Date represent in String format v.s Date)

andy-k-improving avatar Nov 28 '24 19:11 andy-k-improving

@andy-k-improving I agree - we need to review each case and determine the differences and see :

  • do they actually matter for the result ?
  • what is the cause of this diff ?
  • is it simple to change to code to overcome this diff OR add the additional dialect layer that would do the translation based on each command (the dialect will have its own grammar file)

YANG-DB avatar Nov 28 '24 20:11 YANG-DB

[Catch All Triage - 1, 2, 3]

dblock avatar Dec 16 '24 17:12 dblock