A Hive table fails to load if a custom schema is used.

[yruslan]

Describe the bug

Originally, this happened when decimal correction is used with Hive, and there are columns having decimal(38,18) types. Pramen tries to 'correct' the schema by applying a custom schema type.

24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number1' (prec=38, scale=18) to decimal(38, 18)
24/04/29 18:53:02 INFO SparkUtils$: Correct 'tbl.number2' (prec=38, scale=18) to decimal(38, 18)

JDBC connection error for jdbc:hive2://example.com:10000;AuthMech=1; No connection attempts                                                                                                                                 left.
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input '.' expecting {'SELECT', ...
== SQL ==
tbl.number1 decimal(38, 18), tbl.number2 decimal(38, 18)
---^^^
Collapse

Code and/or configuration snippet that caused the issue

correct.decimals.in.schema = true
correct.decimals.fix.precision = true

Expected behavior

• [X] Pramen shoule not 'correct' decimal(38,18) since precision and scale are within the range.
• [x] Investigate if custom schema support for Hive can be fixed since it is needed in case the data type does need correction.

Context

• Pramen/pramen-py version: 1.8.5
• Spark version: 2.4.4
• Scala/Python version: 2.11
• Operating system: Linux