[FEATURE REQUEST]: Support System.Decimal in CreateDataFrame API

[jgavin]

Is your feature request related to a problem? Please describe. I'd like to be able to use the System.Decimal type when using the CreateDataFrame API. Currently when running the following:

decimal? minValue = 4.0M;
decimal? maxValue = 5.0M;
var df = sparkSession.CreateDataFrame(
    new List<GenericRow>
    {
        new GenericRow(
            new object[]
            {
                "1234",
                minValue,
                maxValue,
            }),
    },
    new StructType(
        new List<StructField>()
        {
            new StructField("id", new StringType()),
            new StructField("minValue", new DecimalType()),
            new StructField("maxValue", new DecimalType()),
        }));
df.Show();

We see the following error: System.NotSupportedException: 'Type System.Decimal not supported yet'

This presents issues in trying to unit test code that reads decimal types from a DB.

Describe the solution you'd like To be able to create Dataframes with Decimals without an error 😄

Describe alternatives you've considered Alternatives for us might be to change the type being read from the DB, but this is a significant challenge - for now we are unable to unit test this code.

Additional context N/A

[Niharikadutta]

Thanks @jgavin for creating this issue, we are yet to add support for System.Decimal and will get back to you about the timeline of getting this in.

[dbeavon]

@Niharikadutta

Did you see that decimals were added to Apache Arrow?

• https://issues.apache.org/jira/browse/ARROW-11422
• https://github.com/apache/arrow/pull/9356

This happened recently. Was that a prerequisite? Decimal types are pretty critical for us as well.

[GoEddie]

Hi @Niharikadutta + @imback82 - is there a reason for no decimal or is just waiting for development? is this a good issue to take?

[imback82]

Hi @GoEddie, this is just waiting for development and a good issue to take.