Decimal fails to serialize to JSON

[sergiimk]

Problem

Serializing a dataset that contains columns with DECIMAL(s,p) logical type currently results in a crash.

Steps to reproduce

1. Go to examples/currency_conversion
2. Change data type of all DOUBLE columns to DECIMAL(18,10)
3. Ingest the data
4. Run kamu tail ca.bankofcanada.exchange-rates.daily to see data
5. Now run kamu tail ca.bankofcanada.exchange-rates.daily --output-format json

Actual result

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: JsonError("data type Decimal128(18, 10) not supported in nested map for json writer")', /project/kamu-core/src/infra/utils/records_writers.rs:57:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Expected result

Serialization works fine

Notes

The issue is likely in an upstream arrow-json project, so we should:

• test this issue with latest version of arrow & datafusion
• ensure we are using JSON writer correctly
• search for open issues in arrow repo and submit one if there isn't any
• contribute a fix

[Wizzy-wooz]

@sergiimk I can't reproduce issue... It seems it works for me...Latest master.

[sergiimk]

The last output is clearly not JSON... Have you re-built the master branch after switching to it from your dev branch?

You can also install try this using a release build.

[Wizzy-wooz]

Need debugging - but I have checked out latest master for sure.

[sergiimk]

Tested this on datafusion 37.1.0. It doesn't crash any more but does return an error:

Error: Cannot write output, arrow error occurred : Invalid argument error: JSON Writer does not support data type: Decimal128(18, 10)