Added support for sqlType JSON
I would like to add support for sqlType fields. In this pull request I added support only for JSON type since we need it in order to use this library for loading data to BigQuery but there are more sqlTypes that could be supported in the future.
BigQuery documentation for this field: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#extract_json_data_from_avro_data
Supported case:
{
"type": {"type": "string", "sqlType": "JSON"},
"name": "json_field"
}
I'm sorry created PR a bit too soon.. will close it for now and reopen it once I ran some more tests.
I dont really understand how this fits into the library? Or why the change is needed at all?
Without this feature library removes "sqlType": "JSON" from the schema and with that I end up with is the following (example from the description)
{
"type": "string",
"name": "json_field"
}
in the avro file. Because of that I lose important metadata which is valid case from perspective of Avro specification (this is very similar to logicalType). In my opinion it would also be acceptable if library wouldn't remove this kind of additional data.
The metadata you are referring to is already in the schema via properties. properties are at this time not rendered back out to json as it is generally not needed. To be clear when I say rendered to json i mean MarshalJSON not String as string is the canonical schema which should not include arb properties.
Hi. Due to the lack of activity on this, I am going to close it for now. Should you want to continue on this patch please feel free to open a new PR that takes the discussed direction.