bloomberg
Allowing omitting the "T" separator from a datetime string
Our customers have issue using spark to create a dataframe on a datetime column, because spark interpolates the provided datetime string, but drops the "T" separator when filling the range.
It seems that omitting the "T" character is fairly acceptable, and ISO 8601 does allow it to be omitted in a few cases. This is a very simple patch to allow it.
If we allow strings without T to be converted to datetimes, we have the following conundrum: is "20250101" a datetime or an epoch value?
If we allow strings without T to be converted to datetimes, we have the following conundrum: is "20250101" a datetime or an epoch value?
If the string contains only digits, it's treated as a unix timestamp (code). So "20250101" will still be an epoch value! I've also added a test for it!
This breaks UDF and consumer tests?
There's one case that I overlooked, eg 2025-01-01 UTC. Apparently the type code expects that a timezone has a leading space. But because we now additionally allow spaces in between the date portion and the rest of the datetime, we may not have a leading space when we read the timezone. for instance:
2025-01-01 UTC
^ old code would stop here, timezone would be "[space]UTC"
vs
2025-01-01 UTC
^ new code stops here, timezone is "UTC"