Direct support for custom data formats
One can always create custom adapters for data in any format and convert it into IObservable<T> or IObservable<StreamEvent<T>>. However, for some operations, it may instead make sense to instead compile the query into operators that operate directly over data in its native format. For instance, one may instead of having a batch of T payloads or a batch of columns of T, one may have a batch of data in parquet format.
The current heuristic for data flowing through Trill is that if it begins in "row mode" it will stay in row mode for the duration of the query. If it is in "column mode" it will stay that way until it hits an operator that cannot operate on columnar data, so it switches to "row mode". We may be able to allow an extensibility point wherein we allow a query to operate over data in "parquet mode" or "JSON mode" or some other mode, and then when it can no longer do so, switch to either "column mode" or "row mode".