Streaming Roadmap

[gaborhermann]

This is a meta-issue about what needs to be done for introducing StreamBags to Emma. A lot of more theoretical work needs to be done (especially at optimizations). This issue will reflect the state of implementations, and will change over time.

We will work on branch streaming for now.

User-facing API

• [x] StreamBag abstraction
• [ ] Stabilize user facing API (linear road #201)

Compilation

• [ ] Resugaring comprehensions (reuse Emma batch code)
• [ ] Comprehension normalization (reuse Emma batch code)
• [ ] Define stream target language (i.e. dataflow with stateful flatMaps) (issue #202)
• [ ] Translate comprehensions to target language
• [ ] Backends for target language (Kafka Streams, Flink, Storm)
• [ ] Serialization of streams based on coalgebraic view (to allow nested StreamBags)

Optimizations

• [ ] General comprehension optimizations (group by, equijoin etc.)
• [ ] Time predicates (for windowing)
• [ ] Make heterogeneous Bag-StreamBag comprehensions scalable
• [ ] Windowing