Support for event subscriptions with wildcard topics

[kars7e]

• [ ] Bug
• [ ] Feature
• [X] Enhancement

Detailed Description

#351 added support for Kafka transport for events. Kafka does not support wildcard topics when creating consumers, thus I've made subscriptions literal (user has to provide full source and event types without any wildcards *). Wildcard support seems like a valid use case though, so this issue is meant to discuss how should this be supported.

Context

RabbitMQ supports subscriptions with wildcard topics, e.g. subscription with topic vcenter.* will match all events having vcenter. prefix. This made the implementation of wildcard subscriptions easy. Kafka does not support such wildcard subscriptions, they must match entirely. There are multiple ways to work around that, e.g. we could create a topic dispatch.events, send all events to that topic. Event manager would then consume all messages on that topic, and would be reponsible for filtering them based on subscriptions. This has it's own drawbacks though, as it moves the filtering logic from message broker to event manager. Another option would be to create a topic per source type. As all events are required to specify both source type and event type (where event type is what most know as "topic", e.g. for vcenter driver source type would be "vcenter" and event type would be "vm.created"), source type could be the Kafka topic, and additional filtering would be required to be done by event manager to handle event type wildcards. However, there is an issue of handling multi-tenancy here, which is not yet defined but will eventually come into the picture. RabbitMQ can use different exchanges per each tenant, ensuring isolation.

Regardless of the implementation, here is a short list of considerations to make:

• Multitenancy - how is this taken care of
• Performance - make sure we are not working against Kafka principles.
• Different function execution model - right now, Event Manager executes functions by calling Function Manager HTTP API. What if we wanted to change and, and send these requests using Kafka directly.
• Wildcard subscriptions - do we want to support them at all?
• Topic registration - How to support events coming from external sources that that work in push model (rather than pull model which supported by event drivers).

Complexity

• [ ] Low - Simple enhancement or bug fix, no architectural changes or refactoring
• [X] Medium - Change requires some thought, but is relatively isolated
• [ ] High - Significant architectural change or large refactor

Impact

• [ ] Low - Annoyance, but does not impact business or functionality
• [X] Medium - Issue can be worked around, but is causing pain
• [ ] High - Blocker