[Feature] Multiple transports support

[Sturgelose]

Proposal

I've been using Faust as some internal experiments in my company to have a queuing system, and I must say I love it! So, first of all, thanks a lot for your great effort in your project, it's amazing!

Anyways, in our use case we want to use not only queues but also microservices. Thus, we may ingest data from a Kafka producer and process it by using Faust (let's say with function F(x)) (in this case, we do not have to return any response or answer to the process submitting the data). Also, we would like to use exactly the same logic to receive data from a HTTP Rest call, apply this F(x) and return an answer to this HTTP Rest call (plus if possible, forwarding some data to Kafka as an event).

This way we could use a microservices architecture either feeding data via Kafka (queue fashion without an answer) or from our exposed API (microservice fashion providing an answer back to the requester) while using exactly the same code and resources, mostly because the main function will be the same and the only difference is how the data is returned.. This would reduce maintenance, and also, I think it fits with Faust's idea that is fetching data from a transport and forwarding data to another one.

I'm aware that Faust exposes a Rest API to fetch real time data, but I'm not sure if it allows to insert/write data via it.

I know this is a bit complex, but I'd be willing to hear if this fits (or not) to this project, or any directions on how this could be implemented, as I'd be willing to give a hand and implement this in a PR.

Again, thanks a lot, and do not hesitate to ask me anything if you see any issue or there is any doubt in my explanation.

[luochenxi]

I agree with the above points! Want to get real-time data through Rest API.

Producer:

• Request Method：POST
• URL：http://kaproxy/topic/ + topic_name
• url_parameters: token,partitioner,value eg:

          partition = crc32_client(req_id, self.partitioner)
          url = "http://" + str(self.kaproxy) + "/topic/" + str(self.topic)
          msg = json.dumps(data_dict)
          r = self.http.request('POST', url,
                                fields={
                                    'token': str(self.token),
                                    'partitioner': str(partition),
                                    'value': msg
                                })

Consumers

• Request Method：GET
• URL：http://kaproxy.m.com/group/group_value/topic/topic_value?token=token_value&timeout=timeout_value