StackExchange
Subscribing to multiple Streams in background service
We are currently running one background service with one kafka consumer in each microservice. Each Kafka consumer is subscribing to multiple topics from a variety of producers. A list of topic strings is enough to subscribe. With a switch we can then handle each topic separately if needed.
We would now like to represent the same structure but with Redis Streams, but I'm not sure how to set it up with Stackexchange.Redis library.
For me it seems like we need to create a bunch of parallel running tasks that subscribe individually to a topic/stream and then is also handling the message.
Is that the only way one background service is able to handle multiple Streams ?
@slorello89 Do you have examples anywhere of this? There's a variety of approaches here but it's more of a Redis question than a library one.
@kurtisane @NickCraver :
Here's an example I wrote a while ago about producing to and reading from streams. This is a pretty trivial example but should be adaptable to what you are looking for.
The short answer is - you do not need more than one consumer thread/task to manage the consumption per consumer, a single read to Redis can read from multiple streams in both a consumer group and outside a consumer group see the consumer task spun out - This example only reads from a single stream, but you can pass in an array of StreamPosition arguments to make a single consumer read from multiple streams.
The only downside of using StackExchange.Redis in this instance is that due to its multiplexed architecture, blocking commands are not allowed (you would block every other interactive command if the conneciton was blocked). So you will need to poll the server somewhat regularly with XREAD/XREADGROUP commands.
but keep polling for the message will not be effiecient write, especially if you are polling in 5 services @slorello89 @NickCraver
@mohit-vigo, the connections will always be persistent, and other commands can ride along with the XREAD/XREADGROUPS as the multiplexer automatically pipelines everything it sees. How expensive it will be to poll across five services will be determined by how active your streams are. For example, if you polled once a second, and you only get new messages every 5 seconds, those 5 extra XREADs/sec are going to have a negligible impact on performance on a system that can handle thousands of such operations a second. if your streams are getting 5 messages / second then those XREADs will be more expensive, but you should never be in a position where you would have been blocking in the first place, as there should always be something in the stream to read.
Keep in mind the XREADs will not prevent other operations from running against Redis on the same connection so if in that second interval 50 Sets and 50 Gets come into the multiplexer, they'll all be pipelined to Redis together (they might not all go in the same packets naturally, but they'll generally be pipelined together)