GH-2806 : Receiving an empty list when using RecordFilterStrategy on batch messages

[chickenchickenlove]

Motivation:

• https://github.com/spring-projects/spring-kafka/issues/2806
• In the current batch mode, even if the RecordFilterStrategy filters all records resulting in an Empty List being returned, the KafkaListener is still invoked. In contrast, in single record mode, if record are filtered, the KafkaListener is not called. This difference in behavior between the two modes can cause confusion for users.

Modifications:

• Add public method isAnyManualAck() to Acknowledgment to verify that manualAck is needed on FilteringBatchMessageListenerAdapter.
• Modify FilteringBatchMessageListenerAdapter.
  • add field consumerAware as final (IMHO, we don't need to calculate it every single call onMessage().
  • add logic (if empty list and manual Ack == true, KafkaListener will be invoked. if empty list and manual Ack == false, KafkaListener will not be invoked even if listener is kind of ConsumerAware. In detail, See Discussion section below.)

Result:

• Closes https://github.com/spring-projects/spring-kafka/issues/2806
• When the RecordFilterStrategy filters all records and returns an Empty List, the KafkaListener is invoked only if it is in manual ACK mode.

Discussion

• When using a ConsumerAware Listener, commits can be made using Consumer.commitSync() and Consumer.commitAsync(). However, when using a ConsumerAwareAckListener, it seems possible that commits using the Consumer and commits using Ack could be processed simultaneously. That situation seems quite ambiguous.

[chickenchickenlove]

I see! i reverted all and make new commits.

[artembilan]

I think the problem comes from the MethodKafkaListenerEndpoint.createMessageListenerInstance():

		if (isBatchListener()) {
			BatchMessagingMessageListenerAdapter<K, V> messageListener = new BatchMessagingMessageListenerAdapter<>(
					this.bean, this.method, this.errorHandler);

where we have:

public class BatchMessagingMessageListenerAdapter<K, V> extends MessagingMessageListenerAdapter<K, V>
		implements BatchAcknowledgingConsumerAwareMessageListener<K, V> {

and that leads to the:

		if (listener instanceof AcknowledgingConsumerAwareMessageListener
				|| listener instanceof BatchAcknowledgingConsumerAwareMessageListener) {
			listenerType = ListenerType.ACKNOWLEDGING_CONSUMER_AWARE;
		}

So, the logic in that FilteringBatchMessageListenerAdapter always falls to the consumerAware as true. Therefore sounds like we cannot achieve the requested logic with existing flags. Not sure, though, if that would be convenient to introduce a new one exactly for this use-case for batch filtering.

Maybe RecordFilterStrategy could be improved with extra boolean method to implement? Like:

default boolean ignoreEmptyBatch() {
    return false;
}

[chickenchickenlove]

Thank you for your analysis a lot 🙇‍♂️🙇‍♂️🙇‍♂️🙇‍♂️! The logic remains the same eventually, but I think the direction you proposed is better because it gives users a choice.

[chickenchickenlove]

I make new commit to apply your reviews.

	default boolean ignoreEmptyBatch() {
		return false;
	}

I added ignoreEmptyBatch() to RecordFilterStrategy interface, and use it on FilteringBatchMessageListenerAdapter instead of consumerRecords.isEmpty().

This way, those who want to be as it is can do so without modifying their codes. Meanwhile, it provides a choice for users who have considered this to be an issue until now.

What do you think? When you have free time, take a look please 🙇‍♂️

[chickenchickenlove]

@artembilan , thanks for your comments 🙇‍♂️

We need some tests for this new feature. And, yes, I also think that this should go to 3.3 already . So, please, fix Javadoc respectively. And add some doc, too.

I added a couple of test cases to test new public API. When you have free time, please take a look 🙇‍♂️

[chickenchickenlove]

I added spring-kafka-docs as well to Filtering Messages section.

[chickenchickenlove]

Hi, @artembilan ! long time no see 😄. Thanks for notifying me! I followed your comments about doing rebase and add description! Please take another look, when you have free time 🙇‍♂️

[artembilan]

Merged as https://github.com/spring-projects/spring-kafka/commit/f91f8a918688ccf855207aa66ef5703da0568770.

@chickenchickenlove ,

thank you very much for the contribution (again)!

A couple notes:

• it is better to use @SuppressWarnings("unchecked") if you cannot fix cast warnings in the tests.
• Please, consider to use in the future an issue number for the branch to PR eventually, e.g. this one could be GH-2806. It is much easier to handle that on merge then
• The mock() is there (without Class arg) for those to mitigate generics warning and easier to code.
• The final is not necessary in most cases for local variables. Java makes them "effectively final" anyway.

[artembilan]

Well, merged, so closing

[chickenchickenlove]

Thanks for your time and looking this PR! Also, i will keep your advice in my mind and will do better next contribution 👍 Thanks a lot, again 🙇‍♂️