Spammy logs when using memberlist

[jtlisi]

Describe the bug

I'm seeing a lot of spam in the logs when running memberlist. I haven't taken the time to track down the issue but I wanted to open an issue I can come back to.

For example:

ts=2021-04-23T01:38:59.534725314Z caller=memberlist_logger.go:74 level=error msg="msg type (88) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534741477Z caller=memberlist_logger.go:74 level=error msg="msg type (171) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534754897Z caller=memberlist_logger.go:74 level=error msg="msg type (192) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534785542Z caller=memberlist_logger.go:74 level=error msg="msg type (49) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534804166Z caller=memberlist_logger.go:74 level=error msg="msg type (163) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534822917Z caller=memberlist_logger.go:74 level=error msg="msg type (176) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534849818Z caller=memberlist_logger.go:74 level=error msg="msg type (73) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534866896Z caller=memberlist_logger.go:74 level=error msg="msg type (164) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.53488881Z caller=memberlist_logger.go:74 level=error msg="msg type (45) not supported from=10.132.100.17:7946"
ts=2021-04-23T01:38:59.534907823Z caller=memberlist_logger.go:74 level=error msg="msg type (86) not supported from=10.132.100.17:7946"

ts=2021-04-23T01:39:01.738998232Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"
ts=2021-04-23T01:39:01.739029408Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"
ts=2021-04-23T01:39:01.739057656Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"
ts=2021-04-23T01:39:01.739087467Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"
ts=2021-04-23T01:39:01.739122273Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"
ts=2021-04-23T01:39:01.739151519Z caller=memberlist_logger.go:74 level=error msg="Failed to decode ping request: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=10.134.7.18:7946"

[pstibrany]

This is interesting, and not something that Cortex logs, but coming from memberlist library. Is it coming from that specific IP address only? Can you share your membelist config (wondering if you use new TLS support for example)?

[cabrinha]

could this be prometheus trying to scrape the port? What is at 10.134.7.18 and 10.132.100.17?

[cabrinha]

I'm getting the same thing:

{"caller":"memberlist_logger.go:74","level":"error","msg":"Failed to decompress payload: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=192.168.241.136:7946","ts":"2021-11-10T18:51:48.803036073Z"}
{"caller":"memberlist_logger.go:74","level":"error","msg":"msg type (172) not supported from=192.168.241.136:7946","ts":"2021-11-10T18:51:48.803056321Z"}
{"caller":"memberlist_logger.go:74","level":"error","msg":"Failed to decode alive message: codec.decoder: Only encoded map or array can be decoded into a struct. (valueType: 2) from=192.168.241.136:7946","ts":"2021-11-10T18:51:48.80307918Z"}

The 192.168.83.198 address was actually another distributor pod ... which is odd. The above logs came from a different distributor as well.

I wish these log messages would either not exist or show the actual message.