openwhisk icon indicating copy to clipboard operation
openwhisk copied to clipboard

Normalize Kafka configuration units to bytes

Open dorbae opened this issue 1 year ago • 0 comments

Normalize Kafka configuration units to bytes for message and request size parameters.

Description

Kafka configuration units related to size are in bytes. However, ByteSize supports various units. I believe it's better to normalize the unit to bytes to help reduce human errors.

For example, if max-message-bytes is set with an MB unit, the topic configuration could mistakenly interpret it as 3 bytes instead of 3 MB:

// application.conf
whisk {
  kafka {
     ...
     events {
                segment-bytes   =  536870912
                retention-bytes = 1073741824
                retention-ms    = 3600000
                max-message-bytes = 3 m
            }

Currently, several topics set max.message.bytes as ${whisk.activation.kafka.payload.max} + ${whisk.activation.kafka.serdes-overhead}. Fortunately, this ensures the unit remains in bytes when summing the two values.

Despite this, I believe normalizing the unit to bytes will help prevent unexpected bugs.

Related issue and scope

  • [ ] I opened an issue to propose and discuss this change (#????)

My changes affect the following components

  • [ ] API
  • [ ] Controller
  • [x] Message Bus (e.g., Kafka)
  • [ ] Loadbalancer
  • [ ] Scheduler
  • [ ] Invoker
  • [ ] Intrinsic actions (e.g., sequences, conductors)
  • [ ] Data stores (e.g., CouchDB)
  • [ ] Tests
  • [ ] Deployment
  • [ ] CLI
  • [ ] General tooling
  • [ ] Documentation

Types of changes

  • [x] Bug fix (generally a non-breaking change which closes an issue).
  • [ ] Enhancement or new feature (adds new functionality).
  • [ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

  • [x] I signed an Apache CLA.
  • [x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
  • [ ] I added tests to cover my changes.
  • [ ] My changes require further changes to the documentation.
  • [ ] I updated the documentation where necessary.

dorbae avatar Feb 10 '25 16:02 dorbae