openwhisk
openwhisk copied to clipboard
Normalize Kafka configuration units to bytes
Normalize Kafka configuration units to bytes for message and request size parameters.
Description
Kafka configuration units related to size are in bytes. However, ByteSize supports various units. I believe it's better to normalize the unit to bytes to help reduce human errors.
For example, if max-message-bytes is set with an MB unit, the topic configuration could mistakenly interpret it as 3 bytes instead of 3 MB:
// application.conf
whisk {
kafka {
...
events {
segment-bytes = 536870912
retention-bytes = 1073741824
retention-ms = 3600000
max-message-bytes = 3 m
}
Currently, several topics set max.message.bytes as ${whisk.activation.kafka.payload.max} + ${whisk.activation.kafka.serdes-overhead}. Fortunately, this ensures the unit remains in bytes when summing the two values.
Despite this, I believe normalizing the unit to bytes will help prevent unexpected bugs.
Related issue and scope
- [ ] I opened an issue to propose and discuss this change (#????)
My changes affect the following components
- [ ] API
- [ ] Controller
- [x] Message Bus (e.g., Kafka)
- [ ] Loadbalancer
- [ ] Scheduler
- [ ] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [ ] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
Types of changes
- [x] Bug fix (generally a non-breaking change which closes an issue).
- [ ] Enhancement or new feature (adds new functionality).
- [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
Checklist:
- [x] I signed an Apache CLA.
- [x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
- [ ] I added tests to cover my changes.
- [ ] My changes require further changes to the documentation.
- [ ] I updated the documentation where necessary.