graylog2-server icon indicating copy to clipboard operation
graylog2-server copied to clipboard
verified publisher icon Graylog2

Add data node heap check

Open moesterheld opened this issue 1 year ago • 2 comments

Description

This adds a check for the configured heap size of the data node's OpenSearch vs. the free memory available after starting the OpenSearch process. It deducts a buffer of 2GB from the free memory and if more than 2x the configured size would be available, a notification is shown in Graylog that there are nodes which could potentially be optimized.

/nocl yet

Motivation and Context

resolves https://github.com/Graylog2/graylog-plugin-enterprise/issues/6641

How Has This Been Tested?

unit tests, change heap size manually and set to a value triggering the notification

Screenshots (if appropriate):

Types of changes

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Refactoring (non-breaking change)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • [x] My code follows the code style of this project.
  • [ ] My change requires a change to the documentation.
  • [ ] I have updated the documentation accordingly.
  • [x] I have read the CONTRIBUTING document.
  • [x] I have added tests to cover my changes.

moesterheld avatar Aug 13 '24 11:08 moesterheld

@todvora @tellistone do you have a better idea for determining when to trigger the notification or do you think this mechanism will do?

moesterheld avatar Aug 13 '24 11:08 moesterheld

I think that will work nicely. The notification might recommend the specific remediation, set the HEAP to almost 50% (47% ?) of total memory of the host (that is what AWS Opensearch auto-tune does)

tellistone avatar Aug 14 '24 16:08 tellistone

Perhaps round it to one decimal place.

We should change the copy of the notification to be more perspective IMO.

Eg. This data node only has 1GB Java Heap assigned, out of a total of 16GB RAM. For production performance, please configure this node to use 8 GB HEAP (50% of RAM).

tellistone avatar Apr 23 '25 16:04 tellistone

image

@tellistone I adjusted the text and added the recommendation as a note, since people who run non-production environments (like my dev environment in the example) should not set the heap to a value that would prevent data node startup

moesterheld avatar Apr 25 '25 08:04 moesterheld

Thanks - might be also worth putting line breaks between the three sections of copy so its easier to read

tellistone avatar Apr 25 '25 09:04 tellistone

Thanks - might be also worth putting line breaks between the three sections of copy so its easier to read

Added paragraphs to make it more readable

moesterheld avatar Apr 25 '25 12:04 moesterheld