Allow refreshing Geo-Location Processor files from an S3 bucket
Description
Geo-Location processor city and ASN database files can now be stored in an S3 bucket and pulled onto the Graylog server's filesystem automatically. These changes rely on the AWS SDK DefaultCredentialsProvider and not any settings that may be configured in the Graylog AWS plugin configuration. Using the S3 bucket option is disabled by default and not required. Users who would like to continue to manage their on disk files on their own may continue to do so without making any changes to their processor configuration settings.
Motivation and Context
Currently users must update the database files on each of their Graylog server's filesystems which can be a cumbersome and error prone process. This change allows users to put their files in one place and have their Graylog nodes pull in those files automatically.
How Has This Been Tested?
Tested locally in a development environment using the default credential profiles file option for the DefaultCredentialsProvider (see AWS documentation). Will continue testing with the other options to confirm they work as well.
Screenshots (if appropriate):
Types of changes
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Refactoring (non-breaking change)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
- [X] My code follows the code style of this project.
- [X] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [X] I have read the CONTRIBUTING document.
- [ ] I have added tests to cover my changes.