inlong
inlong copied to clipboard
[INLONG-6151][Manager] Add optional data cleansing task to control storage size, optimize query indexes
- Fixes #6151
Motivation
- By default inlong manager only performs logical deletion.
- In cases where there are huge amount of offline periodic tasks it may be required to cleanse logically deleted data to reduce storage size.
- We may as well use external scripts and add a crontab job to do it, but also introduces considerable maintenance overheads. Besides in cloud environment the crontab job (CronJob) might not be convenient.
Modification
- Provide an optional periodic task to do the data purging in the hope that no external scripts or configuration maintenance occur.
- Optimize some table indexes to speed up query.