inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[INLONG-6151][Manager] Add optional data cleansing task to control storage size, optimize query indexes

Open woofyzhao opened this issue 3 years ago • 0 comments

  • Fixes #6151

Motivation

  • By default inlong manager only performs logical deletion.
  • In cases where there are huge amount of offline periodic tasks it may be required to cleanse logically deleted data to reduce storage size.
  • We may as well use external scripts and add a crontab job to do it, but also introduces considerable maintenance overheads. Besides in cloud environment the crontab job (CronJob) might not be convenient.

Modification

  • Provide an optional periodic task to do the data purging in the hope that no external scripts or configuration maintenance occur.
  • Optimize some table indexes to speed up query.

woofyzhao avatar Oct 13 '22 07:10 woofyzhao