feast icon indicating copy to clipboard operation
feast copied to clipboard

feat: Set optional full-scan for deletion

Open phil-park opened this issue 1 year ago • 1 comments

What this PR does / why we need it:

In this link (https://github.com/feast-dev/feast/pull/3857), I added function to perform a full-scan and delete data to save Redis memory. However, full-scan in Redis does not work beautifully, takes too long at production scale, and often fails in CI that performs feast-apply. So, I think it would be a good idea to add the 'full_scan_for_deletion' option in redis so that the user can select and delete it.

Which issue(s) this PR fixes:

Fixes

phil-park avatar May 09 '24 07:05 phil-park

I think this might be valuable in the interim, although you might want to chime in the discussion here as well #4133. The implicit consensus from there is to add a feast cleanup command that will delete all redundant keys from online store (keys that belong to deleted tables and also keys containing feature values that have expired according to ttl).

tokoko avatar May 09 '24 08:05 tokoko