Maintain secondary indices of keys?

[dgilmanAIDENTIFIED]

Our use case is that we have a complex get() function signature and a custom key() to get a sensible cache key out of it. We'd like to also be able to delete keys in bulk along secondary indices. For example, the Django user could be a secondary index, and we'd want to be able to wipe all cache keys for a given user. But it probably wouldn't be too tricky to make it generic as well: e.g. a secondary index on the user id, but also some user category, or a broad category of "everything" if you wanted to purge all keys maintained by cacheback.

What do you think about an API like so:

Job(secondary_indices=[user.id, user.favorite_color, 'all_users']).get(a, b, ...)

(then, later, when it is time to invalidate)

Job().delete_on_index(user.id)  # drop for this user
Job().delete_on_index("red")  # drop for users with favorite color red
Job().delete_on_index("all_users")

The Django cache API doesn't have an atomic way to maintain the list of cached keys. A generic implementation will be racy, but if you're open to allowing specialization for cache backends you could have custom implementations that use the client in a smarter way.

I can't promise I'll have the availability to do this implementation but interested in your feedback. I know you have process_result, but maybe there's an argument here to make it available in the base class? You'd still have to compute your key() a second time if you did it in process_result or extend its function signature. (Or maybe Call could store the key too?)

[stephrdev]

Hi, I'm not sure if that wouldn't be quite a big change to cacheback and I do think that you could solve that easily from "the outside".

My idea would be, that you maintain a cache key per index lets say index:all_users with a random value. In your call to the job get, you'd pass another argument with the current value of the index:all_users cache key. That value would be part of the cacheback-cache-key. If you want to purge all caches for all_users index, you'd simply change the value of the index:all_users cache key.

Hope that helps.

[dgilmanAIDENTIFIED]

I was interested in deleting, not just orphaning/purging the values, though. In my use case we have no use for the old cached value when we want to do this invalidation.