persist-queue
persist-queue copied to clipboard
Unique value or methods to check if item already exists
Hi,
Is there any way to control unique value during PUT method or any method to check if item already exists?
Thanks
Here is something that I do as I start my application to clear any duplicates where I have an associated event and a job type. It could also be used to keep track of entries and check against it. I'm using a SQLiteAckQueue q.ack in this example to clear it, but you could just get(id=row.get("id")) to clear it in the other Queues. Note this uses the PR fork #153 which is being reviewed for merge.
from collections import defaultdict
q.clear_acked_data(max_delete=0, keep_latest=0, clear_ack_failed=True)
q.shrink_disk_usage()
duplicates = defaultdict(set)
rows = q.queue()
for row in rows:
item = row.get("data")
if item.get("tool_name") not in duplicates[item.get("event_id")]:
duplicates[item.get("event_id")].add(item.get("tool_name"))
else:
print(
"Removing duplicate job: "
+ str(item.get("event_id"))
+ " ("
+ item.get("tool_name")
+ ")"
)
q.ack(id=row.get("id"))
The UniqueQ or UniqueAckQ only allows for unique items. I should be using that myself. lol