persist-queue icon indicating copy to clipboard operation
persist-queue copied to clipboard

Unique value or methods to check if item already exists

Open VictorKozachek opened this issue 5 years ago • 2 comments

Hi,

Is there any way to control unique value during PUT method or any method to check if item already exists?

Thanks

VictorKozachek avatar Mar 24 '20 16:03 VictorKozachek

Here is something that I do as I start my application to clear any duplicates where I have an associated event and a job type. It could also be used to keep track of entries and check against it. I'm using a SQLiteAckQueue q.ack in this example to clear it, but you could just get(id=row.get("id")) to clear it in the other Queues. Note this uses the PR fork #153 which is being reviewed for merge.

from collections import defaultdict

q.clear_acked_data(max_delete=0, keep_latest=0, clear_ack_failed=True)
q.shrink_disk_usage()

duplicates = defaultdict(set)

rows = q.queue()
for row in rows:
    item = row.get("data")
    if item.get("tool_name") not in duplicates[item.get("event_id")]:
        duplicates[item.get("event_id")].add(item.get("tool_name"))
    else:
        print(
            "Removing duplicate job: "
            + str(item.get("event_id"))
            + " ("
            + item.get("tool_name")
            + ")"
        )
        q.ack(id=row.get("id"))

imidoriya avatar Apr 01 '21 14:04 imidoriya

The UniqueQ or UniqueAckQ only allows for unique items. I should be using that myself. lol

imidoriya avatar Apr 01 '21 15:04 imidoriya