Can't dequeue corrupted pickle item

[PhracturedBlue]

Persist-Queue 0.5.0 I don't know how I got into this state, but I have a corrupted queue that I cannot remove items from.

>>> import persistqueue
>>> q = persistqueue.Queue("/tmp/elastic.e6c64f88.queue")
>>> q.qsize()
234
>>> q.get()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/maap/.venv/lib/python3.7/site-packages/persistqueue/queue.py", line 216, in get
    item = self._get()
  File "/var/lib/maap/.venv/lib/python3.7/site-packages/persistqueue/queue.py", line 230, in _get
    data = self.serializer.load(self.tailf)
  File "/var/lib/maap/.venv/lib/python3.7/site-packages/persistqueue/serializers/pickle.py", line 28, in load
    return pickle.load(fp)
_pickle.UnpicklingError: could not find MARK

I can clearly wipe the queue, but I don't think this should be able to happen in the 1st place

[peter-wangxu]

is there any power failure or disk failure on your machine?

[PhracturedBlue]

It is very unlikely there was a hardware reboot, but I can't rule it out. This software runs on hundreds of machines in an enterprise environment and the issue was several months ago. In 99% of our systems, a reboot would wipe /tmp, so the queue qouldn't be there, but there is a small chance we had a reboot on one of the systems that preserves /tmp. There is no realistic possibility of us having a disk failure as that would trigger alerts and a hardware service request.

Regardless, I don' t think this behavior would be expected. Either the quese should be corrupt enough that it is unreadable, or it should be accessible.

[peter-wangxu]

Yes, it's not acceptable that power failure crash the queue meta info file.

I saw another similar issue occurred, will investigate more on this.