Can't restore where blocks are missing

[danielfaulknor]

Describe the bug

Can't restore invalid version after deep-scrub, where metadata/blocks are missing. Benji bails out with a file not found error, even though I've marked the blocks as invalid with a deep-scrub

    INFO: Restoring 10100/80813 blocks (12.5%)
    INFO: Restoring 10504/80813 blocks (13.0%)
   ERROR: Storage backend read failed: Object metadata or data of block 13528 (UID 229a-34d9) not found.
 WARNING: Job executor "Storage-Read" is being shutdown with 70112 outstanding jobs, cancelling them.
   ERROR: InvalidBlockException: Object metadata or data of block 13528 (UID 229a-34d9) not found.
Traceback (most recent call last):
  File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 241, in _read
    data = self._read_object(key)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/benji/lib/python3.11/site-packages/benji/storage/file.py", line 46, in _read_object
    raise FileNotFoundError('File {} not found.'.format(filename))
FileNotFoundError: File /mnt/benji/blocks/6d/94/000000000000229a-00000000000034d9 not found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/benji/lib/python3.11/site-packages/benji/scripts/benji.py", line 357, in main
    func(**func_args)
  File "/srv/benji/lib/python3.11/site-packages/benji/commands.py", line 81, in restore
    benji_obj.restore(version_uid_obj, destination, sparse, force)
  File "/srv/benji/lib/python3.11/site-packages/benji/benji.py", line 656, in restore
    raise entry
  File "/srv/benji/lib/python3.11/site-packages/benji/jobexecutor.py", line 54, in get_completed
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/benji/lib/python3.11/site-packages/benji/jobexecutor.py", line 42, in execute_with_acquire
    return function()
           ^^^^^^^^^^
  File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 279, in job
    return self._read(block_deref, metadata_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/benji/lib/python3.11/site-packages/benji/storage/base.py", line 249, in _read
    raise InvalidBlockException(
benji.storage.base.InvalidBlockException: Object metadata or data of block 13528 (UID 229a-34d9) not found.

To Reproduce

Benji Version 0.17.0

Steps to reproduce the behavior.

• delete some blocks (not what we did, but you asked for reproduction steps)
• deep-scrub the backup
• try to restore

Expected behavior

Restore works, skipping over the missing blocks.