php-spider icon indicating copy to clipboard operation
php-spider copied to clipboard

Add prefetch filter to cache downloads with a max age

Open mvdbos opened this issue 2 years ago • 0 comments

With this prefetch filter in place, skip fetching resources that are already downloaded and younger than max age. This requires that downloads are not segmented per spider id. A simple option is to set the same spider id for runs where you want to use the cache.

mvdbos avatar Feb 22 '24 16:02 mvdbos