owlcrawler
owlcrawler copied to clipboard
Rescan pages if last fetched date is older than x
This will allow us to fetch new content without having to delete all urls for the site we want to refresh the data.