Scott Mansfield
Scott Mansfield
esolang here we come
by host I mean Host header, so really DNS name
I may need to consider a TTFB and whole-message response time as separate metrics. I think the current response time is neither, and I'm unsure of the timing it actually...
crawler-commons has a robots.txt parser: https://github.com/crawler-commons/crawler-commons
This is being done in the terminator project
work done in 22b72fd9f0a43540a39896a8f9e3978a2358503c and 5af862347c29e5d215264b90f61c4bcedb97a62e
crawler-commons may be usable here: https://github.com/crawler-commons/crawler-commons
This is being done in the exo project
work done in 22b72fd9f0a43540a39896a8f9e3978a2358503c 5af862347c29e5d215264b90f61c4bcedb97a62e
The nofollow also implies no HEAD requests to check for Content-Type