Add an option to turn off respecting robots.txt

[mmuehlfeldRH]

I honor that linkcheck respects robots.txt, but it makes the tool mostly useless for me. A link checker should be able to check all links on the specified URL.

How about continuing respecting robots.txt by default but adding a command-line option to turn off this behavior, if needed? For example: --ignore-robots-txt