Add an option to disable generation of `robots.txt`

[CSharperMantle]

As this is a sitemap plugin, it would be beneficial if an option was added to make it generate sitemaps only. The generated robots.txt contains only a Sitemap: directive and does not have much effect on regulating spiders.

Background. I am using Jekyll Collections + permalink to collect miscellaneous static files scattered at the project root (e.g. Bing & Google site authentication, CNAME, robots.txt, ...) into a folder, so copying could be deferred to build time. This plugin's version of robots.txt is causing an annoying warning:

Conflict: The following destination is shared by multiple files.
          The written file may end up with unexpected contents.
          [root]/_site/robots.txt
           - robots.txt
           - [root]/_static/robots.txt

If the custom robot.txt is created directly at the project root, no warnings would be generated, and this plugin's version would be overwritten correctly. Yet in my scenario, this approach of piling everything at the root is unluckily what I am avoiding.

A similar request was put forward some time ago:

Same problem here. It would be beneficial to make a parameter to this plugin with the option to disable robots.txt creation.

Originally posted by @FrancescoBonizzi in #292

Yet it had received no responses, and the issue was closed as stale afterward.

[scudo005]

This, and/or having the possibility to specify some robots.txt directives that are always put in the generated robots.txt file. (like, for example, telling Googlebot to not index specific pages)

[YoureIronic]

This, and/or having the possibility to specify some robots.txt directives that are always put in the generated robots.txt file. (like, for example, telling Googlebot to not index specific pages)

https://github.com/jekyll/jekyll-sitemap?tab=readme-ov-file#exclusions and you specify noindex nofollow for a layout