dubinc
[ feature ] add option to noindex short links
I have been using http://dub.co for creating short links for my app Pika, the issue i'm facing is that since it's as subdomain it's being indexed by Google and I'm getting warnings in Google Search console. These short links are not in the sitemap or anywhere, but since these links are there on the home page I assume Google must've crawled from there
Is there any way I can prevent my short urls to be indexed? If not it would be great to have a noindex option for custom domain short links on [http://dub.shco(https://t.co/5MM1spavfU)
@rishimohan these two links might help you for preventing indexing:
https://chat.openai.com/share/fdf54e64-24a4-44e0-992a-23810fbdfa14
https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results#:~:text=You%20can%20prevent%20new%20content,to%20the%20content's%20head%20HTML.
@steven-tey does dub use bitly under the hood? I am asking this because I am searching for the URL shortener's main script to add the indexing prevention option.
Ref: https://github.com/steven-tey/dub/blob/main/apps/web/app/api/callback/bitly/route.ts
@Yh010 no, Dub does not use Bitly under the hood – that is an entirely unrelated route for users to authenticate with Bitly for the Bitly importer.
To customize this indexing behavior, we'll need to programmatically generate this robots.ts file.
@steven-tey would the links now be noindex by default or is there any option for it?
they will all be noindex by default (except for the root domain) – we haven't had users ask for option to index short links, but if we do get more of it in the future we'll add an option for it