-
Task
-
Resolution: Fixed
-
Should have
-
None
-
3
-
Cloudflare, SEO
-
Sprint #159
All dev and stage sites have a robots.txt file that prohibits bots from indexing these websites. This is configured at planet4-robots.
eg. https://www-dev.greenpeace.org/international/robots.txt
Despite that, some of these website still poping up on search results:
https://www.google.com/search?q=www-dev.greenpeace.org
This can be the result of a missing robots.txt file from the root of the www-dev domain. We could utilize a Cloudflare worker that could just handle this request.