-
Type:
Task
-
Status: CLOSED
-
Priority:
Should have
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.54.0
-
Labels:None
-
Story Points:3
-
Sprint:Sprint #159
-
Section:Cloudflare, SEO
-
Track:Development
All dev and stage sites have a robots.txt file that prohibits bots from indexing these websites. This is configured at planet4-robots.
eg. https://www-dev.greenpeace.org/international/robots.txt
Despite that, some of these website still poping up on search results:
https://www.google.com/search?q=www-dev.greenpeace.org
This can be the result of a missing robots.txt file from the root of the www-dev domain. We could utilize a Cloudflare worker that could just handle this request.