Uploaded image for project: 'Planet4'
  1. Planet4
  2. PLANET-6076

Add robots.txt at the root of dev and stage domains

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Should have Should have
    • 2.54.0
    • None
    • 3
    • Cloudflare, SEO
    • Sprint #159

      All dev and stage sites have a robots.txt file that prohibits bots from indexing these websites. This is configured at planet4-robots.

      eg. https://www-dev.greenpeace.org/international/robots.txt

      Despite that, some of these website still poping up on search results:
      https://www.google.com/search?q=www-dev.greenpeace.org

      This can be the result of a missing robots.txt file from the root of the www-dev domain. We could utilize a Cloudflare worker that could just handle this request.

            nroussos Nikos Roussos
            nroussos Nikos Roussos
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: