Uploaded image for project: 'Planet4'
  1. Planet4
  2. PLANET-6076

Add robots.txt at the root of dev and stage domains

    XporterXMLWordPrintable

    Details

    • Type: Task
    • Status: CLOSED
    • Priority: Should have
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.54.0
    • Labels:
      None
    • Story Points:
      3
    • Sprint:
      Sprint #159
    • Section:
      Cloudflare, SEO
    • Track:
      Development

      Description

      All dev and stage sites have a robots.txt file that prohibits bots from indexing these websites. This is configured at planet4-robots.

      eg. https://www-dev.greenpeace.org/international/robots.txt

      Despite that, some of these website still poping up on search results:
      https://www.google.com/search?q=www-dev.greenpeace.org

      This can be the result of a missing robots.txt file from the root of the www-dev domain. We could utilize a Cloudflare worker that could just handle this request.

        Gliffy Diagrams

          Attachments

            Activity

              People

              Assignee:
              nroussos Nikos Roussos
              Reporter:
              nroussos Nikos Roussos
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: