Set an environment’s visibility to search engines

When you have development environments, you don’t want search engines indexing them and diluting the SEO of your production site.

Search engine indexers are told to ignore all development environments. You can change the setting for Production environments that don’t have a custom domain. can’t guarantee that indexers follow the instructions. If you’re concerned about access, set up HTTP access control.

To change the environment’s visibility to search engines, follow these steps:

  • Select the project where you want to change visibility.
  • From the Environment menu, select the environment you want to change.
  • Click Settings.
  • In the row with Hide from search engines, click Edit .
  • Select or clear the Hide from search engines checkbox.

Run the following command:

platform environment:info -e <ENVIRONMENT_NAME> restrict_robots true

How it’s done 

Search engines are turned away from environments by including a X-Robots-Tag header:

X-Robots-Tag: noindex, nofollow

That tells search engine indexers to not index these sites and not traverse links from these sites. This helps keep non-Production sites out of search engine indexes.

Override it 

You can also send instructions to search engine indexers using a robots.txt file. Your app can serve this as a static file from its disk or as a dynamic response from its passthru. Control either with the location section of your app configuration.

If your robots.txt file includes instructions to ignore a page, search engine indexers may ignore it even if you have configured to not send the header.