Using robots.txt

As Open Orchestra is a multisite platform, you will have to manage several robots.txt files for a single installation. To simplify the maintenance of their content, they can be directly edited in the Back Office. But contribution is only the first step. To be available, these information require to be dumped in files and to be correctly served to the client.

Creation of robots.txt files with Open Orchestra is achieved in 3 steps:

Files contribution

Contributing the content of a robots.txt file is straight forward. As it is specific to each website, you can do it in the website edit form, located in the Administration submenu. Fill the matching textarea: the content of the robots.txt file will be exactly what is written here.


Files generation

Once contributed, the content of the robots.txt is only known from the database. To serve it to a client requesting it, two approaches are possible: use a controller action looking in the database for the information matching the current site, or serve a previously generated file containing these information.

To enhance perfomances, Open Orchestra choose the second option. This solution prevents any PHP action and database access. As these files are static they can easily be cached by a reverse proxy.

Once a robots.txt data is contributed, you have to manually dump it into a file. Open Orchestra provides a console command, available on your Front installation:

app/console orchestra:robots:generate [–siteId=SITEID]

This command dumps one or all the robots.txt files depending on the siteId parameter presence. To be accessible by web client, the generated files are stored in the web directory of your application. Each file is stored in a sub-directory named by the site Id. For instance if you have three sites whose id are ‘site-1’, ‘site-2’ and ‘site-3’, the robots.txt files will be dumped here:

  • web/site-1/robots.txt
  • web/site-2/robots.txt
  • web/site-3/robots.txt

A good practice is to use that command in a cron, to refresh periodically the content of the files.

Files routing

When a client requests a robots.txt, it should be located at the root of the domain. But the files are generated somewhere else, in a subfolder named by the site id. As a result, the webserver must be configured to redirect to the matching file: this can be done by a rewrite rule . The type of configuration to tweak depends on you server type, for instance on Apache this is done via the Virtual Host mechanism.

Here is a configuration example for Apache, in the case of a site with id ‘my-site’:

Once Apache is reloaded, the rewrite rule is used and the robots.txt file is accessible.

Every further modification will modify the file content but not its name, so you don’t have to modify the rewrite rule. Since then you can update the file periodically without having to restart the webserver.