robots.txt

Files that suggest which parts of your site search engines should and shouldn’t crawl.