crawler directive

Instructions to the crawler regarding what you want it to crawl and index on your site.