New sitemap features in robots.txt

Google, Microsoft and Yahoo just released the first result from the joint venture sitemap protocol. For a while now, you have been able to log in to Google’s webmaster tools and from there specify your sitemap XML file. By doing so, you tell Google the location of all the pages on your site so that you make sure that everything gets crawled and indexed.

Yahoo and Microsoft wanted the same functionality but instead of creating their own format, the joined forces with Google to create a standardized format. Now Ask.com announces they also will support the format.

The result is a set of extensions but most importantly the ability of autodiscovery, so you no longer have to manually log in and update the sitemap. Because all search engines support the robots.txt document, they decided to let you specify the path to the sitemap there.

Basically, you just add a single line to the robots.txt that specifies the absolute path to the sitemap. Here is an example from my robots.txt:

sitemap: http://blog.madskristensen.dk/sitemap.axd

If you use BlogEngine.NET, the sitemap is called /sitemap.axd.