View Sidebar

Post Tagged with: robots

Robots & Sitemaps

Robots & Sitemaps

robots.txt This file is used to inform the search engine crawler what folder and files of our site you want to crawl or donā€™t want to. Robot file is normally located on the root of the website. A typical robots file would contain following to allow crawler to view entire […]

February 20, 2013Read More