The robots.txt file should be used to tell search engines how to deal with your site. While there are legitimate reasons for excluding pages from robots.txt, do it sparingly: excessive blocking could be the cause of your penalty.
This website uses cookies.