Robot.txt Flaws

What is Robot.txt Flaws?

The robots.txt file should be used to tell search engines how to deal with your site. While there are legitimate reasons for excluding pages from robots.txt, do it sparingly: excessive blocking could be the cause of your penalty.