Robots.txt

Robots.txt is a text file belonging to a group of web protocols called Robots Exclusion Protocol or REP. This file commands search engine bots like Googlebots on what pages to crawl and index. Robot.txt helps websites avoid being bombarded by too many requests. It plays a huge role in having a good SEO score.