What is robots.txt?

Text file without format that is used to give instructions to Google crawlers or spiders or those of other search engines to inform them of which areas of a website need to be crawled or indexed and which do not.

These files are uploaded to a website to prevent certain parts of that website from appearing in search results, or to avoid the indexing of duplicated content, or to inform Google of its sitemap, including its URL in the file.

 
Download our report on UX trends 2018
Download report See more reports

Other internet
related terms

(

A

B

C

D

E

P

Q

R

S

s

S

T

U

V

W

X

Y