What is robots.txt?

Text file without format that is used to give instructions to Google crawlers or spiders or those of other search engines to inform them of which areas of a website need to be crawled or indexed and which do not.

These files are uploaded to a website to prevent certain parts of that website from appearing in search results, or to avoid the indexing of duplicated content, or to inform Google of its sitemap, including its URL in the file.

 
Download our trends reports
Marketing trends UX trends Data trends

Other internet
related terms

(

A

B

C

D

E

P

Q

R

S

s

S

T

U

V

W

X

Y