# Exclude Files From All Robots: # The "User-agent: *" means this section applies to all robots. # The "Disallow: /" tells the robot that it should not visit any pages on the site. User-agent: * # Disallow: /cgi-bin/ # Disallow: /tmp/ # End robots.txt file