# Robots Exclusion Protocol / Bot directives User-agent: * # Match all bots Allow: /pipermail/ Disallow: /mailman/ Crawl-delay: 5