User-agent: * Disallow: User-agent: bingbot Crawl-delay: 5 User-agent: googlebot Crawl-delay: 5 User-agent: ahrefsbot Crawl-delay: 5 User-agent: semrushbot Crawl-delay: 5 User-agent: blexbot Crawl-delay: 5 User-agent: mj12bot Crawl-delay: 5