User-agent: * Disallow: /admin/ Disallow: /api/private/ Crawl-delay: 5 User-agent: BadBot Disallow: / --- agent: CityPulseStudentBot https://data.example.org/stations https://data.example.org/admin/users https://data.example.org/api/private/keys https://data.example.org/api/public/status