THINK FIRST·CODE LATER

← All labs

Read robots.txt before you crawl

Problem

Standard input: the content of a robots.txt file, a line ---, a line agent: NAME, then URLs (one per line).

Use urllib.robotparser.RobotFileParser (parse, can_fetch, crawl_delay) to print, for each URL, ALLOW url or DENY url (DENY followed by two spaces), then 2 of 4 URLs allowed for NAME and either crawl delay: 5 s followed by minimum time for the allowed URLs: 5 s ((allowed − 1) × delay, at least 0) or crawl delay: not specified (be polite anyway).

Data for this lab

Your program reads its data from standard input. Download the file, then give it to your program as shown below. Start with data set 1: its expected output is given.

v4-4-data-1.txt

User-agent: *
Disallow: /admin/
Disallow: /api/private/
Crawl-delay: 5

User-agent: BadBot
Disallow: /
---
agent: CityPulseStudentBot
https://data.example.org/stations
https://data.example.org/admin/users
https://data.example.org/api/private/keys
https://data.example.org/api/public/status
What your program must print for v4-4-data-1.txt
ALLOW https://data.example.org/stations
DENY  https://data.example.org/admin/users
DENY  https://data.example.org/api/private/keys
ALLOW https://data.example.org/api/public/status
2 of 4 URLs allowed for CityPulseStudentBot
crawl delay: 5 s
minimum time for the allowed URLs: 5 s

Run your program first, then open this and compare line by line: your output must contain these lines exactly.

How to give the file to my program

Terminal (macOS, Linux, or Windows Command Prompt), in the folder that holds both files:

python my_lab.py < v4-4-data-1.txt

Windows PowerShell (the default terminal of VS Code on Windows) does not understand <:

Get-Content v4-4-data-1.txt | python my_lab.py

PyCharm: Run → Edit Configurations → tick Redirect input from and choose the file.

Jupyter: put this at the top of the cell that holds your program:

import sys
sys.stdin = open("v4-4-data-1.txt", encoding="utf-8")

Write it here, in Jupyter or in your IDE, then paste it. Run it on the sample input yourself before comparing. Your code stays in your browser — it is never sent to or stored on the server.