THINK FIRST·CODE LATER

← All labs

Modernize a legacy scraper

Problem

A scraper written a few years ago (in the style of popular GitHub projects) collects a film list:

url = "https://www.imdb.com/list/ls068082370/"
def all_page_link(start_url):
    all_urls, url = [], start_url
    while url is not None:
        all_urls.append(url)
        soup = BeautifulSoup(requests.get(url).text, "html.parser")
        next_links = soup.find_all(class_="flat-button lister-page-next next-page")
        url = None if len(next_links) == 0 else "https://www.imdb.com" + next_links[0].get("href")
    return all_urls

for url in all_page_link(url):
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    for link in soup.find_all(class_="lister-item-content"):
        ID = int(link.find("span", {"class": "lister-item-index unbold text-primary"}).text[:-1])
        ...  # then INSERT INTO movies VALUES ('" + title + "', ...) in MySQL
  1. List every technical, legal and data-quality problem.
  2. Propose a better acquisition strategy for the same visualization goal ("box-office totals by genre and director").
  3. Write the skeleton of a robust, polite version for the case where scraping a permitted site is still necessary.

Work it out on paper, in a document or here, then compare with the model answer. Your answer stays in your browser — it is never sent to or stored on the server.