A scraper written a few years ago (in the style of popular GitHub projects) collects a film list:
url = "https://www.imdb.com/list/ls068082370/"
def all_page_link(start_url):
all_urls, url = [], start_url
while url is not None:
all_urls.append(url)
soup = BeautifulSoup(requests.get(url).text, "html.parser")
next_links = soup.find_all(class_="flat-button lister-page-next next-page")
url = None if len(next_links) == 0 else "https://www.imdb.com" + next_links[0].get("href")
return all_urls
for url in all_page_link(url):
soup = BeautifulSoup(requests.get(url).text, "html.parser")
for link in soup.find_all(class_="lister-item-content"):
ID = int(link.find("span", {"class": "lister-item-index unbold text-primary"}).text[:-1])
... # then INSERT INTO movies VALUES ('" + title + "', ...) in MySQL
- List every technical, legal and data-quality problem.
- Propose a better acquisition strategy for the same visualization goal ("box-office totals by genre and director").
- Write the skeleton of a robust, polite version for the case where scraping a permitted site is still necessary.