A crawler starts from one page, follows its links, then the links of those pages, and so on — and must never visit a page twice, even though pages link to each other in circles.
The first line gives the start page and the maximum number of pages to visit. Each following line describes a page of a pretend web: page: link1 link2 ….
Write MiniCrawler with:
- a
Map<String, List<String>>for the web; - a queue of pages waiting to be visited;
- a set of the pages already seen (visited or waiting) — a link is queued only if
seen.add(link)returnstrue.
A link to a page that has no line is a broken link. Print the pages in the order visited, the number of links examined, the pages still waiting when the limit stopped the crawl, the broken links and the pages never reached (the last two in alphabetical order). For the input
home 10
home: news sports about
news: home world tech
sports: home scores
about: home contact
world: news
tech: news gadgets ai
scores: sports
ai: tech home
archive: home old
the program prints
Visited (8): [home, news, sports, about, world, tech, scores, ai]
Links examined: 17
Still waiting: []
Broken links: [contact, gadgets]
Pages never reached: [archive]