Email the TA (gernhard@cs.rochester.edu) your answers to pre-assignment questionnaire by September 16 (Monday).
Demo on September 23 (Monday) or September 24 (Tuesday).
Please direct your questions to the TA about this assignment.
Your crawler should support focused crawling. In the first mode, you
should limit your page crawling within a given Web domain.
Specifically, only pages whose Web host site has a certain suffix
(e.g., cs.rochester.edu
or rochester.edu
)
should be crawled.
The second mode of focused crawling is to try to crawl pages on a specific topic (e.g., "golf" or "computer science"). Obviously it is impossible to know for certain whether a page is on a certain topic before you download it. But as we discussed in class, you can get heuristics from the anchor text and surrounding text of a hyperlink. The effectiveness of focused crawling can be measured by the proportion of crawled pages that are indeed relevant to the topic.
Your crawler should not crawling the same URL more than once. Without this support, your crawling may fall into a loop that repeatedly downloads a particular set of pages for ever. You should also perform reasonable URL normalization to reduce redundant crawling.
You must follow the
Robot Exclusion Standard
in your crawling.
To simplify your work, you only need to parse and observe the
Disallow
directives.
To limit the resource usage of your crawling, you should enable a
configurable sleep time between consecutive Web page downloads.
In your testing, you should sleep at least one second between
consecutive downloads.
You should carefully implement these politeness measures.
To avoid undesirable impact on external Web sites, you should not test on
pages outside the department until you are confident about your
implementation of these measures.
In class we have discussed optimizations for crawling high-quality pages and machine learning supported focused crawling. You can also find many additional crawling optimizations beyond what we discussed in class. You do not need to implement these optimizations for completing the assignment. However, if you have interests and time, we encourage you to explore such optimizations and we will consider assigning extra credit if you make a successful implementation and can demonstrate its effect.
We have discussed parallel Web crawling in class. For this assignment, however, you do not need to support parallel crawling.
www.cs.rochester.edu
" and count the number
of hyperlinks in the crawled page.
www.cs.rochester.edu/dept/calendar/print.php
"?
Why?
Failure to email your answers on time will result in a 10% loss in your assignment grade. We actually will not grade your answers. This is simply to ensure that you do not wait to the last minute before working on the assignment. You will not receive any feedback on your answers. However, you are welcome to visit the TA's office hours and discuss these questions.