Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Breadth-first search means crawling all of the links on the page and adding all of the links on the child pages to the queue at once rather than drilling down on one link-path first before moving to the other links on the first page. There's probably some highly non-obvious reason for crawling by url length.


The thing is, the deeper you are, the longer are the urls, so if you do a breadth first search, you are more likely to visit short urls first.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: