Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

250ms matters. That's 1/4 of a second. It matters for a crawler tremendously.

DNS query sizes are wrong. I'd double each. But still, inexpensive from a bandwidth standpoint. I get it.



why would it matter at all, given a crawler hits many pages on the same hostname? the fact the initial request takes 250ms more is meaningless.

when you also take into effect that crawlers are either: bound on sleep() if they're friendly, bound on cpu if doing processing and you have a lot of money for bandwidth, or if you don't have much money, bound on bandwidth.

and given that crawlers tend to be massively parallelised, the DNS query could take minutes and you really still wouldn't care...

(go and read up on Amdahl's law)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: