Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.


It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.

If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.


Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.

You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.


Why do you think anyone thinks it's immutable? Do you understand the basic concept of an arms race, or cat and mouse game?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: