> When Google indexes this site will we successfully compute all possible chess moves?
How do spiders know when to stop spidering when they keep getting original content? I assume there's a Gordian solution to the Halting problem like a limit to bytes or seconds. But if you applied the same rules to ebay.com and val.town that doesn't scale.
Big spiders neither stop nor start. If each webpage gives you on average 200 new urls, you need to aggregate them, calculate ranks and schedule the top 0.5% for crawling. Crawling capacity is constant, the only question is the quality of your ranking formula.
There's a lot of heuristics. Big sites get more crawl time. Some crawlers will back off if pages are slow. There's usually some sort of 'interestingness' calculation, so repetitive content won't get crawled as much.
How do spiders know when to stop spidering when they keep getting original content? I assume there's a Gordian solution to the Halting problem like a limit to bytes or seconds. But if you applied the same rules to ebay.com and val.town that doesn't scale.