It seems to me that a way to potentially creep up on google would be to make a specialist search engine. Some topic area that google is not so useful for searching on. Make a search system that 'understands' the domain in question, allowing it to index and search it really effectively.
As a random example, say, a movies search engine. If it could be consistently more useful than google for finding out information about movies, directors, actors, the movie industry etc., then it could become the 'go to' tool for movie search. Potentially, this success could then be leveraged to move into other areas of search.
What you should make is a specialist search engine for hackers. They're the early adopters in search engines, so if you get them you have the quickest path to getting everyone else.
Imagine how alarmed Google would be if there were a different search engine the top 10k hackers all used, even if they had everyone else. And rightly so.
Heh. I'm not amongst the top 10k, but I've been cursing yesterday, trying to use search to find out in what version / gem of ruby {|[]| } is a valid block (with just being random variable names). And a week before that the same problem when trying to find out what &@/ ment in a mathematica script.
Search engines tend to ignore symbols in normal search, and while you can search for symbols in google's code search, that doesn't include any documentation or pages where they explain a particular syntax.
That's optimistic, but to be successful with "hackers" the search engine would most likely have to steer its user experience in directions that wouldn't appeal to an everyday user. Typically, us geeks want both maximum control and elegant usability, always preferring control to usability. With most users, the inverse is true.
Sure, there are obstacles, but my instincts tell me a clever person could find their way around them. Or at least, considering the size of the prize, the odds of finding a solution makes the problem worth taking a shot at.
Well, you're pretty much describing "vertical" search engines. There are a lot of those these days, and I think they have good chances of being successful, too.
It's like what we're doing over at bug.gd-- building up an error-based search engine. Our upcoming version (we hope to announce in the next month or so) will have a variety of techniques optimized for searching for errors on the web. (E.g. filtering of user-specific "noise" in error messages, better integration into software, etc.) With the added bonus of capturing feedback from users that can be used for future users and for software writers.
We've been seeing some great uptake lately, so I'm bullish on the vertical search space.
We've been debating on a two-column search results page for our redesign, so we're watching cuil's success closely to see how people like that kind of presentation.
This is exactly what Microsoft is doing. Microsoft is trying to tackle search in different verticals: healthvault for health, Farecast (acquired) for travel, Cashback for commerce, etc.
The challenge is that users don't always think "hmm, I'm doing a movie query, let me go to imdb". They go to Google and type everything into one box, and expect to get a link to imdb or wherever.
As a random example, say, a movies search engine. If it could be consistently more useful than google for finding out information about movies, directors, actors, the movie industry etc., then it could become the 'go to' tool for movie search. Potentially, this success could then be leveraged to move into other areas of search.