James Surowiecki believes explains in his book "The wisdom of crowds" that the efficiency of Google is actually a result of the intelligence of its users. It depends on the PageRank algorithm: here the crucial factor for the relevance ranking is how many pages link to another. The Google guys did a great job with this algorithm. But in the end it was us, people who maintain a website, which made Google so successful. In the book some example searches where listed, and in all cases an appropriate (i.e. it serves the requested information) search result was amongst the first few hits. That's my experience as well, I rarely look at search results not on the first page. So, whe don't really care if a search returns more then 10 millions results, or 344.000, or even 423.

One of the talks at the Agile conference, and actually one of the main claims of us Semantic Web folks, is that semantic-enabled technologies could enhance precision and recall. Maybe this is true, maybe we can cut down the 344.000 results to, let's say, 523, and all of them are relevant. But why would the user care? S/he still clicks one of the first 3-5 links, which usually gives him more or less the information he needs. We can argue we have more precise searches, and more relevant results in return. But we can not claim that the user experience has changed at all. From a user's perspective there would have been no improvement (and therefore no incentive for any company to implement Semantic Web technologies).

We shouldn't rely on measures based on precision and recall, but should look (only?) at the results on the first page. Does the ranking work correct? Is the most relevant page really on a prominent place? This information depends on context and varies from user to user. Relevance feedback (swarm intelligence again) might help to collect and analyze information about successful (or failed) searches.

One of the interesting outcomes of the book (for now, I've just finished the first chapter), is the idea that the average of a crowd's opinion is usually the best. Searching and suggesting is context dependent, but according to the book you can actually skip the individual, and look directly at the average of all the feedback. So perhaps we don't really need sophisticated user profiles which track all the user movements (which raise the questions for privacy), but we just skip the user and look directly at the crowd?