Google's Matt Cutts On How They Evaluate New Search Algorithms (original) (raw)

Google Head of Search Spam Matt Cutts posted a video today answering how Google goes about evaluating which new search algorithms they use and which they throw away or adapt.

The question was posed by James Foster of Sydney, Australia who asked:

What are some of the metrics that Google uses to evaluate whether one iteration of the ranking algorithm is delivering better quality results to users than another?

Matt Cutts breaks it down to about three steps of the evaluation process:

(1) They test the algorithm offline, benchmarking how the results rank with the new algorithms and if the URLs are higher quality than the previous algorithms in place. The quality is based on how the search quality raters rate the URLs in previous cases. If the URLs were unrated, Google can request these raters to rate the new URLs or compare the old search results to this new test set. Then based on those metrics, Google may decide to move the test to the next phase.

(2) Live tests, where Google will sample a subset of real live searchers and give them the new results with the new set of test algorithms. If Google sees a higher click rate on the new search results, it may imply that the new results are better than the older ones. This is not always the case, specifically with webspam, Cutts said. But in general, the more clicks on a specific search result page, the better quality the results.

(3) Then the Google Search Quality Launch Committee has the ultimate say on if the algorithm goes live to the public or not.

Matt said Google has this down to a “pretty good system” but every now and then they need to refine some of the processes within this workflow.

Here is the video:

New on Search Engine Land

About the author

Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.

In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the "US Search Personality Of The Year," you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O'Clock.

Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.