Boxes And Arrows : The Design Behind the Design

Article Idea:

How to Evaluate Search Relevancy and Precision

suggested by John Ferrara on 2008/04/26

Website search is often the most highly trafficked section of a website, and the one where the information architect bears the least influence. There aren’t enough tools in our hands for us to even appraise the general quality of the search experience, much less strategize ways to improve it. As the user’s advocates, we very much belong in this space. I’m proposing a pair of articles that will explain two methods for quantitatively measuring the quality of your website’s search engine results using data readily available in the search server logs. The first article will explain how to perform a quick and easy relevancy test that yields concise, bottom-line scores of how reliably the search engine returns the best matches first. The second will explain how to conduct a more advanced and insightful evaluation of the results’ precision – the proportion of relevant and irrelevant matches clustered at the top of the list. Both articles will be highly pragmatic, providing readers with: 1. step-by-step instructions for running each of the methods 2. downloadable Excel spreadsheets that do the hard work of processing the data gathered 3. guidance for presenting the results to project teams and management 4. examples of how the findings from both methods can be used to directly guide optimization and strategize functional enhancements to the site. With more practitioners using such instruments, we will have the opportunity to extend our reach within a function that’s so central to the user experience.