Boxes and Arrows: Stories by Karl Fast

Recording Screen Activity During Usability Testing

Karl Fast — Mon, 19 Aug 2002 12:00:01 GMT

Recording what users do is a crucial aspect of usability testing. One of the most useful recordings you can make is a video of screen activity, recording everything on the screen, much like a VCR: the mouse moving, pages scrolling, clicking links, typing in the search terms, and so on. Recording screen activity doesn’t necessarily cost much. Three Windows-based software programs—Lotus ScreenCam, TechSmith Camtasia and Hyperponics HyperCam—range between $30 and $150.A visual record of these mouse movements, keystrokes, and other activities is most useful for usability testing. While there is no substitute for good observational skills, it can be difficult to remember everything that happened during the test. Having a visual record not only reminds you of what happened, it allows for more detailed analysis after the test and comparisons between individuals.

Recording screen activity doesn’t necessarily cost much. Three Windows-based software programs—Lotus ScreenCam, TechSmith Camtasia and Hyperponics HyperCam—range between $30 and $150 and all have free trial versions available for download so you can try before you buy. All three offer good performance, but unfortunately, I can only recommend two, since the third is no longer being actively developed by its maker. How to record screen activity Before we get to the review, let’s take a brief look at the three ways of recording screen activity: a camcorder, a VCR, or software. All the tools described in this article use the software approach, but to understand the benefits and drawbacks it’s useful to compare all three methods.

Camcorder—This is the simplest method. Put your camcorder on a tripod, point it at the screen and record. Although simple, the resulting video will be a bit fuzzy and hard to read. It’s useful for getting an idea of what the user did, but it can be difficult (sometimes impossible) to read small text.
VCR—If your video card has a TV-out option (a feature that’s fairly common on modern video cards) you can probably connect it to a VCR and record directly to tape. The result should be an improvement on the camcorder method, but because the resolution and sharpness of a television is lower than a computer screen the result will still be fuzzy and downgraded from the original image. To get something readable you’ll need to limit your screen resolution to 800x600 at most, preferably 640x480.
Software—In the software solution a program runs in the background, silently capturing everything that appears on the screen and saving it to a video file. The result is a perfect recording with no loss of detail. Each frame of the resulting video could serve as a screenshot. Indeed, that’s one way to think of how the software works: taking series of screenshots and stringing them together into a techno-flipbook (of course the technical details are more involved).

The software approach is the most appealing, but traditionally it’s had one huge drawback: performance. The software has to capture and compress an immense amount of data in real time, without slowing down the machine. When I tested these programs on older hardware they would sometimes bog down so much it took ten seconds for a pull-down menu to appear. In my tests the performance problem vanished when testing on a 1 GHz machine and a good video card. As I write this, 1 GHz machines are near the bottom end of the scale for desktop PCs. Hardware requirements are no longer the hurdle they used to be. There is one obvious limitation to the software approach—it will only record what happens on the screen. It won’t record users themselves. If you want to learn something from the body language and physical movements of the user then you’ll still need a camcorder. Features and requirements This article arose out of a research project I was doing on how people search. For this project I developed the following set of software requirements. They should satisfy most usability testing situations:

Record at 10 frames-per-second and 800x600 in 16-bit color with no noticeable impact on system performance. Obviously lower frame rates, resolution, and color depth would improve performance, but this was my bare minimum. Much of the web doesn’t look good in 8-bit color and even participants in a research project shouldn’t be forced to suffer the indignities of 640x480. While I was willing to settle for 5 frames-per-second, I was hoping for 10 or more. Given a fast enough machine all three programs were able to meet this requirement.
Unobtrusive recording. I wanted the capture software to be invisible during recording. I didn’t want users to be distracted or feel anxiety by being constantly reminded of the recording. Most of the tools didn’t completely disappear when recording, but they all reduced to a small and unobtrusive icon in the toolbar.
Low cost. I couldn’t spend more than a few hundred bucks.
Pause, Fast Forward, and Rewind. Some of the tools use a special video format and thus a special program for playing the video. The playback tool needed have a pause feature and preferably fast-forward and rewind.
Immediate playback. My project used a technique known as retrospective verbal reports, more commonly called a “think after.” In this technique the user is recorded while doing the assigned task. When the task is completely they are shown the video and asked to conduct a think aloud. For think afters it’s best to watch the video immediately after the test to minimize forgetting. The only program that had problems here was ScreenCam which required a minute or two to write out the video file after recording. Even for the think after protocol this wasn’t a showstopper.

Those were my required features. There were a few other features I was also interested in but they weren’t critical.

Record Sound. All three products can record an audio track along with the video. Of course this requires even more computing power. Since I needed to record video for only part of the session, but audio for the entire thing (participants were interviewed after the think after session), I went analog and used a tape recorder for the audio recording. I didn’t need this feature, but you might.
Hotkeys. To minimize futzing with the program during test sessions I wanted hotkeys for important commands like Record, Pause, Play, and Stop. All of the programs had hotkeys. I found this to be a useful feature.
Record “raw” data. My dream program would have recorded a separate data stream of every keystroke, every mouse click, every URL visited, and so on. It would have time stamped each event, and automatically correlated it with the video. None of the programs did anything close to this so I had to record this data by hand by reviewing the video. One possible solution here is using a “spyware” program to record this raw data stream and then manually correlate them. I never seriously investigated this option.

Curiously, none of the tools I investigated were designed for usability testing. They’re mainly used for creating tutorial videos and software demos. This means they have a lot of other features that look nifty, but for someone engaged in usability testing they are thoroughly useless and so I’ve ignored them here. Testing the Software I wound up testing three software packages: Lotus ScreenCam, TechSmith Camtasia and Hyperponics HyperCam. I tested the products on three different machines with differing capabilities. (Note: I was only able to test ScreenCam on Machine A because ScreenCam only runs on Windows 95, 98, and NT.)

	Machine A	Machine B	Machine C
Processor	200 MHz (Pentium Pro)	333 MHz AMD K62-333)	1 GHz (AMD Duron)
RAM	64 MB	320 MB	256 MB
Video Card	Matrox Millenium	Matrox Millenium II	ATI Radeon 8500
Video Card RAM	8 MB	16 MB	64 MB
Operating System	Windows NT 4.0 with Service Patch 6a	Windows 2000 with Service Patch 2	Windows 2000 with Service Patch 2

My test procedure was as follows:

Set the display to 800x600 and 16-bit color.
Set the frame capture rate to 15 frames per second.
Start recording.
Start Internet Explorer, maximize the browser to fill the entire screen, and begin browsing the Web.
If the performance is not acceptable:
- Reduce the frame rate until either performance is acceptable or the frame rate is 5 frames per second. Never go lower than 5 frames per second.
- If performance still suffers and the frame rate has been reduced to 5 frames per second, reduce the color depth to 8-bits (ie: 256 colors). Keep the resolution at 800x600.
- If it still doesn’t work, reduce the resolution to 640x480. Keep the color depth at 8-bits and the frame rate at 5 frames per second.
- If it still doesn’t work give up on the program and have a nice cup of tea. I have found that second flush Darjeeling from the Margaret’s Hope Tea Estate to be particularly relaxing.
If performance is acceptable:
- Continue browsing for about five minutes. Visit sites with long pages (so I can scroll), complex layouts, forms, and other features. My standard routine was a few searches on Google, Yahoo, Amazon, Salon, and CNN.
- Repeat the test at 1024x768. If that works, move up to 1280x1024.

The following aspects of the test environment should also be noted:

The browser cache was cleared before each test.
No proxy servers were used.
The Internet connection was a 384 KBps ADSL line.
Only video was recorded. All of the tools can optionally record an audio track. Camtasia and HyperCam can also add sounds and visual effects to certain events like mouse clicks. None of these features were used.

Lotus ScreenCam Website: http://www.lotus.com/home.nsf/welcome/screencam/ Version tested: Lotus ScreenCam for NT Price: $86 ScreenCam is a story of good news and bad news. The good news is that it offers excellent performance. When compared with Camtasia and HyperCam on the same machine it had the highest frame capture rate while having the least impact on overall system performance. The bad news is that according to the web site “there are no plans to create a version of ScreenCam to work on Windows 2000 or Windows XP.” In other words, ScreenCam is a dead product, though you can still buy it. ScreenCam is only available for Windows 95 and Windows NT. It will also run on Windows 98 and variants like Win98 SE and ME, but won’t work with certain video cards (see the website for details). Results from Machine A (200 MHz) ScreenCam was the clear winner on Machine A, the oldest and slowest system I tested on. It had the least impact on system performance and captured the most data. The resulting video was smooth and sharp. However, to get good performance at 800x600 I had to reduce the color depth to 8 bits. It worked at 16-bit color but it was noticeably slower. Pages were slower to display and scrolling felt chunky. It worked, but not very well. If you’re stuck with an old 200 MHz machine and it’s running an older version of Windows, then ScreenCam is definitely your best bet. Even so, you may be forced to go with 8-bit color depending on the system speed. Results from Machine B (333 MHz) ScreenCam was not tested on Machine B because it’s doesn’t run on Windows 2000. Results from Machine C (1 GHz) ScreenCam was not tested on Machine B because it’s doesn’t run on Windows 2000. Details about ScreenCam For ScreenCam to work you need to install special ScreenCam video drivers. I found this surprisingly painless, but it’s unique to ScreenCam. Neither Camtasia nor HyperCam require special drivers. These video drivers are the reason for ScreenCam’s superior performance, enabling ScreenCam to access the video display through low level operating system calls. The downside to this approach is the ScreenCam must be rewritten to support each version of Windows. That’s why it works on Windows 95, and NT, most versions of 98 (depending on the video card), but not at all on Windows 2000 or XP. ScreenCa m records data to a special file format that can only be played back using the ScreenCam player. The player can be downloaded for free and runs on any version of Windows. The good news here is that while you can only record on Windows 95/98/NT, you can play ScreenCam recordings on any version of Windows, including Windows XP. When recording is finished, ScreenCam needs to spend time processing and creating the final video. The amount of time this takes depends on the length of the recording. For a nine-minute test at 800x600 and 8-bit color, ScreenCam spent approximately 70 seconds “processing data.” This processing creates a temporary file that can then be played back. But this file still needs to be saved if you want to keep it. In my test this took an additional 30 seconds. It’s possible to convert ScreenCam videos to standard AVI movie files, but I don’t recommend it. My nine-minute test produced a 58 MB ScreenCam file. When I converted this to an AVI file at 10 frames per second the resulting file was 2.5GB. Pros:

Better performance than either Camtasia or HyperCam.
Can be used even on older hardware.
ScreenCam player is free and runs on all versions of windows.

Cons:

Only supports Windows 95, Windows NT, and most versions of Windows 98.
No longer being developed. No plans to support Windows 2000 or XP.
Requires special video driver (easy to install).
Uses a proprietary video format and converting to standard formats like AVI creates huge files.

The bottom line: ScreenCam had the best performance of any program tested, but the lack of support for Windows 2000 and XP makes it hard to recommend. It’s probably the best choice if you’re stuck with older hardware running Windows 95, 98, or NT. TechSmith Camtasia Website: http://www.techsmith.com/products/camtasia/camtasia.asp Version tested: 3.02 Price: $150 Camtasia offers excellent performance, the richest feature set, and it runs on all versions of Windows. On Machine C, the fast machine in my test group at 1 GHz, Camtasia had no troubles recording 15 frames per second at resolutions up to 1280x1024 in 16-bit color. Even at 1600x1200 it was able to record 15 frames with only a hint of sluggishness. Camtasia also performed well on the 333MHz machine B. It had no troubles at 800x600 and was only slightly sluggish at 1024x768. There are only two downsides to Camtasia. It has a lot of features that you probably don’t need for usability testing, and it’s by far the most expensive tool in this review. At $150 it’s almost double the price of ScreenCam and five times the cost of HyperCam. Results from Machine A (200 MHz) Camtasia didn’t run particularly well on this machine, but it did run. In 16-bit color at 800x600 I was able to capture 5 frames per second, but just barely. The cursor would flash constantly as the machine tried to keep up, pages loaded slowly, and scrolling felt sluggish. It worked, but it was far too slow for usability testing. Dropping to 8-bit color made a noticeable improvement. Although performance was much improved, I couldn’t increase the frame rate significantly. I was barely able to capture five frames a second at 1024x768 in 8-bit color. ScreenCam was definitely better on this system (which is admittedly ancient). Camtasia was almost good enough to be usable at 8-bit color on this machine, but not quite. Results from Machine B (333 MHz) Camtasia had no troubles capturing the required 15 frames per second at 800x600 in 16-bit color. Bumping the resolution up a notch to 1024x768 was acceptable, though there was a noticeable pause when loading pages. Performance wasn’t quite smooth, but it was usable. For someone used to browsing the web over a 56k modem the pauses would probably seem normal. At higher resolutions Camtasia began to bog down. Still, this was a significant improvement. Machine B is roughly 60 percent faster overall than machine A, but where Camtasia was just barely able to capture 5 frames per second at 800x600 on machine A, it grabbed 15 frames a second on machine B with no performance impact and even worked well at 1024x768. Results from Machine C (1 GHz) Camtasia performed flawlessly on this machine. It recorded 15 frames per second at resolutions up to 1600x1200. There was a slight sluggishness at the highest resolution, but nothing significant. The machine was still perfectly usable. At lower resolutions there were no performance degradations. Details about Camtasia When you buy Camtasia you actually get three pieces of the software. There is Camtasia Recorder for recording the video, Camtasia Player for playing the videos, and Camtasia Producer which is a basic video editing tool. Camtasia also requires that you install a special Camtasia video codec called TSCC (it’s free). Using TSCC dramatically reduces the size of captured video files without any loss in image quality. One of my Camtasia tests ran 19.5 minutes in 800x600 at 16-bit color. The resulting video file was 36.8MB. Installing the codec is easy and quick (and doesn’t require rebooting your system). An important trick to using Camtasia is the “hardware acceleration” setting. It’s counter-intuitive, but turning hardware acceleration off results in a dramatic performance improvement. With hardware acceleration on, Machine B was chunky and sluggish at 800x600. When I turned it off, this sluggishness vanished. The hardware acceleration option is actually a Windows setting and has to do with your video card. Camtasia has an option to automatically disable acceleration when you start recording and enable it when you recording ends. Camtasia will automatically attempt to determine the best video and audio capture rates. For my tests I elected to set these values manually, but I also ran tests to see how the auto-detect feature worked. No complaints here. Unlike ScreenCam, the Camtasia video was available immediately after recording. No post-processing was required for straight video. If sound is being recorded, Camtasia records it in a separate file. When recording is stopped, Camtasia merges the two files. During a five minute test merging the audio and video streams took about 15 seconds on the 333 MHz machine B. Camtasia has a wealth of other features. I won’t go into all of them, but here are the highlights:

You can choose to capture the entire screen, a single window, or a specific region of the screen.
Although Camtasia includes a free Camtasia player, you don’t need to use it. Any video player will work so long as you have the TSCC codec installed.
Camtasia Producer is a video editing tool for combining, editing, and otherwise munging your videos. None of the other tools included something like this.
Camtasia also sells an Software Developer Kit (SDK) “to allow you to easily add screen recording functionality into your Windows application.” The SDK is available as a separate product. None of the other tools offer a similar package.

Pros:

Excellent performance.
Excellent features.
Easiest to use of the programs tested.
Supports all versions of Windows, except for Windows 95 (but does support Windows 95 OSR2).

Cons:

The most expensive tool reviewed. At $150 it’s almost twice the cost of ScreenCam and five times more than HyperCam.
Includes features you probably don’t need for usability testing (like Camtasia Producer).

The bottom line: Camtasia offers the best blend of performance, features, and ease of use among the programs tested. It runs on every version of Windows (except the original Windows 95) and installation is a snap. The only drawback is price, but at $150 it’s still within the range of almost every budget. Highly recommended. Hyperponics HyperCam Website: http://www.hyperionics.com/hc/ Version tested: 1.70.03 Price: $30 HyperCam is by far the cheapest of the products tested, yet it probably has all the features you need for usability testing. It offers slightly less performance than Camtasia, but at one-fifth the price. Almost any machine you buy today will have enough spare computing power to make up the difference. The biggest drawback to HyperCam is that it’s a little harder to configure properly. Most of these are minor and, considering the price, you may be willing to live with them. Results from Machine A (200 MHz) HyperCam performed almost as well as Camtasia on this machine. It was barely able to capture 5 frames a second at 800x600 in 16-bit color. It performed much better at 8-bit color. As with Camtasia it wasn’t great, but it did work at the reduced color depth and at modest frame rates, though not well enough to use for usability testing. Results from Machine B (333 MHz) HyperCam required a bit of coaxing to get it working properly on this machine. Once I got the settings right, which took some fiddling (more on this below), it captured 15 frames per second at 800x600 in 16 bit color. At 1024x768 I could do no better than 11 frames per second, but performance was smooth. Overall Camtasia performed better on this machine, but HyperCam’s performance was certainly acceptable. Results from Machine C (1 GHz) On this, the fastest test machine, the difference between Camtasia and HyperCam was almost negligible. HyperCam had no troubles with the base requirement of 15 frames per at 800x600 and 16-bit color. Even at 1024x768, 1280x1024, and 1600x1200 HyperCam was able to capture a full 15 frames per second with little or no performance problems. Details about HyperCam HyperCam has most of the same features and options as Camtasia, but I found it a little harder to use. For example, HyperCam lets you capture either a window or any rectangular region of the screen. Camtasia does this too, but it also has a one button feature for capturing the entire screen. To capture the entire screen in HyperCam you have to first define a region which covers the entire screen and then press record. Admittedly this is a little thing. But there are three other “little things” related to performance that I found frustrating. Once I figured them out HyperCam worked like a champ, but until I figured them out HyperCam left me unimpressed. The first of the “little things” is hardware acceleration. Like Camtasia, HyperCam works best if the video hardware acceleration is turned off. Unlike Camtasia you have to muck around with the Windows display properties to turn this off, then run HyperCam, and when you’re finished recording you have to turn it back on. Camtasia has a “Disable display acceleration during capture” checkbox that automatically disables acceleration when you start recording and enables it when recording is finished. A small but helpful touch. The second little thing is the frame capture rate. Camtasia will automatically try to determine the best frame capture rate for your system. You can also set it manually, and if you set it too high Camtasia will automatically drop frames and keep recording (though the system will probably slow down). HyperCam takes a different approach to frame capture rates. First, there is no auto-configuration option—you must set the frame rate manually. This isn’t a big deal, but if you set the frame rate too high HyperCam will start recording, then stop suddenly and display an error message saying the frame rate is too high. In my tests I started at fifteen frames per second and lowered the frame rate step-by-step until HyperCam stopped complaining. The third little thing is the video codec. HyperCam lets you select which video codec to use for the recording. Since most users (including me) know nothing about video codecs, HyperCam has an autoselect feature which is “Strongly Recommended.” Unfortunately, HyperCam was much slower than Camtasia when I chose autoselect. Wanting to give HyperCam a fair shake I decided to try other codecs. Scanning the list I saw an entry for the “Techsmith Screen Capture Codec.” This is the codec that Camtasia installed (TSCC). When I tried recording with TSCC, HyperCams’ performance shot up to the point where it ran almost as fast as Camtasia. In other words, HyperCam by itself has some performance problems, but you can overcome these problems by using the TSCC codec from Camtasia. I have been unable to find any reason why this would not be allowed. The TSCC codec from Camtasia is available as a free download and I had no technical difficulties using it with HyperCam. Pros:

Inexpensive. At only $30 USD, that’s one-fifth the cost of Camtasia.
Supports all versions of Windows.
Performs almost as well as Camtasia as long as you’re using Camtasia’s TSCC codec.

Cons:

Not as many goodies and features as Camtasia, but probably enough for the usability professional.
Harder to use and configure for decent performance.

The : My first impression of HyperCam was that for $30 I was getting what I paid for. But once I fiddled with it and found the “secret” of using Camtasia’s TSCC codec, I was entirely satisfied. Unless you need the extra features of Camtasia, HyperCam will probably do the job (but download the trial version and test it to make sure). Summary and recommendations Before you make a decision I strongly recommend that you download these programs and try them yourself. It’s the only way to be sure you’ll get acceptable performance on your hardware. The trial versions are free, installation is a snap, and running some basic tests will take just a few minutes. Camtasia is clearly the best of the bunch, but it’s also the most expensive. With a bit of fiddling, HyperCam will perform almost as well as Camtasia for a fraction of the cost. Camtasia has a lot more features, especially since it includes a basic editing and production tool, but for usability testing the programs are roughly equivalent when it comes to features. I can’t recommend ScreenCam. While it used to be the gold standard in this area, it’s now a dead product with no future. Choosing between Camtasia and HyperCam is difficult. I preferred Camtasia for it’s ease of use. It’s a slightly faster and more polished than HyperCam. Still, HyperCam is a bargain at $30 and it’s probably worth the fiddling required to make it perform well.

	ScreenCam	Camtasia	HyperCam
Purchase Options
Cost (USD), single copy	$86.00	$149.95	$30.00
Free trial for download?	Yes (15 days)	Yes (30 days)	Yes (no time limit, but all videos are stamped with message saying it’s unregistered)
Buy online?	Yes	Yes	Yes
Site license available	Unsure	Yes	Yes
Educational discount	Unsure	Yes	No
Platform Support
Windows 95	Yes	Yes (only on Windows 95 OSR2)	Yes
Windows 98 (including Win 98, 98 SE, and ME)	Yes (may not work with all video cards)	Yes	Yes
Windows NT	Yes	Yes	Yes
Windows 2000	No	Yes	Yes
Windows XP	No	Yes	Yes
Recording Features
Hot-keys to start, stop, and pause recording	Yes	Yes	Yes
Record sound	Yes	Yes	Yes
Record full screen	Yes	Yes	Yes
Record any region	No	Yes	Yes
Set frame capture rate	No	Yes	Yes
Choose codec used for recording	N/A	Yes	Yes
Hide when recording	Yes	Yes	Yes
Playback Features
Pause	Yes	Yes	Yes
Fast Forward	Yes	Yes	Yes
Reverse	No	Yes	Yes
Special player required?	Yes. The player is a free download. It runs on all versions of Windows, including 2000 & XP. You can only record on 95/98/NT, but you can playback on anything.	No. A special player is available as a free download, but any video player will do as long as you’ve got the TSCC codec installed. The codec is a free download.	No. Any video player will work. If playing back on a different machine you must have the codec installed that was used for recording.

Karl Fast was an information architect at Argus Associates. He is currently pursuing a Ph.D. in information visualization at the University of Western Ontario.

All About Facets & Controlled Vocabularies

Fred Leise, Karl Fast, Mike Steckel — Mon, 09 Dec 2002 22:44:53 GMT

“Our aim is to make this complex and important subject accessible to practicing information architects.”Information architects are fascinated with faceted classification and its application to information architecture problems. However, facets remain difficult to understand and there are few options for learning about them. This is the first in a series of articles that aims to correct this situation. We intend to explain both facets and the more general concept of controlled vocabularies. We want to make the subject accessible to those who don't have advanced degrees in library and information science. Furthermore, we want to show how these concepts can be applied to solve information architecture problems for the Web and other digital information environments. The concept of faceted classification is decades old, and controlled vocabularies go back even further. Consequently a great deal has already been written about the subject. But these writings are not always helpful to the practicing IA. Some are too simple, others too academic. Most are hard to find, and many were written decades before this Web thing happened. Throughout this series we will strive to be:

Practical. We will give you a practical guide to controlled vocabularies and faceted classification. We will not only explain the concepts, we will show you how to apply them in solving real information architecture problems.
Readable. Too much of the existing literature is hard to understand. It may be comprehensible to someone with a master's in library and information science, but this excludes a large number of practicing IAs (and we know some librarians who don't understand this stuff). We will use plain talk to explain this stuff, without dumbing it down.
Relevant. We will make this relevant to the Web and other digital information environments. A great deal was written about this topic in the 1950s and 60s. It's excellent material, but back then transistors were still a pretty neat idea. We believe that faceted classification has even more applications today than it did back then.
Accessible. Everything will be published here on Boxes & Arrows: on the web, easy to access, and free. While a lot has been written on this topic, it's often hard to obtain. For example, B.C. Vickery's excellent book, “Faceted Classification: A Guide to the Construction and use of Special Schemes” was written in 1960 and is rather difficult to obtain today (at least one of the authors has resorted to finding a copy in a library and, in desperation, photocopying the whole thing).

The plan Our main goal is to explain faceted classification. However, a faceted classification scheme is actually a special case of what are called controlled vocabularies. To properly explain facets we will begin with this more general topic and work our way up to facets. Our travels through this strange land will include the following:

Controlled Vocabularies. In the first full article in the series we'll describe controlled vocabularies in general. We'll talk about what they are and how they work.
Synonym Rings & Authority Files. Before moving on to facets, we'll describe these simpler types forms of controlled vocabularies. There are many situations where they are more useful solutions because they're easier to create, implement, and maintain. Sometimes they're not enough and it's time to step up to facets.
Facets & Facet Analysis. With the fundamentals in place we will move on to the heart of our subject. This will take a while, but it'll be worth it. We'll also take time to describe facet analysis, the process used to develop facets.
Interface Issues. A long-standing weak point of controlled vocabularies is how to use them effectively in an interface. This is particularly true of facets. We'll explore these issues and give you the best advice we can.
Decision Factors. Not every project calls for a full blown faceted solution. Sometimes a synonym ring is better. How do you know? We'll cover some guidelines for making those decisions.
Future Directions. There are some interesting new applications related to facets and controlled vocabularies such as XFML (http://www.xfml.org/) and Topic Maps (http://www.topicmaps.org/). We hope to cover these as well.

Some final thoughts That's a lot. And yes, we're ambitious. But no, we aren't writing the definitive treatise on the subject. Our aim is to make this complex and important subject accessible to practicing information architects. We view this as a collaborative effort. We anticipate many questions. We'll answer these through the discussion features of Boxes & Arrows. We also plan to address the bigger questions you have in subsequent columns. Let us know what you want to know, and we'll do our best to provide you with answers. Karl Fast is a PhD student in library and information science at the University of Western Ontario. He also has a master's in LIS. His graduate work has included courses on organization of information, subject analysis, thesaurus construction, and facet analysis. Fred Leise, president of ContextualAnalysis, LLC, is an information architecture consultant providing services in the areas of content analysis and organization, user experience design, taxonomy and thesaurus creation, and website and back-of-book indexing. Mike Steckel is an Information Architect/Technical Librarian for International SEMATECH in Austin, TX.

What Is A Controlled Vocabulary?

Fred Leise, Karl Fast, Mike Steckel — Mon, 16 Dec 2002 23:29:17 GMT

“A controlled vocabulary is a way to insert an interpretive layer of semantics between the term entered by the user and the underlying database to better represent the original intention of the terms of the user.”The most effective communication occurs when all parties involved agree on the meaning of the terms being used. Consequently, finding the right words to communicate the message of your website can be one of the most difficult parts of developing it. When we converse, we speak in “natural language.” This is language in all its raw, rich, gooey glory. When we organize our information and label it however, there is so much richness, variance, and confusion in terminology that we often need to impose some order to facilitate agreement between the concepts within the site and the vocabulary of the person using it. This order can come through a controlled vocabulary. Amy Warner defines a controlled vocabulary (CV) as “organized lists of words and phrases, or notation systems, that are used to initially tag content, and then to find it through navigation or search.” This means that a CV is a type of metadata that functions as a “subset of natural language”(Wellisch); it is not how we normally speak. Using a CV is also a way to overtly display relationships among the various concepts that your site covers in order to increase findability. The most basic, and often overlooked, form of controlled vocabulary is a consistent labeling system. If you are careful to call the same thing, or the same concept, by the same name everywhere on your site, you are using a very simple controlled vocabulary. And you're also ensuring that your users start developing a mental model of the information they can find. A controlled vocabulary is a way to insert an interpretive layer of semantics between the term entered by the user and the underlying database to better represent the original intention of the terms of the user. Consider what happens when you do not use a controlled vocabulary. An uncontrolled vocabulary simply uses the natural language of the documents and matches that with the natural language of the user. This is extremely specific, and it gives the user exactly what they ask for. Sounds great right? Consider, however, a site about chemistry, where many of the documents use the chemical name of the element (“iron”), and many use the chemical symbol of the element (“Fe”). Using an uncontrolled vocabulary, the results will only include the terms entered by the user. If the user entered “Fe” in the search box, he will not get any of the results for documents that use the term “iron.” There is a good chance the user is missing some documents he would like to have. Very few users will enter both terms, and many will be reviewing their results thinking they are seeing the results from all relevant documents. The equivalence relationship You probably are aware of certain categories or items on your site that might go by multiple names. You realize that if you said “automobiles” on your homepage and “cars” on the next page, users might get confused. Users will start to wonder if there is a difference between the two terms. Instead you choose “automobiles” and don't use “cars” at all. In this case “automobiles” is the term you prefer to use throughout your site. We call this the “preferred term.” “Cars” is a variant term, a different word representing the same concept. Or, consider this example: Here, each term refers to the same concept, Elizabeth Taylor (your preferred term). We could tell our system, when people ask for “Elizabeth Burton” use “Elizabeth Taylor.” This is more traditionally expressed using standard CV notation as: Elizabeth Fortensky USE Elizabeth Taylor Elizabeth Taylor UF Elizabeth Fortensky (UF = Use For) Or even this: Liz Taylor USE Elizabeth Taylor Elizabeth Taylor UF Liz Taylor Think about Gap's web page (http://www.gap.com). We already know what they sell (they have excellent branding), and most of their content is generally referred to by the same terms as used in our general culture. In other words, people consistently say “jeans,” “pants,” and “shirts.” Even though you might get the occasional person using the word “dungarees” or “slacks,” nearly everyone would see “jeans” and know what the category referred to (the visuals help support this too). Furthermore, Gap does not carry hundreds of pairs of jeans that must somehow be distinguished from one another. If you examine the natural language people use when talking about Gap's products, there's an unusually small amount of term variance. Content like this works great in the very simple organization system used on the Gap site. It works so well that they do not even need to offer search; this is very unusual for an ecommerce site. What they have is a system in which all of the concepts are consistently labeled using language familiar to their users. They're lucky. Few sites have the option to work in this way. Let's say, however, that gap.com decided to offer search. Then they would somehow need to translate the natural language of search into the controlled language of the website. People search in the same language they speak, natural language, so a more advanced controlled vocabulary needs to take the concepts of your users (natural language) and match them to the concepts expressed in the language of your website (controlled vocabulary). That means if the developers of the site began to see that people were searching for “dungarees” and getting zero hits, they would need to create a way to tell the system, “when someone searches for 'dungarees,' give them the results for 'jeans.'” In the language of a controlled vocabulary, “jeans” becomes the preferred term and “dungarees” is a variant term, and they have an equivalence relationship. This can be a powerful tool for increasing findability. There are many examples of the situations that alternate terms cover. Here are a few:

synonyms (two words with the same meaning, like “jeans” and “dungarees”)
homonyms (words that sound the same, but have different meanings, like “bank” the financial institution and “bank” the side of a stream or river)
common misspellings
changes in content (e.g., countries that change their name or have multiple spellings)
identifying “Best Bets” or the most popular pages associated with a certain term (http://www.BBC.com is great at this)
connecting a woman's married name to her maiden name
connecting abbreviations to the full word (e.g., NY and New York, the chemical symbol Si with the element Silicon)

There are two types of synonym equivalence lists: synonym rings and authority files. Synonym rings are generally used for searching behind the scenes as a way to connect the various terms for a concept. It can be used to say, “when someone searches for “Si,” give them all documents with both “Si” and 'Silicon.'” However, what happens when you want to display one of these terms in your navigation? Then you will need to pick one to be your preferred term. Now, you have an authority file. In each of the above examples, different terms may be used, but each one represents the same concept. They are tied together and given meaning by making their equivalent relationship explicit. Hierarchical relationships: broader and narrower terms If your content is more complex, for instance if you sold only pants and you had hundreds of types, you might require more from your controlled vocabulary. The natural language we use to describe the concept of “pants” quickly enlarges as “pants” becomes more specific. In other words, “slacks,” “khakis,” “jeans,” “trousers,” “corduroys,” and other kinds of pants will all need to be differentiated so users don't have to rummage through pages and pages of search results for the word “pants,” when pants are your whole inventory.

What will help is a systematic way to map out the different terms so people quickly find the specific kinds of pants they are interested in. What you need is a hierarchy showing the broader terms (BTs), the narrower terms (NTs), and the variant terms (most often displayed as “USE” and “UF” for Used for). These will show which terms are subsets of larger, broader concepts. You are starting off with a jumble of words that are all related to “pants” in some way. It might look something like Figure 2. We have a bucket we can call “Pants” and inside are a lot of terms with a relationship to the concept of pants. In this example, “pants” is the broader term, and the kinds of pants refer to subsets of the whole universe of pants. In a controlled vocabulary, we might reconfigure the chart above to look like this: This is what people are increasingly calling a Taxonomy. This term makes traditional librarians a little uncomfortable, but we are learning to live with it. Originally it was a term for biological classifications (Genus, species, etc.), but has quickly become a standard word for describing hierarchies. The standard CV notation used to express hierarchical relationships are NT (narrower term) and BT (broader term). Using this notation, the term “Women's Pants” would be expressed like this: Women's Pants BT Pants NT Casual Pants NT Dress Pants NT Sports Pants There is a lot you can do with this hierarchical arrangement. It can help you formulate your homepage navigation. It could improve your searching and browsing. It can help users broaden and narrow their search results quickly by showing them where each set of results fits into the site's hierarchy (see Keith Instone's “attribute breadcrumbs” for more examples). Generally, few sites need to go beyond the level of a taxonomy, but it might be useful to see the next level of complexity in controlled vocabularies. Associative relationships: related terms How far can I extend the pants example? Oh, quite far. Let's say that you are a research institute that studies pants (ridiculous I know, but stay with me). You not only study pants themselves, but the materials they are made from, their history, how they are manufactured, and more. Your institute might do well to take the time to develop what Peter Morville has called the “Rolls Royce of controlled vocabularies”—a thesaurus. A thesaurus shows all of the relationships described so far (BT, NT[LD3], and UF), but will also include related terms (RT). This is an associative relationship. It shows how one term is associated with another. If a user looked to your institute for research on jeans, you would be able to give them that term embedded in a rich series of relationships. An example of the range of relationships would be expressed like this using the standard format for thesauri: Jeans BT Pants NT Levis NT Wranglers UF Dungarees UF Waist Overalls RT Denim RT Overalls Denim is related to Jeans, but not hierarchically. It is not a type of jeans, nor is one a subset of the other. Yet someone interested in one term might be interested in the other because they are related concepts. In the interface, you might identify “Denim” as a “see also” option for “Jeans.” If users looked for the term “Denim” in the thesaurus they might see something like this: Denim BT Fabrics NT Ring Spun NT Dark Indigo NT Stonewash RT Jeans The Denim example alone could be filled with many additional terms, and it is easy to see how well this would accommodate user browsing (and “berrypicking”). This is also one of the dangers of creating associative relationships: knowing when to stop. This relationship is also the most difficult and subjective of all the relationships in a CV. You are identifying a relationship between two concepts that may not be obviously apparent. On an Amazon product page, when the page identifies an item that others have purchased along with the one being displayed, Amazon is identifying a potentially useful associative relationship. To push the concept a little further, if a user is interested in a paper from your pants institute on the “Hemingway wore khakis” advertisement from Gap, they might also be interested in a paper you have on how it was really Rock Hudson's subtle use of Khakis that made “A Farewell to Arms” such a great movie. The connection between the two documents, the intersection of the concepts of “Hemingway” and “khakis,” is less direct than the Denim example above. This is expanding the concept of “related terms” farther than many would be prepared to go, but it is an option. Internal uses of controlled vocabularies So far, we have focused on how controlled vocabularies help the user, but there are also benefits to the organization using the CV. Here are a few:

CVs can help with category analysis or keeping your categories distinct.
CVs can help establish a site's navigation.
CVs can be the basis for personalization features.
CVs can help with preparation for CMS or knowledge management projects, since many of these require this sort of structure to your content to do their magic.
CVs get the organization using the same language as the users (which should result in better communication with them).
CVs can help the organization (and the user) understand what concepts your site covers. Your controlled vocabulary is in reality a “concept map” of what is on your site.

While controlled vocabularies can be powerful, by themselves they are not the magic pill that will cure what ails your site. CVs are a lot of work, they are often difficult and time consuming to maintain, and they can be very political. Some skepticism toward all metadata is a healthy thing (everyone still reading this should see Metacrap). As with anything important, there are a lot of people who are doing it loudly and badly. Conclusion Human beings are natural makers of patterns. That is how we understand what our senses are taking in. When people visit your site, they will immediately begin trying to understand what they see. A well-designed and regularly updated controlled vocabulary can help connect the concepts your users have in their heads to the concepts you present on your site. That is when real communication will occur. Next in the series: How to create a controlled vocabulary.

Wellisch, Hans. Indexing from A to Z. New York: H.W. Wilson, 1995. p.214
Amy J. Warner Taxonomy Primer
http://www.BBC.com
Keith Instone's attribute breadcrumbs
ASIS Summit 2001
Berrypicking
Metacrap
An Annotated Bibliography

Karl Fast is a PhD student in library and information science at the University of Western Ontario. He also has a master's in LIS. His graduate work has included courses on organization of information, subject analysis, thesaurus construction, and facet analysis. Fred Leise, president of ContextualAnalysis, LLC, is an information architecture consultant providing services in the areas of content analysis and organization, user experience design, taxonomy and thesaurus creation, and website and back-of-book indexing. Mike Steckel is an Information Architect/Technical Librarian for International SEMATECH in Austin, TX.

Creating a Controlled Vocabulary

Fred Leise, Karl Fast, Mike Steckel — Mon, 07 Apr 2003 22:43:18 GMT

“Creating a clear plan early on can save you a lot of trouble down the road and minimize unwelcome surprises. The broad strokes of CV design are like any other type of design: planning and preparation are essential, fundamental steps in producing a good design.”You have probably heard IAs discussing the benefits of their latest taxonomy project and how you should be implementing one. But how, you might wonder, can you get started? This article describes a process for building your own controlled vocabulary (CV). A previous article discussed the concept of a CV—the “what.” This article focuses on the “how.” In this article we are looking at a process for creating any kind of controlled vocabulary. While our ultimate goal in this series is to explain facets, the details of facet analysis will be described in a future article. At this point, we are still exploring fundamental concepts and techniques. There are many ways to create a controlled vocabulary. What follows is just one methodology. Also, keep in mind that many of the steps described here are not discrete units. When you actually create a CV, some steps may overlap. Now, let's get started. Imagine we are a company that sells camping gear, and we want to create a controlled vocabulary for our ecommerce site. 1. Develop a strategy. What do you want your controlled vocabulary to do? The natural inclination when developing a CV is to start by gathering potential terms. But first, you need to consider a wide range of questions. Creating a clear plan early on can save you a lot of trouble down the road and minimize unwelcome surprises. The broad strokes of CV design are like any other type of design: planning and preparation are essential, fundamental steps in producing a good design. First, what kind of CV do you need? The answer depends on a variety of issues. Start by thinking about some general questions such as these:

What do you want your CV to accomplish?
Do you want the CV to integrate with your navigation system?
Are you planning on using the CV to improve searching? To improve browsing? Both?
Are you planning to show term relationships in your search results?
How much vocabulary control do you want to provide? Synonym ring? Facets? What level of vocabulary control is appropriate?

Second, think about your dependencies:

Content - Consider this in two parts: specificity and stability. Specificity: If you are selling camping gear, are you selling 7-10 styles of tent or 100 styles? If you are selling 100 styles, you will need terms that are more specific and more exhaustive. This is because you will need to further differentiate among tents that are similar. The more items that are similar, the more specific you need to be. Stability: Do the concepts and names for them change often? Do people generally call the same concept (or item or product) by the same name? In our example, we would ask if there are a lot of variant terms for the kinds of items we're selling. What will be your method for keeping up with changing terminology?
Technology - There are two pieces to this one: tools and integration. Each will help you think about implementation early on. Tools: Think about where the CV will ultimately sit. Do you have a CMS that will be involved? Will you be uploading your CV into a search engine? What software will you use to hold your terms: a thesaurus maintenance program like Multites, Term Tree, or Lexico? Or will you be creating it in Excel? Also consider tools you might use while gathering your terms. Many people collect their terms in a large Excel spreadsheet, others on Post-it Notes, sometimes even a wiki might work nicely. Integration: How will your CV be integrated with the other pieces of your system? If the CV is going to be used in multiple applications, you need to consider the requirements of each. Be sure you talk to someone in IT and outline what your goals are.
Users - CV design is a user-centered process. You must understand the target audience before setting down your terms. Who is the target audience for the site? The general public? Experienced campers? Are they web-savvy? How do they shop? Do they tend to buy one item at a time or several items at once? Do they need to do a lot of research before they buy? In other words, good, standard user-centered design methods, such as interviews and observation, are appropriate.
Maintenance - Who from the organization will maintain your controlled vocabulary? What amount of time can they spend on this task? What is their training? If you decide to create a highly complex controlled vocabulary that your high school intern is going to maintain, you will have to provide additional training for that person. This is also a user-centered design issue, but along a different axis. Above we talked about a process that is extroverted: it looks towards the external users of the system. Here our axis is introverted: it looks towards the internal users of the system, the creators and maintainers of the vocabulary.

At this point, any normal person will say to himself, “Geez! Enough with the questions! Let me get on with creating my controlled vocabulary!” Resist this urge and stick with the discovery process; developing a strategy is important. You will probably change some of your answers as the project develops, but considering these questions up front will prevent you from wasting time later on. 2. Start gathering terms. What are the terms used to describe your content? Now you are ready to start gathering your terms. Your goal here, considering the constraints and strategies that came out of Step 1, is to identify the terms that will bring the most success to your user population, enabling them to find exactly the information they need. This is where the process becomes a little bit like “The Newlywed Game.” In this TV game show, the contestants are newly married couples. While one half of the couple is in a soundproof room, the host asks the remaining partner some intimate questions (often about “making whoopee”). Later, they reunite the couple and ask the other partner the same questions to see how well their answers match up. The couple with the most matched answers wins the big prize. The underlying questions for this game include “How well do the two sides of this relationship know each other?” and “How well can one half of the couple guess the answer the other half will give?” To win the big prize of increased content findability, your site must describe your content in the terms that best match those terms the users are likely to use. When your partner (the user) comes out of that soundproof booth, you want to feel confident that you have provided the terms he will use on your site. There are lots of great ways to get started with this process. A. Look inward. What are the terms you already use to describe items on your site? If you are selling something, what are you selling? Look at each item and start generating terms to describe the object. What are the concepts the terms cover? List them. If we were doing a thesaurus for camping gear, we might start with something like: backpacks, tents, bug spray, etc. Then consider alternative terms you might use for each item. Consider the level of granularity you want to use to reach your target audience or need to use based on the number of similar items you sell. If the target audience for your camping gear CV is beginning campers, you might distinguish thick sleeping bags from your thinner options by making a distinction by season (as in "winter" and "summer" bags). However, if you are targeting expert campers, you may need to describe your bags as "2-season" or "3-season" bags, in terms of insulating material (goose down, Polarguard, PrimaLoft), or by the temperature ratings. You don't need to describe the entire field of camping gear; you need only describe your content in terms that will resonate with your target audience. There is a danger here, however. Don't look inward and exclude the additional options for gathering terms described below. It is important to get outside of your own understanding of terms and their concepts. Be sure to follow the next steps as well. B. Look outward. Where are people using terms related to your content? You might review competitors' sites, journals or magazines on your subject matter, or discussions by subject experts on the web. For example, if you are looking for terms about camping gear, you might look here: http://directory.google.com/Top/Shopping/Recreation/Outdoors/?il=1 Look at the sites on the list and note how they describe items that you also sell. Are there relevant variant terms you didn't include from the looking inward step? Consider the differences between REI and MEC (Mountain Equipment Co-op, a Canadian outdoor equipment store). Note the differences and similarities between the terms they use. In this example, we have shown only the terms for their top-level categories; you should dig deeper and find out what terms they use for sub-categories and individual items.

Sometimes, someone may already have developed a similar controlled vocabulary that you can use or modify. When this happens, we recommend that you perform an exuberant dance of joy. This won't work for our camping gear example, but if you are building a large controlled vocabulary on another topic, you might want to see if you could borrow from one of the controlled vocabularies here: — American Society of Indexers — Publications on thesaurus construction and use More than likely, you will need to simplify anything you use from one of these lists, but they might be worth reviewing. Often, just the exercise of reviewing other CVs can be helpful in discovering ways to improve your own. But be careful. Borrowing terms from other sites can muddy your own particular site's strategy. Don't borrow so much that your message gets confused or loses distinction. C. Log files. If you already offer search, an easy option is to review your log files. Log files are goldmines of valuable customer information. They will give you an idea of what people think they might find on your site, as well as the words they use to describe what they are looking for. If you can get the file to display search results (as in 8 hits, 0 hits, etc.), you can see how successful people are. Or, reproduce the searches yourself to determine if people are getting relevant hits. See how Nordstrom's benefited from this technique. D. Ask people. Is there a way to ask users what they look for on your site? How would they describe your site's contents? Throughout Step 2, you are building into your CV what librarians call “user warrant.” This means that a term “is justified for inclusion in an index (or CV) only if it is of interest to the users of the information service.” (Lancaster, 26). Your CV will have high user warrant if the terms you include are real terms that people use to describe your content. If you include a lot of terms you suspect people might use, but that did not actually show up during your research, you will lower the user warrant. You are taking a risk: You may be unnecessarily muddying your CV. At the end of this process you should have a large number of terms describing your site's content. 3. Establish preferred terms, variants and hierarchies. How do the pieces fit together? After Step 2, we are left with what is essentially a big bucket of unrelated terms. Now we start to put like terms together and identify each one's relationships. For each term, ask what is the broader (more general) term? What are the narrower (more specific) terms? If you are using terms to establish a navigation system, is this a preferred term or a variant? Your controlled vocabulary will start to come together as context is added to each term. Using our camping gear example, a traditional CV notation for the terms we have collected about sleeping bags might look like this: Sleeping Bags BT Camping Equipment NT Down Sleeping Bags NT Synthetic Sleeping Bags NT Family Sleeping Bags NT Cold Weather Sleeping Bags NT 2-Season Sleeping Bags NT 3-Season Sleeping Bags NT Ultralight Sleeping Bags (BT = broader term; NT = narrower term) Some in your group might say, “Hey, sleeping bags should go under Backpacking Equipment, not Camping Equipment.” A perfectly good assertion. Somehow, you will need to decide this issue. Can “Sleeping Bags” be in both places? Should the term live in one place in the CV with a cross-reference from the other location? Maybe there is a distinction among different kinds of sleeping bag that you had not previously considered. It might be a good time to do some research. For instance, ask yourself, “How do REI and MEC describe their sleeping bags?”

MEC does it like this:
Click to enlarge

REI takes a completely different approach:
Click to enlarge

The differences are striking. The main ones include the following:

Depth: The most obvious distinction is how REI goes for increased depth, whereas MEC uses a shallower category set.
Term Choice: REI uses the general term “Sleeping Gear,” whereas MEC uses “Sleeping Bags.” What's interesting is that both sites classify terms for related materials—pillows, stuff sacks, and so on—as narrower terms, yet only REI uses the more generic term “Sleeping Gear” to describe this breadth.
Broader Terms: REI has “Sleeping Gear” as a narrower term under the top-level term “Camp/Hike.” MEC also has a similar top-level term—“Hiking/Camping Gear”—but instead of making “Sleeping Bags” a narrower term they put it at the same level.
Bags and Pads: MEC puts sleeping pads as a narrow term under sleeping bags. REI doesn't put them below sleeping bags, but at the same level in the hierarchy.

Which is better? That's difficult to say. REI is more sophisticated in their categorization, probably because of their larger product line. While REI's scheme is more sophisticated, it's also more complicated, so perhaps the simplicity of the MEC approach is better. Most likely, these differences are the result of differing strategies. Our intention here is not to suggest which is better, only to show how even a simple situation can give many alternative answers. Certainly one can find much to like about these schemes. But in each case, improvements can be made. They are muddled. Concepts are mixed and matched haphazardly. There are questions about scalability and future directions. Material, temperature, gender, and age are combined in surprising and inconsistent ways. For example, why does MEC put sleeping pads as a narrower term of sleeping bag when they are obviously related, yet distinct items? And we're still confused about the distinction between these two terms in the REI scheme: “Kids' Camping Bags” and “Kids Backpacking Bags.” We will return to this example in a future article showing how facets can clarify this situation. But let's not get too far ahead of ourselves. For now, the question is: How do you clarify these issues? How do you make these difficult decisions? Making these decisions can quickly get messy in a group environment. Perhaps you need to ask a smaller team to consider the question and report back to the larger group. Doing some analysis, as we did with MEC and REI, and looking at your own strategy should help clarify what it is you want to do. However you decide your questions, be sure to note why you made the decision you did (for more on this, see Step 5). We have been arguing that a good CV design process is essentially a user-centered process. Getting feedback from users will give you a great deal of insight into the problems we have raised. A simple and commonly used method of getting feedback is called card sorting. Find some people whom you consider to be your target users. Give them cards with examples of items for sale on your site and ask them to arrange them into groups of like objects, or objects that they believe should be together. Then ask them to label their groups of cards. Look for patterns among their responses, compare the results to your original content labels, and make any necessary adjustments. For some good additional materials on card sorting, see the IA Wiki. Yes, it really is that simple and effective. 4. Identify the “see also” terms. What else might be interesting to your target audience? In most cases, related terms need to be identified only for large projects. If you are working on an ecommerce site, here is a way to connect related products that people might buy at the same time. In other words, you need to identify places where interest in one item might lead to interest in another. If your site users are buying camping boots, do they need socks? If they are buying backpacks, would they be interested in water bottles? Often, these are what the Polar Bear book calls “contextual navigation” (116-118). To get you started here, think about these possible relationships when considering related terms:

process/agent (camp fires/matches);
action/product of action (baking/cakes);
agent/counteragent (allergies/antihistamine);
raw material/product (wool/sweater).

Putting this idea of cross-selling into traditional CV notation might look something like this: Sleeping Bags BT Camping NT Down Sleeping Bags NT Synthetic Sleeping Bags NT Family Sleeping Bags NT Cold Weather Sleeping Bags NT 2-Season Sleeping Bags NT 3-Season Sleeping Bags NT Back Packing Sleeping Bags NT Expedition Class Sleeping Bags NT Ultralight Sleeping Bags RT Backpacks RT Ultralight Backpacking RT Sleeping Bag Liners RT Sleeping Pads RT Stuff Sacks RT Pillows (RT = related term) What constitutes a related term? That is something for you to decide. Try to strike the right balance between suggesting options and overwhelming a user with choices. You might want to run the card sorting exercise again, this time giving people a list of items on cards and ask, for each item, if there are any objects from your inventory that they might look for when purchasing it. Adjust your CV accordingly. 5. Establish a record of the rules you are using if you are creating a large thesaurus. I suspect most CV creators do not take the time to do this, and that is unfortunate. Remember all those decisions about what term goes where? Review the decisions you made and record what the decision was and why you made it. This will enable you to maintain consistency as your CV changes and expands. This makes your system easier to learn, and consequently, training your staff is easier. This is especially important for keeping categories pure if multiple people will be adding terms to content. It also makes for better decision-making in the future. I am reminded of an interview with cellist Yo-Yo Ma who told some students, “If you make specific choices in the music, we hear them.” He added later in the class, “If you don't make specific choices, we don't hear them.” This is as true for the actions of a controlled vocabulary as it is for a piece of music. Be aware of the assumptions you are making and make them conscious choices; users will “hear” them. Some possible questions to consider here are: When do you include a new term? What constitutes a relationship or RT? When do you delete terms? What is the basis for choosing a preferred term? When are terms singular or plural? Nouns or verbs? How will you deal with punctuation? A place to look for generating issues you might want to consider is the ANSI/NISO standard for thesaurus construction. Reviewing these guidelines and deciding what is relevant to your particular situation will help ensure the best possible outcome for your CV creation process. Now is also a great time to review the assumptions you made in Step 1. 6. Implement. This step is difficult to write about because implementation is extremely dependant on your specific context. The other steps are not easy, but in the real world implementation is often the most difficult. It is also something the literature on CVs rarely tackles in a meaningful way. For now, we will take the metaphorical 50,000-foot view. If you are using your controlled vocabulary for developing a menu for navigation or categories for browsing, continue your user testing. At this stage you can present a more complete version for users to evaluate. If you have completed some testing earlier, this should involve only minor changes to your CV. If you are using your controlled vocabulary for searching, get ready for more work: Tweaking the algorithms for a search engine is a difficult job involving lots of tradeoffs. It will also require a good relationship with your IT staff (good thing you started this already in Step 1!). A lot of difficult decisions will need to be made. Examples include how you use punctuation, Boolean operators (when to use AND and when to use OR connectors), and recall versus precision. Multiple word terms can sometimes be difficult (if your CV term is “walking staff” and the user enters “Walking Staff Wood,” does he get any variant terms for “walking staff?”). Your solution will depend on the search engine you are using, the audience, the content, and the tradeoffs you need to make to get your project up and running. 7. Test and evaluate. You have done some testing during the CV creation process, now it is time to make sure the assumptions you have made throughout the process are correct when you consider the implementation as a whole. Start with yourself. Use the site to find various types of information based on assumptions you made earlier. Can you identify which content goes in which slot pretty easily? Can you search and get the results you expect? If using your CV to improve searching, enter a term and carefully look at the first page of returns. Are these the results you want your users to get for this search term? After you feel like the CV is working as you believe it should, contact some outsiders and ask them to use your site. Do your terms reflect the concepts these people are searching for? Are they getting the results they expect? Are your terms too broad or too narrow? Remember, you are not always going to be successful. This is another time to keep the 80/20 rule in mind. 8. Go back and refine. What can be improved? A controlled vocabulary is never finished. The goal of the initial creation of your CV is simply to create a system for controlling vocabulary that is agile, easy to update, consistent in both scope (what is covered) and granularity (how deeply it is covered), and helps users find what they are looking for. However, maintenance is required to keep your CV viable and usable. Constant monitoring, evaluation, and tweaking are critical. This may require daily reviews of search logs, regular testing with users, regular conversations with subject specialists, or other analysis. One of the arguments against using a controlled vocabulary is that it requires so much time to maintain, that it doesn't keep up with the changing terminology of the given field. Therefore, constant analysis is key to success. The list of improvements you can imagine needing to make will always be long, but don't lose sight of the smaller, daily “housecleaning” tasks. There is a lot of talk about how controlled vocabularies improve a site's information architecture. If you decide to create one, however, it is important to realize that an effective controlled vocabulary involves regular maintenance. Doing it right will keep you aware of both the dynamic developments of your content and keep you close to the language of your users and their information needs.

Cooper, Alan (1999). The Inmates are Running the Asylum: Why High-Tech Products Drive Us Crazy and How to Restore the Sanity. SAMS publishing: Indianapolis, IN.
Lancaster, F.W. (1986). Vocabulary Control for Information Retrieval (2nd Edition). Information Resources Press: Arlington, VA.
Rosenfeld, Louis, & Morville, Peter. (2002). Information Architecture for the World Wide Web: Designing large scale web sites. (2nd Edition). O'Reilly & Associates: Sebastopol, CA.
An Annotated BibliographyKarl Fast is a PhD student in library and information science at the University of Western Ontario. He also has a master's in LIS. His graduate work has included courses on organization of information, subject analysis, thesaurus construction, and facet analysis. Fred Leise, president of ContextualAnalysis, LLC, is an information architecture consultant providing services in the areas of content analysis and organization, user experience design, taxonomy and thesaurus creation, and website and back-of-book indexing. Mike Steckel is an Information Architect/Technical Librarian for International SEMATECH in Austin, TX.

Synonym Rings and Authority Files

Fred Leise, Karl Fast, Mike Steckel — Tue, 26 Aug 2003 21:08:53 GMT

“Synonym rings and authority files are simple tools that can bridge the gap between natural language and complex controlled vocabularies (taxonomies and thesauri) quite nicely.”

This is Part 3 in our continuing series on controlled vocabularies and faceted classification. Previous parts in the series include:

All About Facets and Controlled Vocabularies (series introduction)
1. What is a Controlled Vocabulary?
2. Creating a Controlled Vocabulary

As any connoisseur of duct tape knows, when you need to get a job done, the simplest tool is often your best friend. This is as true for controlled vocabularies (CVs) as it is for home repair. Remember that our goal for CVs is to “impose some order to facilitate agreement between the concepts within the site and the vocabulary of the person [natural language] using it.” But that doesn't mean the CV has to be complicated. Resources do not always allow for a full-fledged thesaurus, and often such a large undertaking is not necessary. Synonym rings and authority files are simple tools that can bridge the gap between natural language and complex controlled vocabularies (taxonomies and thesauri) quite nicely. We can explain how synonym rings work by way of an example. International SEMATECH, the semiconductor research consortium, had a searching problem. Documents were uploaded to a private research website in a highly decentralized manner. Member company employees from all over the world had the ability to upload their own research documents and meeting presentations to the website. A look at the search logs, however, revealed that people entered search terms that were yielding only a percentage of the documents they were trying to find. The problem was consistency of terminology. A review of the metadata found that those uploading information were equally as likely to call silicon “Si” as they were to spell out the whole name, “silicon.” There were many similar examples. Besides chemical symbols, users were both searching and uploading documents with acronyms (“PSM” vs. “Phase Shift Mask”) and simple variants in spelling (“low K dielectrics” vs. “low-K dielectrics” vs. “lowk dielectrics”). The way the system previously worked, a user who searched for “Si,” “PSM,” or “low K dielectrics” would get only exact matches. In other words, they would miss documents that had “Silicon,” “Phase Shift Mask,” or “low-K dielectrics” in their metadata. Furthermore, they would get enough hits so they might not have realized that some relevant documents were missing (if they had gotten zero hits, they might have suspected something was wrong and tried another term). It was our assumption that when users searched one term, they intended to find the entire set of documents related to that concept. But trying to get such an organization to adopt a style guide for metadata was not viable. The solution was to install a synonym ring into our search engine, Oracle Text.

What the synonym ring does

A synonym ring connects a series of terms together and treats them all as equivalent for search purposes. When a user enters “PSM,” for instance, the search term will be sent through the synonym ring to see if there are any equivalent terms. For “PSM” we would find “Phase Shift Mask” as a synonym. The search engine would then retrieve all documents with either “PSM” or “Phase Shift Mask” in their metadata. The searcher would get the complete set of relevant documents as though they had searched both terms (something few people would think to do). If there is no match in the synonym list, the search is simply sent through the index as usual and any documents with “PSM” are returned. The synonym ring goes into effect only when there is a matching synonym for the term entered into the search box by the user. Although getting a synonym ring up and running sounds pretty simple, the difficulties often come from trying to answer a simple question: “What is a synonym?” The example above was clear case of synonyms: An acronym and the full name of the object. It is not always this simple. A synonym can generally be two words with the exact or very similar meanings. Sounds simple, but how similar is similar enough? True synonyms are a rare thing.

What is a synonym?

Some synonyms may appear to be pretty straightforward. These include:

Acronyms: BBC, British Broadcasting Company; MPG, miles per gallon
Variant spellings: cancelled, canceled; honor, honour
Scientific terms versus popular use terms: acetylsalicylic acid, aspirin; lilioceris, lily beetle

But synonyms, in general, quickly become more difficult. Are “medicine” and “drugs” synonyms? Are “fired” and “laid off”? What about “forest” and “woods” or “arid” and “dry”? With these examples, it is more difficult to say for sure. To answer the question about whether two terms are synonyms, you often have to consider the overall content of your site, as well as the site's context and its users. In our first article, we gave the following example of a synonym (which demonstrates the equivalence relationship): But one could easily argue that these are not true synonyms. You may be looking for information about Elizabeth Taylor only during the time she was married to Larry Fortensky. In this case “Elizabeth Fortensky” might be the only part of the ring you would be interested in. Expanding the results by including results for both “Elizabeth Warner” and “Elizabeth Fortensky” would reduce the precision of the search results. When creating a synonym ring, or any controlled vocabulary, you will spend a lot of time evaluating near synonyms. What guidelines should you use for making these decisions?

Recall and precision

Information architecture in the real world is all about the tradeoffs, right? Librarians have long been aware of the tradeoffs one makes between a search system that is broad and one that is specific. A search system that is broad is one with high recall, while one that is very narrow is one with high precision. Let's look at these two terms a little more closely. Recall is often represented as a ratio:

number of retrieved relevant documents / all relevant documents in a collection

Recall measures how many of the relevant documents are returned to the user. When you are searching a system with high recall, you are able to get a comprehensive set of documents returned, but you increase the possibility that less relevant documents will also get returned. This is great when you want to look through a large number of documents to make sure you have seen everything on a certain topic. Techniques for increasing recall include a synonym ring, stemming (some search engines will automatically return “jumping” and “jumps” when someone searches “jump”), and wildcards. Precision, like recall, is often represented as a ratio:

number of retrieved relevant documents / total number of documents retrieved

You want to return all relevant documents to each user. So why not return all documents in your system for every search? That way you can be sure that every single relevant document is returned to the user, right? Well, true, but you're also returning many irrelevant documents at the same time, making it harder for users to find what they want. Precision ensures that only the relevant documents are returned to the user. When you are searching a system with high precision, your results are specific to your search. This is closer to a known-item search. You want only relevant search results and are less tolerant of getting some irrelevant results mixed in. You can increase search precision by using specific indexing terms (“Ferrari” and not “sports car”), little or no stemming, word proximity operators (how closely words appear next to each other), and search zones. Measuring the recall and precision of a particular search engine can be difficult. Measuring recall and precision using hard numbers is questionable. Relevance is difficult to quantify since it is inconstant (even during the course of a single search, relevance may change) and subjective. A better way to get a handle on precision and recall is to collect responses from your users. What do people complain about? Do they say, “I get too many results?” This really means, “I get too many irrelevant results” and is a sign your recall might be too high. Do people say “I know it is in there, but I can't find it?” or “I get no hits for too many searches?” If so, you might have precision too high. Just remember, recall and precision are inversely related: as one goes up, the other goes down. You will need to strike a balance.

Authority files

So now that we know what a synonym ring is, we can define an authority file. An authority file is similar to the synonym ring, with the addition of one type of term relationship. Instead of all of the terms being equal, one term is identified as the preferred term and the others are considered variant terms. Authority files help with tagging content consistently. Catalogers for large library collections have long used authority files to find approved terms for describing an item. When they get a book about the Italian city of Firenze and another one about the Italian city of Florence, they use one of the names (based on prescribed rules) and describe all books in the collection about the city using a single, consistent term. Similarly, in most major academic libraries, all books about “Native Americans” and “American Indians” are described with the term “Indians of North America.” When someone performs a subject search on “Native Americans” they get a note that says something like “This term is indexed as INDIANS OF NORTH AMERICA.” The authority file is the place you go find which term is the heading (the main term) and which term is the cross reference (the variant term). A more typical example on a website might work like this: Let's say you have a website devoted to comic books. It would be great if when someone typed “Caped Crusader” or “Dark Knight” into the search box, they got results for “Batman.” In this example, “Batman” and “Caped Crusader” would not be considered equivalent terms; the authority file would explain their relationship. You would not want to identify each Batman comic book with all three terms, just the main term. But when a user entered “Caped Crusader,” you would want the system to convert their term to “Batman” and return the appropriate results. The relationships among the terms could be expressed like this: Or in the language of a controlled vocabulary, like this:

Batman
USE FOR: Dark Knight, Caped Crusader

Caped Crusader
USE Batman

Dark Knight
USE Batman

Another way that people use authority files is to reinforce a correct term and to discourage an incorrect term. The Polar Bear Book uses the example of how drugstore.com corrects the spelling of Tylenol using an authority file. If you enter “tilenol” into the site's search box, you get the results for “Tylenol.” Users will see the correct spelling prominently displayed, which will remind them how the word is really spelled. Maybe they will remember the correct spelling in the future.

Guidelines for implementation

When putting a synonym ring or authority file in place, consider the following guidelines:

Show users how their search term was changed or added to by the system and exactly what was searched. At International SEMATECH, a search for Silicon would look like this:
The line under the search box tells users exactly how the search was submitted. When users understand how their term is expanded to include synonyms, they have a better understanding of how the site works. When done well, explanations can also increase confidence that users have in the system, since it shows them that the system understands what they are looking for.
Keep the display simple. Include a search box at the top of the page so users can edit their terms if they see they have made a mistake. Try to follow the prescient words of the old poem:
Give me a look, give me a[n inter] face,
That makes simplicity a grace;
— Ben Johnson (slightly modified) [http://www.bartleby.com/100/146.9.html]
Try to characterize your content and the way your users understand it. At International SEMATECH, the majority of the synonym ring we use is made up of acronyms, since the scientific community seems to love creating and communicating with them. The content is also very narrow and scientific. There is not a great deal of the mushy language that comes from the general culture; most of it is very well defined. A general rule: The broader the content your site covers, the more you will find yourself dealing with near synonyms. Try to make similar evaluations of the content you are searching.
Review search logs every day to look for new terms and synonyms. Is someone looking for an acronym that is not on your list? Try to find out what it means and make sure the next person looking for it gets the correct results.

Conclusions

Synonym rings and authority files are simple, common-sense ways to help users connect the various semantic concepts that are inherently intertwined with the term they choose. They are particularly good for large decentralized sites that are search dominant and have little centralized control over content. Most of us know by now that users tend to use a small number of words for each search. They should not be forced to consider all the synonyms their search terms might have. Tim Bray said it well: “If you need to know about cow farming, you're probably also searching for cattle ranching, beef (or dairy) production, and Kuhbauernhof, whether you know it or not.”

All About Facets and Controlled Vocabularies (series introduction)
What is a Controlled Vocabulary?
Creating a Controlled Vocabulary
An Annotated Bibliography

Fred Leise, president of ContextualAnalysis, LLC, is an information architecture consultant providing services in the areas of content analysis and organization, user experience design, taxonomy and thesaurus creation, and website and back-of-book indexing.

Mike Steckel is an Information Architect/Technical Librarian for International SEMATECH in Austin, TX.