Validity and Think-Aloud Protocol (Toler, 2009)

Toler, T. (2009). Validity and Think-Aloud Protocol. Solid State UX Blog.

Toler provides a nice primer on the think aloud protocol, clarifying the different approaches and discussing the pros and cons of each variation.The validity of think-alouds derives largely from the fact that they are a direct measure of what’s happening in a subject’s short-term memory. Other direct measures of human cognition include response tests (e.g. reaction time indicators) and MRI brain scans.

If a subject is steadily verbalizing while performing a task (e.g. concurrent verbalization), they are assumed to be speaking from short-term memory.

3 levels of think-aloud protocols (Dumas & Redish, 1993):

  • Level 1 verbalizations: where the emphasis is on pure thoughts with no or minimal explanations
  • Level 2 verbalizations: same, but the participant is dealing with non-verbal information, like shapes, which must be internally “coded” in order to be articulated verbally
  • Level 3 verbalizations: or “thinking plus explanations.” Also referred to as retrospective reports.

At Level 3, researchers are “no longer getting a read out of short-term memory… rather it is the interpretation of the process they are using or the reasons they have selected a strategy.” Retrospective reports are less valid, but they are necessary to clarify a respondent’s statements and actions.

Active vs. inactive moderation:

  • Inactive moderation: emphasis is on experimental control and creating a unified experience for all test subjects. Researchers stand behind the glass and the respondent sits in the room by themselves talking out loud. Participants are typically “coached” on how to deliver a think-aloud protocol. During the actual experiment, the moderator prompts only when the participant ceases to verbalize.
  • Active moderation: the experimenter employs probing questions to focus the participant’s attention on particular features or to elicit and clarify subjective explanations of their behavior. Moderators are skilled in asking neutral, non-leading questions to minimize bias. In addition, active listening techniques are employed to emulate the clinician’s empathic stance. Rule of thumb for “non-directed” interviewing: questions should be concentrated on immediate experience, nonjudgmental, focused on a single topic, open-ended, and non-binary (e.g. yes-no, true-false).

Taylor and Dionne (2000): probes are best deployed for collection and verification of data in retrospective reports; they have a detrimental impact on validity if used during concurrent think-aloud protocols.

Preece (1994): the role of the moderator on the participant is both interruptive and imposes additional cognitive load.

Nielsen (1993): the moderator should intervene as little as possible, yet direct the flow and direction of the interview to maximize the number of usability issues found.

Most clients of usability research do not share the academic’s interest in validity. They want enough validity to feel good about the process and the results, but ultimately they want their specific questions answered for a reasonable amount of time and expense.

Nielsen et. al. (2002): “The human is a psychological being engaged in a psychological interaction, which cannot be reduced to that which is concurrently verbalized.”


DUMAS, J.S. (2001) “Usability Testing Methods: Think-Aloud Protocols,” in Design by People For People: Essays on Usability, UPA, pp 119-129

DUMAS, J.S. & REDISH, J.C., (1993) A Practical Guide to Usability Testing.
Norwood, NJ, Ablex Publishing Corp.

ERICSSON, K.A. & SIMON, H.A. (1984, 1993) Protocol analysis: Verbal reports as data (Rev. ed). Cambridge, MA: MIT Press

KUNIAVSKY, M. (2003) Observing the User Experience: A Practitioner’s Guide to User Research, San Francisco: Morgan Kaufmann Publishers, Inc.

NIELSEN, J., (1993) Usability Engineering. Chestnut Hill, MA: Academic Press, Inc.

NIELSEN, J., CLEMMENSEN, T., & YSSING, C., (2002) “Getting access to what
goes on in people’s heads? – Reflections on the think-aloud technique”, paper presented to NordiCHI, Arhus, Denmark, October 19-23

PREECE, J. (1994), Human-Computer Interaction, Addison-Wesley, England

TAMLER, H. (2001) “How (Much) to Intervene in a Usability Testing Session,” in Design by People For People: Essays on Usability, UPA, pp 165-171

WHITESIDE, J., BENNETT., J.L., & HOLTZBLATT, K., (1988) “Usability
Engineering: Our Experience and Evolution,” in Handbook of Human Computer
Interaction; edited by Helander, M. New York, NY: Elsevier Science Publishers

One Response to “Validity and Think-Aloud Protocol (Toler, 2009)”
Check out what others are saying...
  1. […] Theodorou, E. (2010). Let the Gamers Do the Talking: A Comparative Study of Two Usability Testing Methods for Video Games. Retrieved From University College London: […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: