Game Usability: Advice from the Experts for Advancing the Player Experience (Ch. 4, 5)

Amaya, G., Davis, J. P., Gunn, D.V., Harrison, C., Pagulayan, R. J., Phillips, B., & Wixon, D. (2008). Games User Research (GUR): Our Experience with and Evoluation of Four Methods. In K. Isbister and N. Schaffer (Eds.), Game Usability: Advice from the Experts for Advancing the Player Experience (pp. 35-64). New York: Morgan Kaufmann.

Hoonhout, H. C. M. (2008). Let the Game Tester Do the Talking: Think Aloud and Interviewing to Learn About the Game Experience. In K. Isbister and N. Schaffer (Eds.), Game Usability: Advice from the Experts for Advancing the Player Experience (pp. 65-77). New York: Morgan Kaufmann.Chapter 4

In this chapter, the Games User Research group at Microsoft Game Studios provides an overview of their user research on in-house games.

The authors point out that an important thing to keep in mind is that games fall into the category of discretionary products; therefore research in games should focus both on what users do and how they feel about what they are doing. Factors like usefulness and usability are secondary and relevant to the extent that they affect the likelihood of a rich and engaging experience. Consideration of the user’s experience is primary for games since their sole purpose is to provide a positive emotional experience to the player. [maybe creating a positive experience can enhance any learning being imparted? Look through affective cognition research…]

In usability testing, the focus is on players’ behavior to reveal areas win which the player experience does not match with the design intent (behavioral data). Measure aspects such as time to completion or frequency and nature of errors for a task. In playtesting, the goals is the same, though the method focuses on players opinions (attitudinal data). Playtesting can reveal more about what the subjective experience of playing the game is like.

Discount usability methods don’t necessarily do a good job of quantifying users’ opinions in a reliable fashion. To do that, the authors employ their own version of the playtest, a large-sample, survey-based methodology to measure and quantify users’ perceptions, attitudes, and opinions about  a game. Recently, the group has been expanding their program to also include ancillary behavioral data to provide a more holistic view of the user experience. System for tracking user behavior in real-time in their Playtest labs called TRUE instrumentation (Tracking Real-time User Experience).

Three things critically important to a Playtest program: standardization; reference data; focused research questions.

Using beta-versions of the product for testing:

Some game research questions may require hundreds or even thousands of participants to answer. Release a beta version of the product (aka community technology previews). Compared to the conditions of a playtest, the game needs to be more stable (not being used in the testing facility); more difficult to collect qualitative feedback and understand context; less control over the test and managing player behavior; difficult to update builds; beta testing allows for extended testing (weeks or months).

In order to collect gameplay feedback such as difficulty levels, need to recruit a wider variety of players.

Chapter 5

This chapter discusses the use of think-alouds in user game research. A think-aloud, in which the player is asked to verbalize their thinking while working through a task, allows the researcher to get a “glimpse” of what is going on inside a player’s head. These verbalization reports are often called verbal protocols.

Practically any question regarding the usability aspects of a game interface can be addressed in a verbal protocol analysis. Typical research questions include:

  • Does the game pose an interesting and adequate challenge? (Is the challenge due to game content or game controls?)
  • Does the game stay challenging til the end of the test session? Does it stay challenging after several sessions?
  • How do the different elements in the application contribute to the user’s experience?
  • How easily can the player learn how to work with the application?
  • For multiplayer games, how does the (social) interaction develop?

Note that think-alouds are less suitable for questions about the level of enjoyability of a game or to investigate the potential engaging power of a game (bc asking a player to “think aloud” while playing changes the experience).

Influential publications:

Ericsson, K.A., & Simon, H.A. (1984). (A revised edition was published in 1993) Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.

Verbalizations simply refer to what the participant is attending to, and in what order. Any inferences that the participant makes about her own cognitive processes or opinions should not be considered for analysis due to unreliability.

Nisbett, R.E., & Wilson, T.D. (1977). Telling more than we know: verbal reports on mental processes. Psychological review, 84(3), 231-241.

Verbalizations will mainly contain observed facts and results of decisions that then can be interpreted by the researcher, in order to come to an idea of hte possible underlying cognitive processes. Overall, it is the opinion of the author that “…a usability evaluation context adopting a less strict approach to think-aloud studies than advocated by Ericsson and Simon does result in usable and useful data about important product aspects” (p. 67).

Two main approaches: Concurrent and retrospective. Both produce comparable results in terms of number and relevance. Concurrent think-alouds more likely lead to “observable” problems. In terms of testing games, retrospective think-alouds are preferred, in order to preserve the experience during game interaction. Concurrent think-alouds may be used to examine game interface usability.


  • Collecting and analyzing think-aloud data can be time consuming.
  • Verbal protocols are less suitable for asking participants to describe in detail the cognitive processes that they employ during game interaction (i.e., make inferences about their own cognition). It is up to the researcher to infer those processes.
  • Thinking aloud may change the way the task is performed. The participant may realize that the task could be performed a different way, or go for the one that is more easy to describe, or perform a task sequentially, rather than simultaneously.
  • Some actions may be hard to verbalize or some steps may be omitted.
  • Think-alouds are not practical for games that require players to talk to each other or social situations.
  • Not suitable for tasks that can be performed in an automated or virtually effortless manner.
  • Participants may consciously try to avoid sounding irrational, hesitant, unknowing, or foolish.
  • Biases may be even more likely in a retrospective think-alouds, when participants have time to consider their responses.
  • Thinking aloud may seem unnatural to many people. For this reason, children are sometimes allowed to work in pairs.

**Good paper to understand inter-rater reliability: Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.

**Handbook that describes how to conduct verbal protocol studies: Van Someren, M.W., Barnard, Y.F., Sandberg, J.A.C. (1994). The Think Aloud Method. A Practical Guide to Modelling Cognitive Processes. London: Academic Press.

**To learn more about interview studies:

Oppenheim, A.N. (2000). Questionnaire Design, Interviewing, and Attitude Measurement (2nd edition). London: Continuum.

Wilson, J.R., & Corlett, E.N. (Eds.) (2005). Evaluation of Human Work. Taylor and Francis, London.

One alternative approach is simply to collect and analyze any comments made by participants while engaging in the software/game. Should be used in combination with other forms of data collection (logging of interactions, recording observable behavior, conducting closing interview, administering questionnaires, etc.)

“Think-aloud will be an appropriate technique to inspect the usability aspects of the game interface, but seems to be much less suitable to study the “fun factor” of a game–basically, thinking aloud while playing a game has a large impact on the game experience, because it involves basically conducting two tasks at the same time: interacting with the game and verbalizing one’s thoughts on that” (p. 75).

Interviewing after gameplay may be better for collecting feedback on the enjoyability of a game, though additional measures (observations, physiological measures, questionnaires, etc.) is also recommended.

Interviews provide more general information about the game and the game interface, whereas verbal protocols provide more information about details of the interaction in a particular context.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: