Experimental Design

Comments on the paper Research Methods in Human-Computer Interaction, by Thomas K. Landauer, in Handbook of Human-Computer Interaction, M. Helander (ed.), Elsevier, pp. 905-928, 1988.

Good points

  1. Landauer's approach is typically very pragmatic. He starts right off by placing research into its proper context: as the engineering solution to factors such as the egocentric intuition fallacy and "the unrelability of intuition and the variability of behaviour." I have been amazed at how firm a designer's belief in the usability of his own, untested and unmeasured, work can be (and I am no exception to this -- it is an easy trap to fall into); Landauer puts this into its correct perspective.
  2. His point about richness in a research design: allow a large number of expected and unexpected outcomes; allow many things to happen and be observed; take multiple measures rather than one. His cautions about knowing what you are testing are well-taken, as well: testing only for ease-of-use, for example, can produce systems that are easy for novices to use but not for experienced users (Norman's "knowledge-in-the-head"); testing a single feature or interaction sequence (e.g. time to position the cursor) may not yield useful results in a more complete context.
  3. Test early, test early, test early. Like the paper prototyping paper, the earlier testing can be performed, the less expensive changes will be. (This resonates with some maxims of software engineering in general: the "release early, release often" principle of free software described in The Cathedral and the Bazaar; the pushing forward of formal inspections to the design and even architectural stages of a project.)

Bad points

Bad points are mostly nit-picky, really...

  1. I found the thread of the paper wondered somewhat. Partly, this is because of its length. Still, in the main section on Research Designs, I found myself constantly referring back to other sections in order to place the discussion back into context.
  2. I wanted to know more about factorial design. Landauer twice suggests how powerful this technique is, but gives only cursory detail.
  3. Overall, I thought the paper suffered from the usual problem of overview papers, which is too much breadth and not enough meat. I didn't feel, after reading it, that I knew very much about the nitty-gritties of designing an experimental evaluation of a user interface.

John Reekie, February 12th, 1998.