Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings (Brown, 1992)

Brown, A. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. The Journal of the Learning Sciences, 2(2): 141-178.

Brown provides a broad commentary on design experiments, including important theoretical and methodological issues.  She also calls for new and complex methodologies to capture the systemic nature of learning, teaching, and assessment and reminds readers of the history of prior attempts to reorganize school and work environments.

Brown characterizes her work as the “study of learning in the blooming, buzzing confusion of inner-city classrooms” (p. 141). Inherent in this is a convincing rationale for considering one’s research in relation to a system of interrelated components. “Thus, we are responsible for simultaneous changes in the system, concerning the role of students and teachers, the type of curriculum, the place of technology, and so forth. These are all seen as inputs into the working whole. Similarly, we are concerned with outputs from the system, a concern that -leads us to look at new forms of assessment. It is essential that we assess the aspects that our learning environment was set up to foster, such as problem solving, critical thinking, and reflective learning” (p. 143).

“Another critical tension in our goals is that between contributing to a theory of learning, a theoretical aim that has always been a keystone of our work, and contributing to practice. This is intervention research designed to inform practice…an effective intervention should be able to migrate from our experimental classroom to average classrooms operated by and for average students and teachers, supported by realistic technological and personal support” (p. 143-4).

“Educationalists have been interested in the problem because of two tumbling blocks to lasting learning: (a) inert knowledge (Whitehead, 1916): students acquire facts that they cannot access and use appropriately; and (b) passive learning: students do not readily engage in intentional, self-directed action. These two problems are commonly regarded as diseases of schooling (Brown, 1977). Attempts to study these phenomena led to a research agenda that saw fundamental changes in the focus of developmental theory, an agenda that focused on active strategies for learning, what it means to learn (rote learning vs. understanding), the content that learners are to acquire, ad the context in which they are to acquire it” (p. 146).

The tension between and relative contributions of classroom and laboratory studies:

A great example of the necessity of classroom studies is demonstrated by Brown’s recounting of how the development biological theories was understood to occur, based on lab studies alone: “Although in the laboratory this development from noticing to using, and from surface to deep, was thought to be age-dependent, the classroom work suggests that the shift is knowledge-based, occurring microgenetically within a year as readily as cross-sectionally across several years” (p. 154?)

Classroom work can also motivate laboratory practice: “Trends discovered in spontaneous classroom discussions can be tested in the laboratory under more controlled conditions” (p. 154).

Important decisions to make when planning a study [methodological issues]:

1) Grain size: “The classic distinctions are laid out neatly in introductory textbooks. Developmental psychologists are informed of the strengths and weaknesses of several such options. First there is the choice between an idiographic and a nomothetic approach, that is, few or many. Does one want to study a “single variable in many subjects for the purpose of discovering general laws or principles of behavior” or adopt the idiographic approach, ‘the thorough study of individual cases, with emphasis on each subject’s characteristic traits.’

2) Time scale: “One major option is between cross-sectional designs, where data are taken from, say, 7-, 10-, and 14-year-olds, and inferences about developmental trajectories made; and longitudinal designs, where, for example, all 4-year-olds who watched ‘Sesame Street’ in 1969, 1971, and 1973 are followed up for a period of years. Another option is the microgenetic design, an important tool in the psychologist’s kit that is receiving renewed interest. Here children are observed over a relatively short period of time (days, weeks) as they acquire a certain form of understanding. This approach is most often taken with very young children who are at a stage of rapid learning (DeLoache, Brown, & Kane, 1985) but also with older children learning a particular scientific concept (Karmiloff-Smith & Inhelder, 1974-1975; Kuhn, Amsel, & O’Loughlin, 1988)” (p. 155-6).

A nice recipe for a paper: “…in the initial study of reciprocal teaching Palincsar & Brown, 1984), we provided pretest and posttest data on the 37 participating students, mini case studies on six children, together with transcripts from two children who differed considerably in how quickly they picked up the reciprocal teaching procedure, a precedent we have allowed in more recent work (Brown & Palincsar, 1987, 1989). This approach enables us to see the magnitude of the effect in terms of outcome measures and to get a feel for the phenomenon itself by looking at a articular child or group in depth. (Brown et al., in press; Campione et al., in press)” (p. 156).

3) Data selection: Especially important “when portions of edited transcripts or clinical interviews are selected to illustrate a theoretical point, or when descriptions of planning sessions, peer tutoring, or teacher coaching are culled from a vast array of potential examples. With access to daily ethnographic notes, teacher’s logs, and video and audio tapes, it is clear that we must select a very small sample from a large data base, and that selection is obviously going to buttress our theoretical stance. This selection issue is nontrivial. The problem is how to avoid misrepresenting the data, however unintentionally” (p. 162). Beware of the Barlett Effect! (see p. 162).

Common criticisms (and Brown’s reactions to them):

1) The Hawthorne Effect (Roethlisberger & Dickson, 1939) — the nature of control in multifaceted interventions.

“The Hawthorne Effect, as it appears in standard texts, refers to the fact that any intervention tends to have positive effects merely because of the attention of the experimental team to the subjects’ welfare. I have never taken the Hawthorne criticism of my work seriously because of the very specific nature of the improvements obtained. If I were creating a true Hawthorne effect, I would not be able to predict which performance would improve. But in fact we see a close coupling of the cognitive activities practiced and the type of improvements shown” (p. 162)

2) The Dewey Effect: the romanticization of the process of discovery.

“Although it is commonplace for teachers to be called upon to foster ‘discovery,’ the role of the teacher in discovery learning classrooms is still largely uncharted. Invoking comfortable metaphors such as the teacher as coach does not tell us how and when the teacher should intervene. We know that challenging students’ assumptions, providing them with counter-examples to their own rules, and so forth are good instructional ploys. But how intrusive should teachers be, when should they guide, when should they teach? When should they leave well enough alone? In short, how can teachers foster discovery and at the same time furnish guidance?” (p. 169-70).

“Although recognizing an intellectual debt to Dewey, it is also true that a great deal of theoretical and methodological work is still needed if we are to render these ideal educational types into anything close to reality. Collins (in press) argued that educational theorists (Plato, Rousseau, Dewey, Bruner, and Illich) have addressed themselves to the process of education for many years. The contemporary agenda is for experimentalists to fulfill this legacy with concrete design experiments. Such experiments must provide a level of description that would afford the opportunity to uncover mechanisms of learning that are not captured in preexisting theoretical descriptions. So in response to the criticism, “It’s all Dewey, what’s new?” the answer is, nothing and everything” (p. 171).

3) The Reality Principle: the problem of the shelf life of successful interventions.

“In the light of the poor long-term track record of the laboratory schools of Binet, Dewey, and others, one must consider the shelf life of educational interventions warily. Received wisdom tells us that such innovations as the Dewey school and the science curriculum reforms of the 1960s were experiments that were tried and failed. But it is not clear what was meant by tried or failed, nor is it clear what the criteria were against which success or failure is measured (Dow, 1991; Shymansky, 1989). Historians tend to be pessimistic about educational reform (Cuban, 1984, 1986, 1990), an attitude of “la plus ca change” (Cohen, 1989) prevails. The argument is that successful interventions are a chimera or at least are extremely fleeting and fragile, not readily transportable to settings outside the innovator’s control. Because of this skepticism, it is extremely important for the design experimenter to consider dissemination issues. It is not sufficient to argue that a reasonable end point is an existence proof, although this is indeed an important first step” (p. 171).

Characteristics of an intervention with “a long shelf life”:

“In this light, the long-term track record of reciprocal teaching is encouraging; the method enjoys widespread dissemination compared with other innovative instructional designs with which I have been involved. I believe that the reasons for this are (a) it looks deceptively easy to implement, (b) the term reciprocal teaching has been picked up by researchers, teachers, and textbook publishers and has become part of the discourse of the reading community, and (c) most important, the procedure slots neatly into a hallowed classroom niche (Cuban, 1984). Reading group has a long-standing place in the school day-teachers are used to arranging their classes to accommodate reading group; all (all!) that is needed is to redefine the activities that take place in a socially sanctioned niche” (p. p. 171)

“To say the least, it is a cautionary note for contemporary designers that Dewey (1901) a century ago warned that educational reform would not be easy to engineer. Dewey’s description of cycles of innovation and resistance sounds uncannily like Cuban’s (1984, 1990) contemporary Cassandra bulletins. First comes unrest concerning the schools and how they operate, followed by fervent claims and promises from reformers. Intensive research by the converted is then carried out in a small set of classrooms rich with human and, today, technological resources. ‘The victory is won and everybody—unless it is some already overburdened and distracted teacher— congratulates everybody else that such advanced steps can be taken’ (Dewey, 1901, p. 334). But then come the frustrated attempts by ordinary teachers to adopt the new methods in the absence of support, followed by the inevitable decline in use, and the eventual abandonment of the program. As Dewey argued, ‘within a short time, complaints are heard that children do not read as well,’ ‘or a public outcry calls for the reforms to be rescinded in favor of the status quo.’ One major question facing contemporary designers is how to avoid repeating the Cuban-Dewey cycle: exhilaration, followed by scientific credibility, followed by disappointment and blame. I see the problem as analogous to the alpha, beta, and gamma phases of software development. The alpha, or developmental, phase is under the control of the advocate, and by definition it must work for there to be any later phases. It works, though, under ideal supportive conditions. Next comes the beta phase, tryouts at carefully chosen sites with less, but still considerable, support. Critical is the gamma stage, widespread adoption with minimal support. If this stage is not attempted, the shelf life of any intervention must be called into question” (p. 172).

“My classroom research agenda now is far less that of the frustrated Hawthorne experimenter trying to unconfound variables for explanatory power, although I still care about control issues. My agenda is more like that of a designer or engineer. I need to unconfound variables, not only for theoretical clarity, but also so that necessary and sufficient aspects of the intervention can be disseminated. The question becomes, what are the absolutely essential features that must be in place to cause change under conditions that one can reasonably hope to exist in normal school settings?” (p. 173).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: