Design Experiments in Educational Research (Cobb et al., 2003)

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32, 1: 9 – 13.

A great introduction to Design Experiments — what they are, why they are useful in educational research. The authors also discuss some common issues with implementation and post-hoc analysis of data. Overall, design experiments are iterative, interventionist, and theory-oriented; also they speak to practical educational contexts and concerns.  

“Design experiments have both a pragmatic bent — engineering” particular forms of learning — and a theoretical orientation — developing domain specific theories by systematically studying those forms of learning and the means of supporting them” (p.9).

These theories often target domain-specific learning processes (i.e., a students’ learning of key disciplinary ideas in that domain — in other words, the sequence and patterns of students’ reasoning and how those patterns can be supported).

“Design experiments ideally result in greater understanding of a learning ecology — a complex, interacting system involving multiple elements of different types and levels — by designing its elements and by anticipating how
these elements function together to support learning. Design experiments therefore constitute a means of addressing the complexity that is a hallmark of educational settings” (p. 9).

(p. 9) Elements of a learning ecology: student tasks or problems, classroom discourse, norms of participation, the tools and related material means provided, classroom teachers’ actions. Ecology implies a series of interacting systems rather than a collection of activities or a list of separate factors that influence learning.

“Beyond just creating designs that are effective and that can sometimes be affected by ‘tinkering to perfection,’ a design theory explains why designs work and suggests how they may be adapted to new circumstances. Therefore, like other methodologies, design experiments are crucibles for the generation and testing of theory” (p. 9).

5 features of design experiments:

1) They seek to develop theories about the process of learning and the means to support that learning. Processes of learning may include the “evolution of learning relevant social practices and even constructs such as identity and interest” (p. 10). The means for supporting learning may include ” the affordances and constraints of material artifacts, teaching and learning practices, and policy levers (e.g., performance-based pay), as well as other forms of mediation that might, for example, include the negotiation of domain-specific norms, such as what counts as a ‘good’ scientific question in a classroom (Wertsch, 1998)” (p. 10).

2) They are highly interventionist in nature. “Design studies are typically test-beds for innovation. The intent is to investigate the possibilities for educational improvement by bringing about new forms of learning in order to study them. The design developed while preparing for an experiment draws on prior research and attempts to cash in the empirical and theoretical results of that research. By its very nature, the study of phenomena as complex as learning ecologies precludes complete specification of everything that happens. It is therefore all the more important to distinguish in the specification of the design between elements that are the target of investigation and those that may be ancillary, accidental, or assumed as background conditions. The use of prior research to both specify a design and justify the differentiation of central and ancillary conditions is central to the methodology” (p. 10).

3) They test theories, during the process of which new ones form. “…Design experiments always have two faces: prospective and reflective. On the prospective side, designs are implemented with a hypothesized learning process and the means of supporting it in mind in order to expose the details of that process to scrutiny. An equally important objective is to foster the emergence of other potential pathways for learning and development by capitalizing on contingencies that arise as the design unfolds. On the reflective side, design experiments are conjecture-driven tests, often at several levels of analysis. The initial design is a conjecture about the means of supporting a particular form of learning that is to be tested. During the conduct of the design study, however, more specialized conjectures are typically framed and tested” (p. 10).

4) They contain cycles of iteration. “As conjectures are generated and perhaps refuted, new conjectures are developed and subjected to test. The result is an iterative design process featuring cycles of invention and revision…The intended outcome is an explanatory framework that specifies expectations that become the focus of investigation during the next cycle of inquiry” (p. 10).

5) They are pragmatic. “Theories developed during the process of experiment are humble not merely in the sense that they are concerned with domain-specific learning processes, but also because they are accountable to the activity of design. The theory must do real work. General philosophical orientations to educational matters — such as constructivism — are important to educational practice, but they often fail to provide detailed guidance in organizing instruction” (p. 10).

When planning a design experiment, it is important to ask: “Does the theory informs prospective design and, if so, in precisely what way?” (p. 11).

“…Design experiments tend to emphasize an intermediate theoretical scope (diSessa, 1991) that is located between a narrow account of a specific system (e.g., a particular school district, a particular classroom) and a broad account that does not orient design to particular contingencies…In contrast to most research methodologies, the theoretical products of design experiments have the potential for rapid pay-off because they are filtered in advance for instrumental effect. They also speak directly to the types of problems that practitioners address in the course of their work” (p. 11).

Preparing for a design experiment:

  • Learning Trajectory: The research team must clarify the theoretical intent by  being able to “identify and account for successive patterns in student thinking by relating these patterns to the means by which their development was supported and organized” (p.11).
  • Prospective Endpoints: The team must “specify the significant disciplinary ideas and forms of reasoning that constitute the prospective goals or endpoints for student learning. This usually involves drawing on and synthesizing the prior research literature to identify central organizing ideas for a domain (e.g., the notion of distribution as a central idea for statistical analysis, Lehrer & Schauble, 2002; McClain, Cobb, & Gravemeijer, 2000)” (p. 11).
  • Starting Points: The team must specify “its assumptions about the intellectual and social starting points for the envisioned forms of learning. To achieve the instructional agenda, the team identifies current student capabilities, current practices, and other resources on which it might be able to build” (p. 11).
  • Testable Conjectures: The team then plans a design that makes “testable conjectures about both significant shifts in student reasoning and the specific means of supporting those shifts” (p. 11).

Conducting a design experiment:

4 important functions that require ongoing direct engagement in the research setting (p. 12)

1) A clear view of the anticipated learning pathways and the potential means of support must be maintained and communicated within the research team, even while responding to contingency.

2) Cultivation of ongoing relationships with practitioners. These relationships are sustained by the negotiation of a shared enterprise, which is typically developed over the long haul as lead researchers consistently demonstrate their personal commitment.

3) Development of a deep understanding of the ecology of learning — not simply to facilitate logistics, but because this understanding is a theoretical target for the research.

4) Regular debriefing sessions are the forum in which past events are interpreted and prospective events are planned for. These sessions are the sites where the intelligence of the study is generated and communicated.

It is important that the team generates a comprehensive record of the ongoing design process (via audio records of meetings and logs to document the evolving conjectures, together with the observations that are viewed as either supporting or questioning a conjecture) (p. 12). In addition, design experiments produce a complex array of data sources that include products of learning, such as student work; classroom discourse; body posture and gesture; tasks and activity structures; patterns of social interaction; inscriptions, notations, and other tools; and responses to interviews, tests, or other forms of assessment.

Conducting restrospective analysis:

“An educational accomplishment is characterized by contingency in which earlier events open up, enable, and also constrain the events that follow. Accounting for this process requires an historical or retrospective explanation, one that provides a trustworthy account of the process whereby a series of events — each of which is local and contingent ·- can be seen as part of an emergent and potentially reproducible pattern” (p. 12).

A primary aim is to place the design experiment in a broader theoretical context, thereby framing it as a paradigm case of the more encompassing phenomena specified at the outset. In this regard, retrospective analyses can be contrasted with the analyses conducted while the experiment is in progress in that the latter are typically oriented toward the goal of supporting the learning of the participants.

“‘What works’ is underpinned by a concern for ‘how, when, and why’ it works, and by a detailed specification of what, exactly, ‘it’ is. This intimate relationship between the development of theory and the improvement of instructional
design for bringing about new forms of learning is a hallmark of the design experiment methodology (p. 13).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: