Design-Based Research Design-Based Research: Putting a Stake in the Ground (Barab & Squire, 2004)

Barab, S. & Squire, K. (2004). Design-Based Research: Putting a Stake in the Ground. The Journal of the Learning Sciences, 13(1): 1-14.

In this opening article in a special journal issue devoted to design-based research, the authors discuss many of the “hot issues” around DBR, including how to evaluate the credibility of findings based on DBR and the acceptable roles of researchers/designers undertaking DBR.

Why lab-based research and pure ethnography is insufficient: “If one believes that context matters in terms of learning and cognition, research paradigms that simply examine these processes as isolated variables within laboratory or other impoverished contexts of participation will necessarily lead to an incomplete understanding of their relevance in more naturalistic settings (Brown, 1992). Alternatively, simply observing learning and cognition as they naturally occur in the world is not adequate given that learning scientists frequently have transformative agendas” (p. 1-2).

Table comparing psychological experimentation and design-based research methods:

Findings from DBR should be relevant to other contexts. Stake (1995) called it a “petite generalization.” Clifford Geertz (1976, 1983) preferred to say that work should show experience-near significance and experience-distant relevance.

Differences between DBR and formative studies. “What separates design-based research in the learning sciences from formative evaluation is (a) a constant impulse toward connecting design interventions with existing theory, (b) the fact that design-based research may generate new theories (not simply testing existing theories), and (c) that for some research questions the context in which the design-based research is being carried out is the minimal ontology for which the variables can be adequately investigated (implying that we cannot return to the laboratory to further test the theoretical claims)” (p. 5).

Formative evaluation methodologies and instructional design models seek to improve a particular designed artifact or design process, whereas DBR is concerned with using design to develop models of how humans think, know, act and learn — to not only meet local needs, but to advance a theoretical agenda, to uncover, explore, and confirm theoretical relationships (p. 5).

“Design-based research requires more than simply showing a particular design works but demands that the researcher (move beyond a particular design exemplar to) generate evidence-based claims about learning that address contemporary theoretical issues and further the theoretical knowledge of the field” (p. 5-6).

DBR has a pragmatic philosophical underpinning, in which the value of a theory lies in its ability to produce changes in the world, in its “ability to do work in the world” (Dewey, 1938; Peirce; both promulgated “systems of inquiry rooted not in claims of truth, but rather in the viability of theories to explain phenomena and produce change in the world” p. 7).

Advancing credible assertions in DBR — what counts as credible evidence?

“Schoenfeld (1992) argued that a sound methodological argument in the social sciences should touch on issues of trustworthiness, credibility, and usefulness as well as the range of contexts in which the researcher believes the assertions should extend” (p. 7).

“The emphasis on understanding the value of a theory through its consequences on naturalistic systems also borrows from Messick’s (1992) notion of evidence of consequential validity for testing. His argument is that the validity of a claim is based on the changes it produces in a given system. These changes or consequences can then be considered evidence in support of validity. Messick’s original formulation of consequential validity argues that inquiry is a social enterprise and evidence for the validity of an assertion can be gathered by examining the effects of that assertion on a system; a classic violation of this principle is when standardized tests result in undesirable practices in schools and routinized, shallow learning, suggesting that perhaps standardized tests are a poor instrument for generating assertions about student achievement (Linn, 1998)” (p. 8).

“Design-based research offers a mode of inquiry that embraces this notion of consequential validity, but design researchers need to be clearer about the kinds of claims they make from design experiments and the limitations of their findings” (p. 8).

“…the goal of design-based research is to lay open and problematize the completed design and resultant implementation in a way that provides insight into the local dynamics. This involves not simply sharing the designed artifact, but providing rich descriptions of context, guiding and emerging theory, design features of the intervention, and the impact of these features on participation and learning” (p. 8).

“Narrative, as one way of making sense of design-based research, is a historical method that involves conveying a series of related plots and describing the temporal unfolding of the design over time (Abbott, 1992; Mink, Fay, Golob, & Vann, 1987). A core challenge in building narrative is what historiographers refer to as the “central subject problem” in which the boundaries of the case itself are delimited (Hull, 1975). Although in some instances the case may have clear boundaries, more often than not a crucial difficulty lies in “drawing boundaries around the central subject given the continuous character of the social manifold” (Abbott, 1992, p. 63)” (p. 8).

“It is the unpacking of these transformations, describing what the case endures, and relating these changes to underlying theory that philosophers refer to as the ‘colligation’ problem. The important and somewhat disheartening point with respect to this problem is an appreciation of each event being complex, enduring multiple transformations, having multiple antecedents, and resulting in a myriad of consequences (Isaac, 1997). This led Abbott (1992) to discuss a case as a sequence of major turning points (kernels) and sets of situational consequences flowing from these kernels. As such, a fundamental challenge in presenting design narratives lies in uncovering these events so that the reader understands their complexity but doing so in a way that lends itself global relevance while at the same time meaningfully capturing the dynamic unfolding of the phenomena” (p. 9).

DBR involves the “production and testing of theory that can be used to generate, select, and validate specific design alternatives; revealing how various designs predicated on different theoretical assumptions are differentially consequential for learning” (p. 9).

The role of the researcher/designer in DBR

How do we account for the role of the researcher in the design experiments and the associated threats to validity that they bring with them? If a researcher is intimately involved in the conceptualization, design, development, implementation, and researching of a pedagogical approach, then ensuring that researchers can make credible and trustworthy assertions is a challenge. Researchers working in schools often face difficult ethical choices. Do they stand idly by andwatch a teacher struggle to use their curricula, or do they intervene providing additional support? Do researchers share stories of struggling students with teachers and allow them to change instruction accordingly, or do they play a “hands-off” role, minimizing their impact on classroom practices?” (p. 10).

“Ironically, although Brown (1992) introduced design experiments in part as a method for developing a richer appreciation of variables as they occurred in naturalistic contexts, her role as context manipulator may have undermined the credibility of her claims. In other words, each systematic alteration of the designed context potentially contributes to the findings and claims being more artificial and less naturalistic” (p. 10).

“In Cobb et al.’s (1999) “teaching experiment” approach, this problem of intervention is turned on its head so that issues that arise in the environment are to be accounted for and integrated into existing theory. It is through understanding the recursive patterns of researchers’ framing questions, developing goals, implementing interventions, and analyzing resultant activity that knowledge is produced. Rather than remain detached from the research context, researchers are implored to intervene where possible, using interventions as opportunities to examine core theoretical issues and explore learning” (p. 10).

Ultimately, the authors feel that researchers must: (p. 10)

  1. Draw on methodological practices consistent with other qualitative methods (e.g., see Glaser & Strauss, 1967; Lincoln & Guba, 1985) to convince others of the trustworthiness and credibility of claims being advanced.
  2. Remember that claims are based on researcher influenced contexts and, as such, may not be generalizable to other contexts of implementation where the researcher does not so directly influence the context.

“The goal is not to ‘sterilize’ naturalistic contexts from all confounding variables so the generated theory is more valid and reliable. Instead, the challenge is to develop flexibly adaptive theories that remain useful even when applied to new local contexts. This potential of flexibly adaptive theory does not result because the theory was somehow generated in a context that was free of confounding situational variables, but rather, because the theory is supple enough to maintain its robustness even in the context of changing situational variables. Theory generated from design-based research, from this perspective, must strike a balance between refinement and adaptability” (p. 11).

“Fishman et al. (2004) argue that most design-based research does not explicitly address systemic issues of usability,
scalability and sustainability…They suggest a conception of design-based research that includes research on innovations in the context of systemic reform and that explores usability in terms of ‘gaps’ between the culture, capability, and policy/management structures. This work pushes us to reconsider the boundaries of context when carrying out design-based research, pushing beyond that which can be designed to a greater appreciation for the constraints of those real-world contexts through which our contexts of implementation are nested” (p. 11-12).

Important words of caution from the authors:

“More generally, as a field we have over-theorized the role of context, and at the same time we have done little to characterize the role of context in ways that can usefully inform our design work. When we leave the relatively impoverished context of the classroom and inquire into phenomenon in more naturalistic contexts, boundaries become less defined and more problematic. Just as we create boundaries for the sake of control and explanation, we need to remember that the world does not divide itself at researcher-defined seams. These seams, rather than being black-boxed or ignored, must be problematized and examined as part of design work, helping to lend both ecological and consequential validity to our work. Ignoring or limiting the fundamental role of context will lead to both impoverished designs as well as under-specified theories that lack generalizable power. As such, much of the design-based research results in boutique projects that have little impact beyond the researcher’s vita. As a community we must work to conceptualize and inquire about the material, social, and cultural contexts through which our work takes on meaning” (p. 12).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: