Cambridge Handbook of Multimedia Learning — Ch. 4 (Schnotz)

Mayer, R.E. (Ed.) (2005). Cambridge Handbook of Multimedia Learning. New York: Cambridge.
Chapter 4: An Integrated Model of Text and Picture Comprehension (Wolfgang Schnotz)

Introduction to the Integrated Comprehension of Text and Pictures Model (aka the Integrated Model):

There are two basic forms of representation:

  • Descriptions — texts, mathematical expressions. These are all symbols, which have no similarity with their referent. More powerful at expressing abstract knowledge. Can use the disjunctive “or”
  • Depictions — pictures, such as photos, drawings, bar graphs, maps. These all consist of icons, which are signs that bear some similarity or structural commonality with their referent. Informationally complete, therefore more useful to draw inferences, because information can be read off directly from the representation.

When one reads a text, three kinds of mental representations are constructed (follow left side of diagram):

  • Text-surface (descriptive): allows repeating of what is said so that the next representation can be made. (Full understanding of the text has not occurred)
  • Propositional (descriptive): includes the ideas expressed in the text on a conceptual level; independent from specific wording and syntax of the sentence.
  • Mental model (depictive)

When one processes a picture, a perceptual representation (depictive visual image) is created, from which a mental model is formed.

Verbal information is processed via symbol processing. Pictorial information is processed via structure mapping. While this model bears similarities to Paivio’s (1986) dual-coding concept, it also assumes that multiple representations are formed in both channels.

ICTP Model:

Working memory: Recent research suggests that there may be a visual working memory (sketchpad) for visual image and a spatial working memory for mental model construction.

LT memory: text and picture comprehension are based on external and internal (i.e., prior knowledge) sources of information. Prior knowledge can partially compensate for lack of external info, insufficient working memory capacity or incomplete propositional representations.

Meaningful learning results from selection of information, organization of information, activation of prior knowledge, and active coherence formation by integration of information from different sources. In general, the learner selects relevant information from an external source, organizes the information, activates related prior information, and then constructs a coherent propositional representation (if textual) and a coherent mental model.

ITPC model vs. Mayer’s CTML model:

  • Sensory modality and representational format — According to CTML, sensory modality and representational format are merged by the assumption of an auditory-verbal channel and visual-pictorial channel. According to the ITPC model, verbal info is not necessarily associated with the auditory modality and pictorial info is not necessarily associated with visual modality; for example sound images can be conveyed by the auditory modality.
    • Mental models — Even though both models assume that pictorial and verbal material are integrated in working memory, CTML assumes the construction of verbal and pictorial mental models that must later be integrated while ITPC assumes that only one mental model is constructed that integrates information from different sources from the beginning. The integrated model assumes that one joint mental model is constructed from illustrated text.
    • Coherence and contiguity (spatial and temporal) principles — Both models explain why coherence and contiguity btw pictures and words leads to better comprehension.

    Other implications of ITPC model:

    • Modality — According to ITPC, if a picture is combined with written text, all info must enter working memory through the visual channel, leading to selective information processing. Split attention (the use of one information channel for different sources of info) results in reduced amount of info in working memory. Also results in differences in amount of working memory capacity that can be utilized for further cognitive processing.
    • Picture-text sequencing effect — It is better to present a picture before a corresponding text because text never describes a subject matter with enough detail to fit just one single mental model. Therefore, the picture that follows the text will interfere with the mental model.
    • Reading ability and prior knowledge — poor readers profit more from illustrations in written texts than good readers.
    • General redundancy effect — A combination of text and pictures that has a positive effect on mental model construction in low prior knowledge learners may have a negative effect for experts because the eye will wander between the two sources (split attention effect) regardless of whether the additional information is necessary for comprehension. Related to expertise reversal effect.
    • Structure mapping effect — pictures are beneficial for learning only if task-appropriate forms of visualization are used. During picture comprehension, information processing is performed in the pictorial channel by structure mapping. Thus, the form of visualization affects the structure of the mental model.
    • Deep vs. superficial processing — if a picture is added to a text and if the same amount of mental effort is invested into learning, text information becomes less important due to the additional picture information. The text will be processed less deeply, resulting in lower recall of text information.
    • Memory for information sources — mental models can be held in LT memory for a longer time than prepositional representations. Thus, after a long retention interval, learners can no longer remember correctly whether specific information was presented in the text or in the picture. Text information will more likely be associated falsely with the picture than picture information is associated falsely with the text.
    • Cognitive economy — if the benefits from processing an additional information source are lower than the required costs, the learner will not engage in further cognitive processing. Use of multiple forms of representations is not always effective.
    • Control of processing principle — written text can be better than spoken text if paired with a static (vs animated) picture, if text is difficult, and if learning time is not limited.

    Limitations/Future Directions:

    • There may be multiple levels of propositional representations in the verbal channels and multiple mental models in the pictorial channel.
    • Mental models may be constructed directly from the text-surface representation. Similarly, a propositional model may be constructed directly from a perceptual representation of a picture without a mental model.
    • Learning from a complex nonlinear hyperspace requires the learner to also construct a mental model of the hyperspace to keep track of the sources of information.

    Leave a Reply

    Fill in your details below or click an icon to log in: Logo

    You are commenting using your account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s

    %d bloggers like this: