Text Image .org Matthew Peterson

Asst. Professor of Graphic Design

N.C. State University

Research Teaching Design   Contact ↦



Performative Image Function

Imagery, processed in a special component of human working memory, performs upon and with its reader. I am investigating the interpretational processes supported by imagery, especially as an aid to reasoning and learning across disciplines. To consider the processing of imagery in cognitive terms is to address interaction in the most meaningful sense. Following the work of educational psychologist Joel Levin and others, I am developing a taxonomy of performative image function, a classification system that describes images exclusively in terms of how they are processed by readers. It is inherently user-centric. Currently the performative image function taxonomy describes thirteen distinct functions.

How to Do Things with Pictures: The Experience of Challenging Imagery in Visual Media

Dialectic, 1(2), 2017

Pages 14–35

(Inset illustrations by Sander Weeks)

Journal (PDF access) ↦

Abstract: Some imagery poses an interpretational challenge that must be met in order to fully process it in the context of goal-oriented visual media. To evaluate such imagery it can only be considered in terms of a reader’s experience, an interpretational sequence that plays out in time. Pictures that are “challenging,” as discussed here, qualify as “integral experiences” according to Dewey: they have meaning, they are best understood as processes rather than elements, and they promote conceptual change in a reader. Four kinds of pictures that represent challenges to the reader are identified: problematic, linguistic and creative imagery, and visual metaphor. The experiential impacts of text–image relationships are discussed. Boundary examples of imagery are addressed to distinguish challenging imagery from other types and to establish the distinction of “performative” imagery, that which instigates and targets particular interpretational processes. The types of challenging pictures presented here qualify as performative in that they do more than repeat textual content or offer a moment of distraction.

The Production of Narrative through Static Imagery: Examples from a Peculiar Medieval Illustration

Visual Communication, in press; advance publication online (January 3, 2018)

(Inset illustration by James Wisdom)

Journal (PDF access) ↦

Abstract: Certain images perform upon and with their readers. Among the performative capabilities of imagery is narrative function, where the reader actively constructs a sense of time from an otherwise static surface. An extraordinary medieval illustration of Saint Margaret’s emergence from the belly of a dragon is used to demonstrate the range of narrative imagery. (Some consideration is given to the original conditions of this “Margaretene,” as it was designed for a medieval readership.) The degree to which the setting of narrative imagery is graphically constructed or natural impacts the work necessary for the reader to interpret it. The artist’s gestural repetition of figures can more directly cue the reader to the narrative, while the absence of repetition requires a more astute reader. Figures and spaces are considered in a strategical axis of narrative imagery. In increasing degrees of naturalism are graphic projective, partitioned, graphic repetitive, natural repetitive, intrafigural, and evidentiary strategies. Alternate versions of the Margaretene, rendered by artist James Wisdom, are included to illustrate the six narrative strategies.

How Imagery Models Interpretation: The Classification of Image Function

13th Annual Hawaii Conference on Arts and Humanities Proceedings, 2015

Complete Paper (PDF) ↦

Abstract: Much imagery is merely decorative or intended to be relevant and appropriate. But some imagery, on its own or situated within a meaningful layout, involves its reader more deeply, performing upon or with her. The image can model reader interpretational processes in the sense that the reader’s creative construction of knowledge is guided by its structures. Otherwise inert imagery, when activated by a reader, can become metaphorical, exploratory, constitutive, narrative, comparative and computational, among other functions. These classifications of performative image function represent a range of possibilities for the designer, advertiser and visual artist. Performative image function is a means to look at visual representations anew, by focusing on the seconds and milliseconds of the image’s effective “life.” The typology builds upon the work of cognitive psychologists, such as Joel Levin, who sought an honest assessment of the efficacy of textbook illustration for learning. Performative image function conceptualizes all imagery as designed for learning, as the reader seeks to make sense of the representations she encounters.

How Imagery Can Directly Model the Reader’s Construction of Narrative (Including an Extraordinary Medieval Illustration)

13th Annual Hawaii Conference on Arts and Humanities Proceedings, 2015

Complete Paper (PDF) ↦

Abstract: Certain images perform upon and with their readers. Among the performative capabilities of imagery is narrative function, where the reader actively constructs a sense of time from an otherwise inert surface. Narrative imagery can be classified according to five fundamental strategies, all of which have long been explored. At its most conventionalized, the passage of time can be implied through a framed sequential strategy, where distinct moments are sectioned off from one another graphically, across which figures and spaces are repeated. The reader performs closure across and between frames. In graphic repetition and natural repetition strategies, figures are repeated in space without frame-based separations. Here the reader must recognize repeated figures as representing a single figure’s change over time, and not a set of independently acting twins, triplets or quadruplets. With the intrafigural strategy, the reader is not faced with any repeated figures. Here the passage of time is implied through changes internal to a figure or interacting figures. A final evidentiary strategy provides the reader with clues to past events in a single captured moment in time. The active reader must make a series of deductions. An evidentiary narrative image is entirely natural, and is as such far removed from the conventions of a comic strip.

Performative Image Function (excerpt)

This is a position paper, not written for publication. It outlines the performative image function typology and utilizes exemplar images (in the process of being properly documented) to demonstrate the range of each function. Attached here are the introductory pages and the section on metaphorical imagery. A future publication will properly document all cited images.

Pages 1–3, 16–18, & 27–28


Excerpt PDF ↦

How to Do Things with Pictures (Invited Lecture)

Boise State University

April 5, 2017

Utilizing the Performative Nature of Print Media for Active Learning: Textbooks and Other Educational Materials as Experience

Learning Conference 08, The Fifteenth International Conference on Learning, University of Illinois at Chicago, Chicago, IL

June 4, 2008

Visual Metaphor

While we typically think of metaphor as a rhetorical trope, a form of fanciful speech, Lakoff and Johnson (1980) revealed it as an underlying capacity in the human conceptual system. We don’t just use metaphor; we think through it. Beginning in my broader image function research, I have been inquiring into visual metaphorical expressions, in most cases in the absence of text altogether. This has taken the form of theory development beginning in advertising, where there was existing formal analysis, and moving into applications in STEM education, where visual metaphor is used to help learners reason through abstract concepts.

How Different Visual Metaphors Influence Resource Allocation and Memory for Advertisements

Wise, Peterson, Xiong, & Wang

Interventions: Communication Research and Practice (ICA 2017 Conference) Proceedings, in press

Abstract: In previous research (Authors, in press), we explored how two structures of visual metaphor (juxtaposition and fusion) influence cognitive resource allocation and recall memory for advertisements. However, because an immediate recognition test yielded a ceiling effect, no insights emerged into how these visual structures differ with respect to recognition memory. Because secondary task reaction time (STRT) measures resources allocated to encoding, and recognition memory is also an indicator of the extent to which a stimulus was encoded, this leaves a gap in the literature. Here, we report the results of two studies meant to address this gap. The first study (N=43) is a direct replication using the same STRT paradigm and adding a third visual structure (replacement) for additional comparison. The second study (N=150) explores recognition memory for all three structures of visual metaphor using a delayed recognition task. Results generally support the previous finding that fusion images require more sustained resource allocation. Recognition memory is influenced not only by the type of visual metaphor but also by whether a target or foil is being recognized. These results are discussed in terms of implications for both visual cognition and advertising.

Memorable Metaphor: How Different Elements of Visual Rhetoric Affect Resource Allocation and Memory for Advertisements

Peterson, Wise, Ren, Wang, & Yao

Journal of Current Issues & Research in Advertising, 38(1), 2017

Pages 55–74

(Inset illustration by Scott Durand)

Journal ↦

Abstract: This paper explores how different uses of visual metaphor affect cognitive resource allocation and memory for pictorial advertisements. Metaphor is ubiquitous in advertising, and metaphors are often expressed entirely through imagery. Based on Phillips and McQuarrie’s seminal typology of visual metaphor, we selected advertisements that featured either juxtaposition or fusion structures. We then conducted a within-subjects experiment in which 36 participants viewed a series of juxtaposition and fusion ads. While viewing each ad, participants were prompted to respond to a series of visual probes. Their reaction times to these probes served as an indicator of cognitive resource allocation. Afterwards, we assessed participants’ recognition and free recall accuracy. Reaction time data showed that fusion ads required greater cognitive resource allocation and yielded more accurate recall than juxtaposition ads. These results are discussed in terms of both theories of visual metaphor and limited capacity models of cognition.

Understanding and Implementing Visual Metaphor

Peterson, Wise, Lindgren, Cox, & Mathayas

White paper funded by the Illinois Learning Sciences Design Initiative, University of Illinois, 2015

(Inset illustration by Sander Weeks)

Complete Paper (PDF) ↦

Proposal: Our goal is to begin development of a proof-of-concept for productive research in science education that focuses on the use of visual metaphor in instructional media. By “productive research,” we mean not only contributing to existing scholarship but also developing original media as both stimuli for the research laboratory and instructional content for the classroom. For instance, a comparative study might analyze differences between three versions of a tablet app on gravity, visualizing orbiting bodies, where the version deemed most successful would then be made available for others to implement. The science textbook currently functions as the source of curriculum in many science classrooms (Chiappetta & Koballa, 2002), despite deficiencies that will be difficult to address given the established North American textbook production model (DiGiuseppe, 2014). A growing resource for teachers consisting of tested curriculum materials would provide an already proven alternative to the monolithic science textbook. By developing methods for producing and studying complex and authentic instructional media, we seek to make the investment in empirical research double as direct assistance to science teachers. . .

Previous interstitial: visual metaphor variations of a Welch’s ad, by Lucas Albrecht.

Text–Image Integration

How does the integration of text and image in media impact the reader’s cognitive processing? To answer this question, I have defined three distinct levels of text–image integration, building on the earlier work of educational psychologists. My initial work in text–image integration has focused on instructional media for science. But the potential impact extends beyond science education. My holistic approach addresses psychological principles while remaining authentic to how designers work.

Schemes for Integrating Text and Image in the Science Textbook: Effects on Comprehension and Situational Interest

International Journal of Environmental and Science Education, 11(6), 2016

Pages 1365–1385

Complete Paper (PDF) ↦

Abstract: Background. The science textbook serves as curriculum in many science classrooms. Science education researchers have turned their attention to the use of images in textbooks, both because pages are heavily illustrated and because visual literacy is an important aptitude for science students. Some researchers have focused on the integration of text and image. Material and methods. Text–image integration is described here as composition schemes in increasing degrees of integration: prose primary (PP), prose subsumed (PS), and fully integrated (FI). These schemes enable research with authentic media, where the complexity of the illustrated page is not diminished for experimental purposes. . . Seventh grade subjects used media variants across 3 treatments with differing lesson content in a within-subjects design (N=158). Results. Comprehension tests revealed higher comprehension for FI over PP media variants in one of three sessions (P=0.0073). FI scores were highest for the other tests but not significantly so (P>0.05). A post-test situational interest measure revealed a significant preference pattern of FI>PS>PP. Conclusions. As textbooks follow a general PP strategy, expansion and confirmation of the results would suggest increasing the degree of text–image integration for middle school science textbooks. . .

The Integration of Text and Image and Its Impact on Reader Interest

Visible Language, 48(1), 2014

Pages 22–39

Journal ↦

Abstract: This paper addresses the design of instructional media both holistically and authentically by focusing on text–image relationships at the level of design strategy. The schema used is sensitive to working memory and cognitive load theory. Three text–image integration strategies are proposed and illustrated: prose primary (PP), with a central prose column and marginal imagery; prose subsumed (PS), with shorter prose segmented by imagery; and fully integrated (FI), where smaller textual chunks populate imagery. One hundred and thirty-seven (137) middle school students rated their interest in science textbook pages designed according to the outlined strategies. Interest measures are closely aligned with the situational interest construct in psychology. The subjects’ selections favored higher levels of text–image integration, such that FI was rated more interesting than PS, which was in turn more interesting than PP. Results were rated reliable and significant at a 95% confidence level. Comprehension and sense of task difficulty are briefly addressed.

The Integration of Text and Image, Its Cognitive Impacts for Learning with Media, and Science Instruction: A Ph.D. in Design Study

Connecting Dots: Research, Education + Practice, AIGA Design Educators Conference, 2014

Pages 124–137

Complete Paper (PDF) ↦

Abstract: This paper outlines a recent quasi-experimental study in fulfillment of the Ph.D. in Design. The study stands as an example of new research in the formation of a discipline of design (from a practice or field), according to a statistical model most consistent with psychological research. (The Ph.D. in Design can also extend into historical and argumentative models of research.) The study outlined (Peterson, 2011) addressed the design of instructional media both holistically and authentically by focusing on text–image relationships at the level of design strategy. Three text–image integration strategies are proposed and illustrated herein: prose primary (PP), with a central prose column and marginal imagery; prose subsumed (PS), with shorter prose segmented by imagery; and fully integrated (FI), where smaller textual chunks populate imagery. Over 150 middle school students participated in this study. Text–image integration strategies were tested for comprehension of abstract concepts, sense of difficulty, and interest level. Subjects completed comprehension tests using the supplied media. They rated their sense of difficulty and their interest in science textbook pages designed according to the outlined strategies. Their selections of interest strongly favored higher levels of text–image integration, such that FI was rated more interesting than PS, which was in turn more interesting than PP. Interest-level results were rated reliable and significant at a 95% confidence level. Comprehension results were less conclusive, with one treatment of FI proving more effective than PP, while two other treatments were found statistically insignificant. There was no suggestion that text–image integration strategy impacts sense of task difficulty.

Comprehension with Instructional Media for Middle School Science: Holistic Performative Design Strategy and Cognitive Load

Dissertation, North Carolina State University

Committee: Meredith Davis (Chair), Nilda Cosco, James Minogue, & John Nietfeld

Fall, 2011

Library entry w/ full document ↦

Abstract: This study identifies three distinct levels of text-image integration in page design in a linear relationship of lesser to greater integration: prose primary, prose subsumed, and fully integrated strategies. Science textbook pages were redesigned according to these holistic design strategies for 158 7th-grade students. There were three separate treatment tests, as well as a pre-test and post-test, and pilot tests with both undergraduate students and the subjects themselves. ¶ Subjects found the fully integrated strategy to produce the most visually interesting designs and the prose primary strategy to produce the least interesting, with prose subsumed definitively in between (according to 95% confidence intervals). ¶ The strategy employed significantly altered interest in science subject matter in one of three treatments (ANOVA, P=0.0446), where a Student’s t-test revealed that the prose subsumed strategy produced higher interest in subject matter than prose primary. ¶ The strategy employed significantly altered comprehension of abstract relationships in one of three treatments (ANOVA, P=0.0202), where a Student’s t-test revealed that the fully integrated strategy resulted in greater comprehension than prose primary. For the same treatment condition significant differences were found through ANOVA for factual-level knowledge (P=0.0289) but not conceptual-level knowledge (P=0.0586). For factual-level knowledge prose primary resulted in lesser comprehension than both prose subsumed and fully integrated. Comprehension is defined according to cognitive load theory. ¶ No strategy impact on perception of task difficulty was found. ¶ This study was approved by North Carolina State University’s Institutional Review Board and Wake County Public School System’s Research Review Committee.

Learning with Text and Image: The Relationship of Text–Image Integration to Interest and Comprehension

Toward an Illinois Learning Sciences Design Laboratory, University of Illinois

February 27, 2015

Comprehension with Instructional Media for Middle School Science: Holistic Performative Design Strategy and Cognitive Load (Doctoral Defense)

North Carolina State University

November, 2011

Inset: Mapping reader interactions in illustrated media (science textbook pages with overlays), by Britt Cobb.

Curriculum & Pedagogy

Gaze-Based Interaction Design (Speculative Interface)

Project in Master of Graphic Design Program at N.C. State University, College of Design

With Brad Tober, Deb Littlejohn, Rachael Paine, Dajana Nedic, Bree McMahon, Amber Ingram, Mac Hill, Grace Anne Foca, & Clément Bordas

(Inset by Rachael Paine)

Online Summary ↦

This project was undertaken by seven students in the Master of Graphic Design program at N.C. State University in spring 2017. They were introduced to an inchoate body of research on eye tracking as an input device (i.e., in lieu of mouse or trackpad) such that a system interface dynamically responds to the user’s gaze point. We were interested in how a gaze-responsive interface might change manipulation of elements, navigation through information environments, and our reading system itself. This studio-wide investigation is envisioned as an early stage in a greater research program, where designers’ abilities — to think laterally and open up a problem space — are leveraged to recognize potential for later empirical work, likely in other disciplines. Students prototyped individual studies and imagined more complex and situated systems, suggesting possible applications for a developing technology.

An Update on the Vertical Studio Implementation at the University of Illinois

With Brad Tober

UCDA 2014 Design Education Summit Proceedings

Complete Paper (PDF) ↦

Institutionalizing the Vertical Studio: Curriculum, Pedagogy, and the Logistics of Core Classes with Mixed-Level Students

With Brad Tober

Connecting Dots: Research, Education + Practice, AIGA Design Educators Conference, 2014

Pages 138–144

Complete Paper (PDF) ↦

Abstract: The vertical studio is a single-class combination of students at different levels (in this case: sophomores, juniors, and seniors) in a given program of study. The particular vertical studio model discussed here seeks to make a virtue of larger class sizes by encouraging a wider range of solutions from a more diverse group of students, and leveraging that range as instructional material. This demands very specific and tightly controlled changes to pedagogy. The administrative benefits of larger class sizes are perhaps obvious: it is a scheduling convenience rich in instructional units. Less obvious is how to make the vertical studio work as a core offering, such that it becomes a benefit for students. This paper outlines a test run and subsequent institutionalization of the vertical studio (though still temporary) in the Graphic Design program at the University of Illinois at Urbana-Champaign. The integration of the vertical studio into a core curriculum presents a better test of the concept than does an elective. Discussion covers principles, benefits, and problems of the vertical studio model believed to be relevant to instruction for all disciplines of design. Program-level logistics are addressed. The idea of the vertical studio, and especially this particular model, is interrogated. . .

One-on-One with iPads in a “Vertical” Graphic Design Studio Course

With Brad Tober

UCDA 2013 Design Education Summit Proceedings

Pages 201–206

Complete Paper (PDF) ↦

Abstract: The vertical model, as a strategy for structuring a studio-based course, has, until only recently, primarily seen interest and use in undergraduate architecture programs. Rather than segregate students by academic level — into sophomore, junior, and senior-level studio courses, for example — the vertical studio combines students at different levels. This means that students at different points in their academic careers have the opportunity to engage and interact, offering each other varied perspectives and bases of knowledge to work from. This acknowledges that students learn much more from seeing what others around them are doing, rather than from an instructor directly feeding them information. The vertical studio also offers a myriad of advantages related to curricular planning, such as effectively accommodating larger class sizes and integrating non-major students into what are otherwise typically core major courses. ¶ This paper presents a case study of a preliminary implementation of the vertical model into a graphic design curriculum. As a trial run for a more comprehensive future implementation, this study examines the creation of discrete vertical studio modules within an existing structure of sophomore and junior graphic design studio courses. . .

To Teach Image by Its Function: Structuring Image Making for Graphic Design Students According to Cognitive Outcomes

UCDA 2014 Design Education Summit Proceedings

Complete Paper (PDF) ↦

Abstract: Graphic design students often have access to image-oriented studio electives within schools of art and design, taught by experts in photography, illustration, and painting. What is distinct about a core image-making course for graphic designers? This question addresses the fundamentals of graphic design, possibly in opposition to what is implicitly claimed as such in many discipline-spanning foundation courses. This paper outlines two iterations of the author’s image-making course in two graphic design programs, both at public research universities, with special emphasis on the method employed in its development. The conception of image making embodied in the course is rooted in the author’s research on the cognitive function of imagery. Considering imagery in terms of the cognitive processing of its readers is an inherently user-centered approach, which distinguishes the coursework as design pedagogy. Critique methods reinforce the emphasis on goal-oriented interpretation and are critical to the integrity of the course. Assignments are classified here according to designations of exercise, project, and investigation, largely based on contextuality. Exercises isolate some fundamental aspect of graphic design without overwhelming beginning students with undue complexity, though over-simplification is a danger. . .

Inset: Educational Objectives Schematics

Visual summaries for reference, available now for free!

Covers Bloom’s Taxonomy Revised (Anderson & Krathwohl, et al., 2001) and Marzano’s New Taxonomy (Marzano & Kendall, 2007).

Two Schematic Posters (PDF) ↦

A Pedagogical Flip: Teaching Typographic Complexity to Introductory Students

With K.T. Meaney

TypeCon2012: Type & Design Education Forum, Society of Typographic Aficionados (SOTA), Chicago, IL

August 2, 2012

Design Work

Ninth Letter, Arts & Literary Journal, 4 issues

Creative & Art Director

Ninth Letter is jointly produced by the Creative Writing and Graphic Design programs at Illinois. The art director produces each issue in a semester’s class. Students serve as the designers, photographers, and illustrators. The design team proposes and oversees art and design features.

(Inset illustration by Kamila Glowacki)

NinthLetter.com ↦

Vol. 9, No. 1

Designers: Lucas Albrecht, Hannah Burtness, Garrett Campagna, Katy Dondzila, Alex Dye, Olivia La Faire, Taekyeom Lee, Tanner Mei, Annaka Olsen, Diane Park, Will Ryan, Cassie Tu

Photographers: Maggie Day, Brent Hofacker, Marco Novielli, Jaci Wandell

Gag comics by Billy Fore

Vol. 9, No. 2

Designers: Grace Hamann, Caitlin Howe, Scott Jackson, Maria Ludeke, Serena Moy, Elaine Palutsis, Will Ryan, Tyler Schmidt, Heather Stickney

Photographers: Harrison Hakes, Kendall McCaugherty, Alejandra Rodriguez

Vol. 12, No. 2

As Creative Director

Art Director: Joe Carpenter

Designers: Katie Geary, Robert Marohn, Benjamin Minard, Vin Park, Courtney Podgorski, Eric Pryor, Alyssa Sparacino, Gracie Sullivan

Illustrators: Jina Seo, Mew Tachibana, Sander Weeks, James Wisdom

Vol. 10, No. 1

Designers: Brandon Barker, Lauren Blackburn, Andrew Fishel, Scott Jackson, Evan Jarzynksi, Elaine Palutsis, Daryl Quitalig, Nathalie Rock, Heather Stickney, Michael Zhang

Photographers: Brittany Keating, Jack Kendall, John Menchaca, Megan Roche, Randi Stella

Illustrators: Kamila Glowacki, Colin Mosely