Abstract: This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development. This project will investigate how students’ understanding of scale and number sense (numeracy) can be improved in virtual reality (VR). Research shows that students of all ages hold inaccurate ideas about the size of scientifically relevant entities. For instance, students have difficulty distinguishing the magnitude of difference between the sizes of molecules and cells, or between Earth’s moon and the sun. Because many STEM fields — from virology to astronomy — involve work conducted at extremes of scale, students must develop accurate conceptions of scales that exist well beyond their everyday experience of the world. An inaccurate understanding of scale can obscure the applicability of numeric representation to the real world and become a barrier for entry to STEM. This project will create Scale Worlds, an immersive virtual environment, where students can see various scientific entities in relation to their own bodies and conduct realistic size comparisons that cannot be replicated in everyday experience. . .

Scale Worlds is conceptually a single extensive space: its entities are sized authentically in relation to one another. However, only four entities are visible at once (further larger and smaller entities cannot be displayed due to software limitations). When the user chooses to grow or shrink by a power of ten (A–F), an animation correspondingly shrinks or grows (respectively) the entire environment. The apportion of scientific entities and scale armatures is designed to reinforce the sense of the user’s growing and shrinking, and to provide means by which users can estimate the size of the entities (and of themselves).

This internal grant funded the original Scale Worlds prototype. Grace Wonaphotimuke served as developer according to specifications I provided with Karen Chen and Cesar Delgado. The prototype was designed for and implemented in Karen Chen’s Virtual and Augmented Reality Lab (Industrial & Systems Engineering, N.C. State University).

Abstract: Pedagogical content knowledge of science teachers includes knowledge of representational forms, such as visual displays. Past research has provided a general means to evaluate pictures, but is not fine-grained on features internal to pictures. Furthermore, many existing picture function typologies consider pictures as subservient to text, though pictures have their own explanatory power. We here adapt a general ‘performative’ image function typology into a science-specific taxonomy of cognitive image functions with fine-grained distinctions of design strategies related to interpretative processes. The authors, including science educators and graphic designers, refined and expanded functions of imagery using visual displays in science textbooks following a constant comparison method. The resulting performative image function taxonomy consists of a structural framework that distinguishes conceptual elements internal to pictures (i.e. concepts, entities, components, attributes, adjuncts, configurations); a process model that identifies three interpretative phases necessary to understand visual displays (i.e. approach, activity, outcome); and 17 functions across the interpretative phases. We demonstrate these functions through analysis of and modifications to visual displays from science textbooks. Many of the modifications presented here can be repeated with rudimentary drawing over existing illustrations (e.g. arrows, dividing lines, labels), and they are thus accessible to science teachers and students.

This grant is supporting an eye tracking experiment conducted in Kevin Wise’s media lab at the University of Illinois. Research question: How does user control of non-depictive visual adjuncts to imagery in multimedia for science learning affect visual cognition, cognitive load, and meaning comprehension? Our proposed research explores the relative effects of adjuncts and textual labels as interactive overlays in multimedia for science learning. To what extent do virtual adjuncts and configurations enhance visual cognition, resource availability, and meaning comprehension? To what extent do students utilize these features while consuming multimedia for science learning, and how do their effects unfold dynamically over time? The answers to these questions have the potential to broadly impact both the understanding and design of educational technology across a wide range of learning domains (e.g. biology, ecology, physics) and, to a lesser extent, individual differences (e.g., age, expertise, socioeconomic status).

Abstract: Metaphor is routinely expressed through pictures in contemporary advertising. Earlier work on visual rhetoric in advertising sought direct analogs for the tropes and schemes specific to verbal rhetoric. More recently, theory has developed out of characteristics particular to pictures, culminating in a typology of visual rhetoric by Phillips and McQuarrie (2004). Subsequent work by Šorm and Steen (2013) considers processing stages in metaphorical interpretation. Phillips and McQuarrie’s typology is adapted and expanded here in order to better describe the variation found in ads that utilize visual metaphor and its allies, thus identifying a range of examples that confound the original. A new processing model that directly addresses visual features described in the expanded typology is outlined, which builds on ideas from Šorm and Steen but represents a distinct framework. Additional variables are identified that advertising researchers should either investigate or control in experimental studies of visual metaphor.

Abstract: This article explores how different uses of visual metaphor affect cognitive resource allocation and memory for pictorial advertisements. Metaphor is ubiquitous in advertising, and metaphors are often expressed entirely through imagery. Based on Phillips and McQuarrie’s seminal typology of visual metaphor, we selected advertisements that featured either juxtaposition or fusion structures. We then conducted a within-subjects experiment in which thirty-six participants viewed a series of juxtaposition and fusion ads. While viewing each ad, participants were prompted to respond to a series of visual probes. Their reaction times to these probes served as an indicator of cognitive resource allocation. Afterward, we assessed participants’ recognition and free recall accuracy. Reaction time data showed that fusion ads required greater cognitive resource allocation and yielded more accurate recall than juxtaposition ads. These results are discussed in terms of both theories of visual metaphor and limited capacity models of cognition.

Figure 1. Pictorial exemplification of structural features. (A) Physical evidence: birds. Illustration by Micaelah Scott, after the original in McDougal Littell Science, North Carolina Edition, Grade 8, McDougal Littell, p. B48. (B) Seasons. Illustration by Micaelah Scott, after the original in Science Voyages, North Carolina Edition, Grade 8, Glencoe, 2000, p. 122. (C) Visual inventory of select entities, components, attributes, adjuncts, and configurations from (A) and (B) — concepts, by definition, cannot be shown.

Abstract: HCI researchers have repurposed diagnostic eye tracking technology as a mode of user input. Existing applications are numerous, but primarily address severe motor disability, with a recent increase in gaming enhancement. As noted by cognitive psychologist Nadiya Slobodenyuk, gaze-based HCI represents a fundamental change to the human–computer relationship if adopted for general interaction and information design purposes. A gaze-responsive system can make inferences on a user’s mental state and respond rapidly without explicit user commands. The implications of such a system are significant, and are difficult to imagine and anticipate. We introduce the tech receptivity interval (TRI) as a framework to guide speculative design investigations that imagine potential applications of nascent technology. TRI distinguishes infancy and maturity conditions of receptivity, emphasizing the need for users to adapt to technologies before technological affordances can be fully realized. We provide case reports on gaze-based interaction, using TRI and conducted in an academic design studio. The case reports suggest applications not yet addressed in the literature. The case reports also suggest gaze-responsive changes to information structures in the form of temporal hierarchy and temporal text, which break from the long tradition of language representation in static lines and paragraphs.

Abstract: Certain images perform upon and with their readers. Among the performative capabilities of imagery is narrative function, where the reader actively constructs a sense of time from an otherwise static surface. An extraordinary medieval illustration of Saint Margaret’s emergence from the belly of a dragon is used to demonstrate the range of narrative imagery. (Some consideration is given to the original conditions of this “Margaretene,” as it was designed for a medieval readership.) The degree to which the setting of narrative imagery is graphically constructed or natural impacts the work necessary for the reader to interpret it. The artist’s gestural repetition of figures can more directly cue the reader to the narrative, while the absence of repetition requires a more astute reader. Figures and spaces are considered in a strategical axis of narrative imagery. In increasing degrees of naturalism are graphic projective, partitioned, graphic repetitive, natural repetitive, intrafigural, and evidentiary strategies. Alternate versions of the Margaretene, rendered by artist James Wisdom, are included to illustrate the six narrative strategies.

Abstract: Design units can better integrate themselves within research universities by producing scholarship consistent with institutional expectations, in the form of publications, grants, and patents. But significant challenges face the design faculty who must actualize this integration. I summarize my own strategic research program over three phases: an initial appointment as assistant professor; a second assistant professor appointment up to the submission of my tenure dossier; and the year following that submission. This story of phases in an early academic career illustrates challenges that are particular to junior faculty in design units, and it reveals the work required to secure publications and grants. In established research disciplines, junior faculty continue the work they began when earning their research degrees, much of which occurred in productive labs. But junior faculty in design must often initiate a research program from scratch, while already on the tenure clock. Furthermore, because research in design is heterogeneous, junior design faculty must define their own particular model of research, which also takes time, and advocate for their adopted model. I provide recommendations for design administrators and junior design faculty as a way to help elevate scholarship in design. My final recommendation for junior design faculty is to adopt the mindset of the hedonistic scholar, who does not depend upon success for personal fulfillment, but finds joy in the craft of scholarship. This is important because, as demonstrated by my own research program, significant scholarly outcomes are likely the product of years of dedication with numerous failures along the way.

Making the Master’s Degree in Design More Essential within the Social Sciences

Keynote Presentation

Matthew Peterson

A Symposium on Design Graduate Education, University of Washington, April 30, 2021

Recording

An Update on the Vertical Studio Implementation at the University of Illinois

Matthew Peterson & Brad Tober

UCDA 2014 Design Education Summit Proceedings

Complete paper (PDF)

One-on-One with iPads in a “Vertical” Graphic Design Studio Course

Brad Tober & Matthew Peterson

UCDA 2013 Design Education Summit Proceedings

Pages 201–206

Complete paper (PDF)