Article / Multimodal Communication
Roy K. Humble / 17 July 2020
This is Part 2 of a four-part series on Multimodal Communication that's intended as a supplement to The Humble Argument. Part 1 provides a general introduction to multimodal communication and its terminology. This articles introduces the central concepts and terminology of multimodal communication. Part 3 provides an overview of the general multimodal expectations that college professors have for your college paper. Part 4 wraps things up with more specific guidelines for different academic disciplines and some help with using visual elements within college papers.
To learn the basics of multimodal communication and apply them in college, we have to start with the concepts involved and that means learning the technical terms that scholars use to discuss them.
We'll tackle this in three waves of vocabulary building:
Rhetoric is communication that tries to influence the thinking of others. The rhetorical situation is when and where rhetoric happens.
Last night, for example, you were in a rhetorical situation when a Nike ad on TV tried to convince you that you should buy a new pair of running shoes even though you don't actually run, not by choice. This morning, your roommate created a rhetorical situation when she told you that you should put on a coat because it's cold outside and starting to rain. She was trying to change your mind about what you should wear, which was basketball shorts and a tee shirt. And as you sat down with your homework this afternoon, you found yourself in yet another rhetorical situation because it required you to write an essay that presents your opinion about a poem, an essay that your English professor will read, consider, and grade.
In The Humble Argument, the rhetorical situation is presented practically as the situation surrounding a college essay. Here, we look at the rhetorical situation is more theoretically so that it can provide a more flexible framework for understanding all kinds of multimodal communication.
The examples above of running shoe ad, roommate's warning, and poetry paper all have three main ingredients, and these are the core ingredients for any rhetorical situation:
These three main ingredients in the rhetorical situation imply two other elements that are also worth considering:
Here's how these five ingredients form what unnamed experts claim is the second most painful rhetorical situation of all time, the job interview. The first most painful rhetorical situation is buying a used car.
These five ingredients are all present as Greek works in the explanation of communication from the ancient Greek philosopher Aristotle more than two thousand years ago. Western scholars have tinkering with these concepts ever since and have translated his Greek terms into more useful modern terms, but to be honest, nobody's really improved much upon Aristotle. What he described two thousand years ago is still the rhetorical situation because these are still the ingredients that are present when people try to change each other's minds.
The only thing that's really changed since Aristotle's time is that technological advancements have given us new ways to share our ideas with others — new modes of communication. In Aristotle's day, the spoken word was the main mode of communication and real time speeches or discussions were the main rhetorical situation. After the printing press became an important part of the world, the written word was king and allowed for asynchronous rhetorical situations between writers and readers.
Now that personal computers are so prevalent, there isn't really a king anymore. Personal computers allow us to combine images and words and sounds and even videos into a single digital text. The Internet allows us to weave resources from around the world into that same single text. At the same time, though, we continue to write papers and go to job interviews and simply argue with each other over coffee.
The rhetorical situation for multimodal communication in the writing classroom works like this:
Whatever the particulars of this rhetorical situation may have been, the result is that the senders have changed the thinking of the receivers over time. Multimodal is now the Big Thing in college writing. This is why college writers like yourself are now being asked to make videos and websites and posters instead of or in addition to old-fashioned academic essays. This is why entire departments of sentence-loving English teachers are being asked to teach low-grade versions of design, podcasting, videography, and public speaking in their writing classes.
This happens from time to time in English departments. When I was a young writing teacher, formal logic and Greek terminology was the next Big Thing among writing teachers. Slightly before then, young writing teachers took the conflict-resolution ideas of a marriage counselor, Carl Rogers, and turned them into the next Big Thing. Slightly after that, discourse communities were the next Big Thing. That was followed by the great hypertext scare, when the introduction of the World Wide Web made English visionaries predict that soon every college papers would be riddled with hypertext links, like at Wikipedia.
Eventually, multimodal communication will be claimed by the Speech and Communication department where it belongs, and young writing teachers will discover a new Big Thing to rally around and force upon their elders. Until then, however, it's here for you to learn, and learn it you will. As you wade into this, there are three things for you to keep in mind:
According to multimodal communication theorists, there are five main modes or modalities or ways of communication — linguistic, visual, aural, special, and gestural. We'll take a briefly look at each of these modes and how they might show up in a multimodal text.
This is where you use your words — written or spoken. Using your words includes several important activities that can help to make your communication more effective, such as these:
For college papers, this is the most important mode for you to use. The more you understand it, the more effective your papers will be in presenting your message and convincing your audience that your ideas make sense. For other types of communication — speeches, posters, videos — it's still important but less so.
The visual mode, as you might expect, refers to what you and your audiences sees. This means images, of course, but in a college paper, it can also include a lot of other features that help you to format and present your words so that they are easier to for readers to understand:
The visual mode also includes different ways to help readers focus on the key elements of illustrations, photographs, and images-based types of communication like videos or websites.
This is what your audience hears. This is the primary mode of communication with public speaking, podcasts, radio, and other speech-based types of communication, but it's also an important ingredient in videos and film. This model extends the spoken words of the linguistic mode and the images of the visual modal with many other sound elements:
This mode presents information through the physical arrangement of images and information.
This is especially important when the communication is three-dimensional. How you arrange the chairs for your presentation to a class, for example, affects how that audience will respond. If you arrange them in a circle, you imply that you are connected to your audience and invite them to participate in the communication. If you arrange the chairs in rows that you stand in front of, you imply that you are the expert and that your audience should sit back, shut up, and receive your ideas.
The spatial mode also matters in two-dimensional forms of communication. How you divide website information into pages and tabs shows your readers how your information or ideas are structured and helps them to navigate to easily within that web-based presentation. If you put two illustrations next to each other on a page in your essay, the proximity of those images to each other suggests to readers that they are related. If you make one large and the other images small, it implies that one is more important than the other.
This is the way that people communication information nonverbally through facial expressions, body language, hand gestures, and touching.
If you cross your arms and say "I love you," for example, then your audience is getting a mixed message. The linguistic mode tells them that you love them, which is nice. However, the gestural mode tells them that you're also not 100% happy with them. They could do a better job helping out around the apartment. If nothing else, they could put on a coat before going outside in the rain.
You don't have any gestural modes with your college papers — except when you turn it in, perhaps, by slamming it down your professor's desk or something. However, this is really important to consider as you interact with others or as you watch the interaction of others.
In any rhetorical situation, the point of contact between the sender and receiver is a message, an idea. However, that idea is not transmitted directly from one brain to another. The idea has to first be translated into a tangible form — a television ad, a spoken suggestion, or a written paper. The vocabulary of multimedia communication offers a few important terms that relate to the tangible form of a message.
Traditionally, the word "text" has describes something like a book or newspaper article or poem that is made with written words. With multimodal communication, however, written words are just one of many options, so "text" here refers to any kind of communication created with any or several or all of the following five modes. A speech is a text. A poster is a text. A video or podcast or graphic novel or Internet meme is a text. In the three examples from early in this section, the television ad, spoken suggestion, and poetry essay are all texts, too.
This one is tricky, so pay attention. The medium refers to whatever technology or mechanism is used to transmit the text from the sender to the receiver. It's the go-between.
So with a television ad text, which is basically a short film, the idea presented by that text is that you, friend, should buy some running shoes. The medium of television is the technology that transmits the text from the sender to the receiver.
With a message to put on a coat, your roommate could have texted you the text of that message on her phone. In that case, the text would have been a text, which is cool, and the medium would have been smartphone technology. She could have written you a note and left it on the back door. In that case, the text would be the note itself, and the medium would have been paper with sticky stuff on the back of it. However, instead of those impersonal media, your thoughtful roommate used the medium of direct, face-to-face conversation to transmit her message. She translated her idea into words that she sent to your ears with her voice. That was kind of her.
With the essay you are trying to write about that poem, the message will be your opinion of the poem. The text will be the essay you write that brilliantly explains and defends your opinion. The medium you will use to transmit your text will be ink on paper. You will print your words onto clean, white paper just as the assignment instructions require.
This term refers to the strengths or potential uses of a given medium — the sort of messages that a medium allows you to transmit more effectively. Television, for example, is great for combining visual information, sound, and words into a cohesive message that is received and digested immediately. Face-to-face speaking is great for direct, face-to-face communication and allows you to add support to the message with gestures, such as your roommate's hands on her hips and frown. Paper is a great medium for messages that are received when the receiver has time to finally sit down and read the words on the page, probably after the kids are finally in bed.
In the broader world outside the multimodal writing classroom, the term "genre" (ZHAHN ruh) usually refers to different types of creative works — written texts, music, or movies, for example. Fiction is a genre for writing, and nonfiction and poetry are genres. Blues is a genre of music. Horror and romantic comedies are two genres of movies.
What distinguishes one genre of writing, music, or movie from another is that each genre has different shared characteristics. As receivers of these texts, we expect that any text belong to a particular genre will have the features of that genre. You expect a fictional book to follow the convention of telling you made-up stories about made-up characters. You expect poetry to rhyme and be about love — or maybe you have a broader expectation after you take a poetry class, or maybe not. You expect a romantic comedy to end happily — horror films, not so much.
These expectations are known as conventions. Conventions are the rules that the texts of any given genre must follow.
With multimodal communication in college, "genre" can apply to any kind of multimodal of text, written or otherwise — class presentation, PowerPoint slides, research paper, or website, for example. This includes conventions for the message itself and the different modes of communication with the text.
Outside the college classroom, you'll find even more genres for multimodal communication — phone calls, texts, email, conversations with old people, and more. You might not have considered the conventions for these genres, but you still know what they are. You know how to follow them, and you can tell when someone violates them.
Every social media platform is its own genre, too. The conventions for Facebook are different from the conventions for Twitter or Instagram or whatever TikTok is. You'll find that the most effective posts on those platforms are the ones that follow those conventions in creative ways, not the ones that break them.
This discussion continues in Part 3 of this series, "The Multimodal College Paper."