The Terminology of Multimodal Communication

This is Part 2 of a four-part series on Multimodal Communication that's intended as a supplement to The Humble Argument. Part 1 provides a general introduction to multimodal communication and its terminology. This articles introduces the central concepts and terminology of multimodal communication. Part 3 provides an overview of the general multimodal expectations that college professors have for your college paper. Part 4 wraps things up with more specific guidelines for different academic disciplines and some help with using visual elements within college papers.

To learn the basics of multimodal communication and apply them in college, we have to start with the concepts involved and that means learning the technical terms that scholars use to discuss them.

We'll tackle this in three waves of vocabulary building:

1. The Rhetorical Situation:The vocabulary in this section defines the general elements of communication that are present whenever people share ideas that they want others to accept.
2. The Modes of Multimodal Communication:This section focuses on the five main modes of communicating ideas or information to others. It ends with related terms that are useful for understanding what makes multimodal communication more or less effective as you move from one rhetorical situation to another.
3. Other Important Terms: This final section offers a few other important terms that don't easily fit into the first two sections. They had to go somewhere.

1. The Rhetorical Situation

Rhetoric is communication that tries to influence the thinking of others. The rhetorical situation is when and where rhetoric happens.

Last night, for example, you were in a rhetorical situation when a Nike ad on TV tried to convince you that you should buy a new pair of running shoes even though you don't actually run, not by choice. This morning, your roommate created a rhetorical situation when she told you that you should put on a coat because it's cold outside and starting to rain. She was trying to change your mind about what you should wear, which was basketball shorts and a tee shirt. And as you sat down with your homework this afternoon, you found yourself in yet another rhetorical situation because it required you to write an essay that presents your opinion about a poem, an essay that your English professor will read, consider, and grade.

In The Humble Argument, the rhetorical situation is presented practically as the situation surrounding a college essay. Here, we look at the rhetorical situation is more theoretically so that it can provide a more flexible framework for understanding all kinds of multimodal communication.

The Main Ingredients

The examples above of running shoe ad, roommate's warning, and poetry paper all have three main ingredients, and these are the core ingredients for any rhetorical situation:

• Sender: This is who sends out the idea. This is the author of written works, the speaker of speeches, and the designer of illustrations. In the examples above, these are the senders:
  1. Nike
  2. Your concerned roommate
  3. You the college writer
• Message: This is the main idea that the sender wants the receiver to consider and accept:
  1. Buy more shoes.
  2. Put on your coat.
  3. This poem makes no sense at all. (your paper's thesis)
• Receiver: This is who the sender is trying to influence with the message. "Audience" is a good alternative term. In these examples, these the receivers:
  1. You the consumer who was watching television when you should have been working on your English paper
  2. You the under-dressed roommate who was about to go outside without a coat, even though it's maybe forty-two degrees outside and starting to rain.
  3. The English professor who dearly loves the poem you are writing about and will probably judge your paper according to how well you love the poem, too.

These three main ingredients in the rhetorical situation imply two other elements that are also worth considering:

• Purpose: This is the reason why the sender presents the message to the receiver in the first place. Another way of thinking about purpose is to consider what problem needs to be solved.
  1. Nike wants you to buy more shoes because it is a business that wants to generate income for its shareholders, and if you buy more shoes, they will then have more of your money.
  2. Your roommate wants you wear your coat because she cares about you and wants you to stay healthy and also because she wants to avoid the unhealthy version of you, which is pretty whiny. Let's be honest about that.
  3. You write the essay because that will help you to earn a passing grade in this poetry class that is forcing you to read way more poems than you expected.
• Context: This is the broader historical, social, and cultural situation that surrounds the more immediate moment of communication and influences it. In all three examples, the author and audience are likely to share the same time frame, place, and culture, so the author can rely on elements of the context in explaining the message.
  1. Nike can use the cultural value of physical fitness to convince you to buy new running shoes.
  2. Your roommate can point at the rain that's starting to fall outside to convince you to put on your coat.
  3. You can refer to Twitter to explain to your professor that each line of this stupid poem sounds like a political tweet.

Rhetorical Situation Analysis

Here's how these five ingredients form what unnamed experts claim is the second most painful rhetorical situation of all time, the job interview. The first most painful rhetorical situation is buying a used car.

• Sender: The woman facing the four unfriendly faces is the interviewee. She is the one who wants to affect the thinking of the interviewers with her amazing answers to their fairly dumb questions.
• Receivers: The four unfriendly interviewers are listening to her amazing answers and thinking about whether the sender is the person they are looking for.
• Message: The sender wants to present the message that she is the right person for this job. That's the idea she wants to interviewers to accept.
• Purpose: The sender is sending the message because she really, really, really wants this or any job so that she can quit her current job of managing two-year-olds at a day care center. She was not meant for that kind of craziness. Also, she will need to earn more money in order to fix the back bumper of her car.
• Setting: There are many layers to the setting for this rhetorical situation. The interview is happening in a conference room at this high tech advertising agency, so the sender feels a little out of place. It's early afternoon, which means that the interviewers are acting a little sleepy from lunch. Those problems affect the situation. They force the interviewee to present herself as a little perkier than normal to keep them engaged. However, this also happens in a time of almost full employment where she lives, so they are probably having a hard time finding a qualified person. That allows the interviewee to have a little more confidence.

These five ingredients are all present as Greek works in the explanation of communication from the ancient Greek philosopher Aristotle more than two thousand years ago. Western scholars have tinkering with these concepts ever since and have translated his Greek terms into more useful modern terms, but to be honest, nobody's really improved much upon Aristotle. What he described two thousand years ago is still the rhetorical situation because these are still the ingredients that are present when people try to change each other's minds.

The only thing that's really changed since Aristotle's time is that technological advancements have given us new ways to share our ideas with others — new modes of communication. In Aristotle's day, the spoken word was the main mode of communication and real time speeches or discussions were the main rhetorical situation. After the printing press became an important part of the world, the written word was king and allowed for asynchronous rhetorical situations between writers and readers.

Now that personal computers are so prevalent, there isn't really a king anymore. Personal computers allow us to combine images and words and sounds and even videos into a single digital text. The Internet allows us to weave resources from around the world into that same single text. At the same time, though, we continue to write papers and go to job interviews and simply argue with each other over coffee.

The Rhetorical Situation and Multimodal Communication

The rhetorical situation for multimodal communication in the writing classroom works like this:

• Sender: Up-and-coming writing teachers around the United States and the world collectively presented their message.
• Message: Multimodal communication should be a central ingredient in college writing classes because it helps prepare writing students for real life.
• Receivers: English departments listened to these up-and-coming writing teachers and for one reason or another agreed.
• Setting: About 20 years ago, young writing teachers noticed that technological changes had opened the door for a wide variety of multimodal communication in a wide variety of digital settings. People were emailing words and letters together. Social media sites combined text, images, sounds, and moving images. Smart phones allowed people to create videos, add headings to those videos, and share them online. It was wild how much multimodal communication was going on out there, and it seemed to young writing teachers that writing courses should help prepare students to be good multimodal communicators. This led to the development of theories and guidelines about how to communicate multimodally and how to teach multimodal communication.
• Purpose: One possible purpose for sending this message might be to serve students better by giving them better skills for multimodal communication — both as senders and receivers. Another possible purpose for championing a new Big Thing is that it allows young writing professors to build careers in a high competitive corner of higher education. Some of the earliest champions said they wanted this to give people the power to change the world. Some of my more cynical colleagues say this is just another conspiracy from commercial textbook companies to force faculty to buy new books.

Whatever the particulars of this rhetorical situation may have been, the result is that the senders have changed the thinking of the receivers over time. Multimodal is now the Big Thing in college writing. This is why college writers like yourself are now being asked to make videos and websites and posters instead of or in addition to old-fashioned academic essays. This is why entire departments of sentence-loving English teachers are being asked to teach low-grade versions of design, podcasting, videography, and public speaking in their writing classes.

This happens from time to time in English departments. When I was a young writing teacher, formal logic and Greek terminology was the next Big Thing among writing teachers. Slightly before then, young writing teachers took the conflict-resolution ideas of a marriage counselor, Carl Rogers, and turned them into the next Big Thing. Slightly after that, discourse communities were the next Big Thing. That was followed by the great hypertext scare, when the introduction of the World Wide Web made English visionaries predict that soon every college papers would be riddled with hypertext links, like at Wikipedia.

Eventually, multimodal communication will be claimed by the Speech and Communication department where it belongs, and young writing teachers will discover a new Big Thing to rally around and force upon their elders. Until then, however, it's here for you to learn, and learn it you will. As you wade into this, there are three things for you to keep in mind:

1. Even though most writing teachers aren't particularly well qualified to teach multimodal communication, yours probably is. And even if yours isn't, you should still pay attention to them and to this guide because understanding the basics will help you be a more thoughtful receiver of multimodal messages and a better multimodal communicator.
2. No matter what the Big Thing is at any given moment in college writing, the rhetorical situation is going to be a part of it. It isn't going anywhere, and it's always going to useful to you as a sender of messages:
   • As a sender of messages, it helps you consider your purpose in sending a particular message. Is it a noble purpose, one that's worth your time and creative energy and careful thinking?
   • If so, you can then consider the message. Is it a valid message? Is it reasonable? Can you explain it in sufficient detail and defend it with adequate evidence?
   • If so, then consider your audience, the receivers. What is the best way to communicate that message to them? (Here's where the different modes of communication come into play most prominently.) What does your audience expect from you in this rhetorical situation? What do you know about their preferences and values?
   • Do you share any common ground within the broader setting of this rhetorical situation? The more carefully you consider this and the other elements of the rhetorical situation, the more effective your communication will become.
3. As a receiver of messages, your consideration of the rhetorical situation is a kind of self-defense against bad ideas trying to get into your brain.
   • Who is the sender of the message? What's the purpose of sending it to you? What's in for them if you let it influence my thinking? Those three questions will karate chop 99% of all advertising messages and expose them as manipulative attacks on your otherwise good judgment.
   • If you find the sender somewhat credible, take a look at the message itself. Does the main idea make sense? Has the sender provided sufficient, credible evidence for it? Are the reasons logical? What different modes of communication are working together here? Is there anything shady going on? Is a picture of a vegan bodybuilder telling you that you'll get a killer body by not eating meat?
   • Is the message relying on ideas or sentiments from the current setting to make its point? If so, is that appropriate or manipulative?. It usually doesn't take much questioning to expose shoddy thinking, and that helps you to keep bad ideas from becoming your own.

2. The Modes of Multimodal Communication

According to multimodal communication theorists, there are five main modes or modalities or ways of communication — linguistic, visual, aural, special, and gestural. We'll take a briefly look at each of these modes and how they might show up in a multimodal text.

Linguistic Mode

This is where you use your words — written or spoken. Using your words includes several important activities that can help to make your communication more effective, such as these:

• Word choice: Choosing words that explain your idea precisely and that are understandable by the receiver.
• Tone: Choosing words that present an appropriate attitude toward a subject — positive, negative, or neutral.
• Organization: Grouping your words into understandable sentences and carefully arranged paragraphs so that the supporting parts of your idea make sense by themselves and so the whole message makes sense.

For college papers, this is the most important mode for you to use. The more you understand it, the more effective your papers will be in presenting your message and convincing your audience that your ideas make sense. For other types of communication — speeches, posters, videos — it's still important but less so.

Visual Mode

The visual mode, as you might expect, refers to what you and your audiences sees. This means images, of course, but in a college paper, it can also include a lot of other features that help you to format and present your words so that they are easier to for readers to understand:

• Color: Draw attention to important terms with a contrasting color. Connected related terms by using the same color for both.
• Tables: Use column formatting to organize similar bits of information into side-by-side groups.
• Fonts: Use a book font to make paragraphs easier on the eyes and thus more readable. Use more dramatic fonts to make headings stand out.
• Headings: Use headings and subheadings to visually present the sets of information and hierarchy of ideas within your paper.

The visual mode also includes different ways to help readers focus on the key elements of illustrations, photographs, and images-based types of communication like videos or websites.

Aural Mode

This is what your audience hears. This is the primary mode of communication with public speaking, podcasts, radio, and other speech-based types of communication, but it's also an important ingredient in videos and film. This model extends the spoken words of the linguistic mode and the images of the visual modal with many other sound elements:

• Music: Set a mood.
• Sound effects: Suggest what is going on visually.
• Tone of voice: Illustrate the speaker's attitude toward a subject
• Volume: Rising or falling volume increases or reduces the emphasis on what's happening.

Spatial Mode

This mode presents information through the physical arrangement of images and information.

This is especially important when the communication is three-dimensional. How you arrange the chairs for your presentation to a class, for example, affects how that audience will respond. If you arrange them in a circle, you imply that you are connected to your audience and invite them to participate in the communication. If you arrange the chairs in rows that you stand in front of, you imply that you are the expert and that your audience should sit back, shut up, and receive your ideas.

The spatial mode also matters in two-dimensional forms of communication. How you divide website information into pages and tabs shows your readers how your information or ideas are structured and helps them to navigate to easily within that web-based presentation. If you put two illustrations next to each other on a page in your essay, the proximity of those images to each other suggests to readers that they are related. If you make one large and the other images small, it implies that one is more important than the other.

Gestural Mode

This is the way that people communication information nonverbally through facial expressions, body language, hand gestures, and touching.

If you cross your arms and say "I love you," for example, then your audience is getting a mixed message. The linguistic mode tells them that you love them, which is nice. However, the gestural mode tells them that you're also not 100% happy with them. They could do a better job helping out around the apartment. If nothing else, they could put on a coat before going outside in the rain.

You don't have any gestural modes with your college papers — except when you turn it in, perhaps, by slamming it down your professor's desk or something. However, this is really important to consider as you interact with others or as you watch the interaction of others.

3. Other Important Terms

In any rhetorical situation, the point of contact between the sender and receiver is a message, an idea. However, that idea is not transmitted directly from one brain to another. The idea has to first be translated into a tangible form — a television ad, a spoken suggestion, or a written paper. The vocabulary of multimedia communication offers a few important terms that relate to the tangible form of a message.

Text

Traditionally, the word "text" has describes something like a book or newspaper article or poem that is made with written words. With multimodal communication, however, written words are just one of many options, so "text" here refers to any kind of communication created with any or several or all of the following five modes. A speech is a text. A poster is a text. A video or podcast or graphic novel or Internet meme is a text. In the three examples from early in this section, the television ad, spoken suggestion, and poetry essay are all texts, too.

Medium

This one is tricky, so pay attention. The medium refers to whatever technology or mechanism is used to transmit the text from the sender to the receiver. It's the go-between.

So with a television ad text, which is basically a short film, the idea presented by that text is that you, friend, should buy some running shoes. The medium of television is the technology that transmits the text from the sender to the receiver.

With a message to put on a coat, your roommate could have texted you the text of that message on her phone. In that case, the text would have been a text, which is cool, and the medium would have been smartphone technology. She could have written you a note and left it on the back door. In that case, the text would be the note itself, and the medium would have been paper with sticky stuff on the back of it. However, instead of those impersonal media, your thoughtful roommate used the medium of direct, face-to-face conversation to transmit her message. She translated her idea into words that she sent to your ears with her voice. That was kind of her.

With the essay you are trying to write about that poem, the message will be your opinion of the poem. The text will be the essay you write that brilliantly explains and defends your opinion. The medium you will use to transmit your text will be ink on paper. You will print your words onto clean, white paper just as the assignment instructions require.

Affordance

This term refers to the strengths or potential uses of a given medium — the sort of messages that a medium allows you to transmit more effectively. Television, for example, is great for combining visual information, sound, and words into a cohesive message that is received and digested immediately. Face-to-face speaking is great for direct, face-to-face communication and allows you to add support to the message with gestures, such as your roommate's hands on her hips and frown. Paper is a great medium for messages that are received when the receiver has time to finally sit down and read the words on the page, probably after the kids are finally in bed.

Genre and Conventions

In the broader world outside the multimodal writing classroom, the term "genre" (ZHAHN ruh) usually refers to different types of creative works — written texts, music, or movies, for example. Fiction is a genre for writing, and nonfiction and poetry are genres. Blues is a genre of music. Horror and romantic comedies are two genres of movies.

What distinguishes one genre of writing, music, or movie from another is that each genre has different shared characteristics. As receivers of these texts, we expect that any text belong to a particular genre will have the features of that genre. You expect a fictional book to follow the convention of telling you made-up stories about made-up characters. You expect poetry to rhyme and be about love — or maybe you have a broader expectation after you take a poetry class, or maybe not. You expect a romantic comedy to end happily — horror films, not so much.

These expectations are known as conventions. Conventions are the rules that the texts of any given genre must follow.

With multimodal communication in college, "genre" can apply to any kind of multimodal of text, written or otherwise — class presentation, PowerPoint slides, research paper, or website, for example. This includes conventions for the message itself and the different modes of communication with the text.

Outside the college classroom, you'll find even more genres for multimodal communication — phone calls, texts, email, conversations with old people, and more. You might not have considered the conventions for these genres, but you still know what they are. You know how to follow them, and you can tell when someone violates them.

Every social media platform is its own genre, too. The conventions for Facebook are different from the conventions for Twitter or Instagram or whatever TikTok is. You'll find that the most effective posts on those platforms are the ones that follow those conventions in creative ways, not the ones that break them.

This discussion continues in Part 3 of this series, "The Multimodal College Paper."