Ken Rafanan
Brian Tobin
Contact: raf@stanford.edu
December 3, 2002
In order to keep up with market demands and with the competition, a company needs to train new employees well and keep current employees up-to-date with the current policies. Traditionally, corporate training occurs in a lecture or workshop setting at company gatherings or on retreats.
The biggest drawback is that such training can be costly since large companies would require numerous trainers to do the same programs with many groups of employees.
E-learning has been emerging as an alternative to the traditional corporate training. Corporate Elearning involves giving employees access to any number of electronic information sources, including computerized readings, hypertext, information databases, message boards, and email groups.
Regardless of the quality of an e-learning program, if a user loses interest in learning from the program or gives up on using it the program will cease to benefit that learner as well as the company. User frustration with the software with the software or even with computers in general could lead to discontinued usage or effectiveness of the program.
We spent 70 hours investigating current research and propose that designers employ the following design principles for maintaining the effectiveness of E-learning programs:
 Motivate users to manage their frustration with the program
 Convince users that they are in control of their emotional state
This chapter will explore the relationship between e-learning and frustration. First, we will explain the issues surrounding frustration and e-learning. Next, we will clearly state our guidelines and rationale. Then, we will use scenarios and storyboards to give examples of how an e-learning program may deal with user frustration. Next, we will address variation and tough decisions that arise when dealing with frustration in e-learning. Then, we will explain what we don’t yet know about frustration in e-learning and give some references and suggestions for how
Rafanan & Tobin raf@stanford.edu
1

Frustration in E-Learning to learn more about dealing with frustration in e-learning. Finally, we will wrap up this chapter by summarizing our guidelines and recommendations.
Frustration detracts from learning by lessening users’ attention, retention, learning, thinking creatively, normal reasoning capacities, and increases the likelihood of frustration with that source in the future. Currently, software provides, at best, passive emotional regulation, like pulling attention away from the source of frustration or allowing the user to vent. However, passive emotional regulation does not encourage people to overcome the source of frustration- rather it simply alleviates the current emotional state.
An effective way to help users overcome or deal with frustration is for the software to provide active listening by giving sincere, non-judgmental, sympathetic or empathetic feedback to a frustrated individual. Klein, Moon, and Picard (2002) found that even rudimentary active listening responses from software can significantly decrease users’ frustration and cause longer usage and more satisfaction with frustrating software.
Our research and recommendations will benefit adult employees of large and small businesses who participate in e-learning programs. Their participation in such training and learning programs is likely mandatory.
Research by Nass and Reeves showed that people anthropomorphize computers and form relationships with media. Wenger (1998) stated that positive relationships between participants in learning environments can enhance the resulting learning. Thus, we assert that the relationships that users form with e-learning software can influence positively users’ learning. If users become extremely frustrated with the software, then this poor relationship with the software will detract from their learning.
Designing an e-learning environment is especially difficult because not only have to worry about users not just as users but also as learners. Designers of e-learning programs can alleviate user frustration in two major ways:
1.
Finding and removing frustrating tasks and elements from the program
2.
Helping users how to better deal with potentially frustrating tasks and concepts
The first of these recommendations should be a familiar principle of good user interface design.
The second recommendation is more difficult to deal with. Sometimes it is the challenging and potentially frustrating content that offers the most learning opportunities and what is challenging and enjoyable for some users may be simply frustrating for others.
Our goal is to provide designers with guidelines on how to create the most effective e-learning environments. We firmly believe that the most effective learning can occur only when the software helps learners deal with potential frustrations.
Rafanan & Tobin raf@stanford.edu
2

Frustration in E-Learning
We suggest adhering to two main guidelines for helping users deal with possibly frustrating situations:
1.
Teach your learners general frustration management techniques
2.
Human learners are emotional beings so design your software accordingly
Teach your learners general frustration management techniques.
We believe that the most important lesson your e-learning program can teach your employees is how to manage their frustration better. If your learners understand themselves more and know the ways they can work through frustrating situations, then they will be less likely to become frustrated and will use your e-learning software better, longer, and will learn more.
There are many different suggestions and proposed methods for how individuals
can deal with frustration. We have included some sample frustration management techniques in this sidebar but
From “Controlling anger- before it controls you”, recommend that you develop a targeted curriculum for your employees that provides them with recommendations and gives them opportunities to practice dealing with frustration. pamphlet by American Psychological Association

Relax and refocus
 Change the way you’re thinking about the problem

Change the way you think about your situation

Give yourself a break
From “Dealing with Frustration”, by David Gershaw

Keep trying: repeated efforts, successes, and failures may help you achieve your goal

Consult with others on how to achieve your goal

Decide if there are alternate methods to achieving your goal

Determine whether there are similar or alternate goals that are more achievable
Human learners are emotional beings so design your software accordingly
Use human-human interaction as your model
Giving your e-learning system some key human characteristics will go a long way in helping your learners to become comfortable with using the system and maintaining a positive disposition towards the software even when frustration is encountered. A number of persuasion techniques should be employed in building the relationship with the user.
Make a good first impression
Build a positive relationship with the learner from her first interaction with the program. As with any human-human interaction, laying out the relationship in advance and explaining the role of
Rafanan & Tobin raf@stanford.edu
3

Frustration in E-Learning each participant will help the relationship continue running smoothly. The e-learning program should state what it will do and what the user can do. The system should build credibility as a learning tool by letting the learner know how well it’s been designed. For example, it could say
“I’ll present the lessons in a way that’s proven effective for most people. If you don’t understand something then I can give you a little more help”. The user will eventually learn that some of the content or some aspects of the system will be frustrating so allow the system to gain the users’ trust by confiding to the user that it is not flawless and will not always be easy to use. In addition, computer users are not immune to flattery. If the program tells learners how intelligent they are, how quickly they pick up on things, or other flattering things the user may enjoy working on the program more.
Pay attention to users’ feelings and needs
Even simple attention to users’ feelings can build rapport with the system or encourage users to work through frustration. Users’ frustration may be predicted by identifying commonly frustrating situations or modules in the system, monitoring time spent or mouse clicks, or through more complicated methods of measuring stress. Likely points of frustration could be greeted with examples of active listening by the computer. Active listening could have many components. We suggest the following:

Actively solicit information about the user’s emotional state, especially frustration
How are you feeling about your progress?
1.
I’m doing fine. Thanks for asking!
2.
I’m having a little trouble but will manage.
3.
I’m pretty frustrated.
4.
I’m totally frustrated with this program!!!
1.
User: I’m doing fine. Thanks for asking!
System : That’s great! I’m glad we’re working well
2.
User: I’m having a little trouble but will manage.
System : Alright. Your work ethic is impressive!
3.
together.
Keep at it.
User: I’m pretty frustrated.
System : I’m sorry. If you keep at it maybe you’ll get it. Would you like to review your frustration management techniques? See the bulletin board?
Asking the user how he or she is feeling about progress is a simple way to obtain users’ frustration
4.
Send feedback?
User: I’m totally frustrated with this program!!!
System : I’m really sorry! Do you want to go to the
Bulletin Board? Just move on? Would you like to levels and to determine if an intervention is necessary. give the programmers feedback?

Respond to users’ frustration with sympathy and empathy
Even the simplest examples of affective computing have been proven effective at lessening users’ frustration levels and increasing.

Predict what support a learner might need
By providing support before a learner experiences frustration, your system validates the learner’s emotional state and lessens her propensity to become frustrated.
Rafanan & Tobin raf@stanford.edu
4

Scenario 1: Junko
Junko, 25 year old new hire from Japan knows little about the company and its processes and procedures. She wants to learn everything and to learn it well. She is both intrinsically motivated because she wants to be a good employee and is extrinsically motivated because she must complete the elearning program to become a manager.
Junko has been using the e-learning system for a few hours and is about to begin the section about budgeting. Since the software designers have identified the budgeting module as a likely source of frustration for users, the software warns Junko that the module is particularly difficult and asks if she would like to review the frustration management techniques she learned earlier.
She chooses to review these lessons and this review helps her relax and stay focused on the material.
Frustration in E-Learning
Rafanan & Tobin raf@stanford.edu
5

Frustration in E-Learning
Scenario 2: Steve
Steve, a 35 year old white male – has been a manager for six years. Steve thinks that he knows everything he needs for his job managing sales people in his region, and he doesn’t appreciate being forced into the mandatory training. Steve wants to get through the training as fast as possible so that he can concentrate on what he feels are his higher job priorities. Because of this disposition,
Steve is likely to suffer frustration if he can’t work through a module on the first try.
Steve works on a lesson going over the company’s personnel policies dealing underperforming employees. Thinking that he is well-aware of the policies, Steve barely skims the material. Due to a recent change in labor law, the company’s “first response” policy now requires managers to provide a written request for improvement as well as a 360 evaluation soliciting feedback from the target employee.
In the module post-test, which determines whether the learner may move on, a question about this company policy has been made pass/fail – users answering incorrectly will be forced to work through the module a second time. In the post-test,
Steve receives a score of 8 of 10, but he fails the test because he missed the question about the company’s first response policy. Steve becomes frustrated – he believes he knows the material well enough as evidenced by an 80% score.
The frustration response system knows that a failure on the test can lead to learner frustration.
Before Steve has a chance to move beyond frustration into anger and suffer the corresponding decrease in learning ability, the FRS polls Steve on his level of frustration. His response of 4 out of 4 (indicating a very high level of frustration) causes the system to provide options of using a
Bulletin Board, skipping the section or providing feedback to the programmers as ways to work through this frustration.
Rafanan & Tobin raf@stanford.edu
6

Frustration in E-Learning
Steve chooses the bulletin board because he wants to know why he isn’t passing with an 80% grade.
On the bulletin board he asks if anyone else has had the same problem. Another user had read her test results a little more carefully; she tells Steve that some of the questions require correct responses for a passing grade. Steve looks back at the detailed test results and sees that he had indeed missed one of the pass/fail questions. He is still not very happy, but he isn’t as frustrated because he better understands his situation.
Steve returns to the module paying closer attention this time.
Attempting to build a relationship between the learner and a learning system can be very beneficial. However, a designer needs to be aware that a poor relationship may be worse than none at all. As is clear from people’s reactions to Microsoft Word’s Clippy agent, overly frequent interventions or misdiagnosing problems can lead to greater frustration than no intervention at all.
We must also guard against eliminating all sources of frustration as these may be rich learning opportunities. One user’s engaging challenge can be another’s source of immense frustration.
Rafanan & Tobin raf@stanford.edu
7

Frustration in E-Learning
There are not yet many studies investigating what frustrates users in e-learning and what design principles designers can use to alleviate or help users deal with frustration. We make the best recommendations we can with the current body of research but propose a few future studies to investigate:
1.
particularly frustrating situations, tasks, and concepts specific to e-learning environments
2.
reliable methods to monitor users’ frustration while using a computer
3.
additional effective techniques for reducing frustration within an e-learning environment
Frustration and Computing
Professor Rosalind Picard is leading the Affective Computing Research Group at the MIT Media
Lab (http://affect.media.mit.edu/AC_affect.html). They state: we think mechanisms of emotion will be needed to build "human-centered" machines, which are able to respond intelligently and sensitively to the complex and unpredictable situations common to human-computer interaction
Their research will prove valuable when determining how frustration management is possible within e-learning.
Users’ Relationships with Computers
We recommend reading The Media Equation for ideas on the social relationships users can form with computers. Learners will form some sort of relationship with an e-learning program so designers of these systems should decide what kind of relationships they want to foster.
E-learning has the potential to greatly augment corporate training. Within any curriculum, the teacher/designer must endeavor to keep users motivated, but we must expect some learner frustration and the resulting decrease in motivation. E-learning designers, then, should be prepared to address this frustration using our guidelines.
We developed these guidelines using grounded theories and current research on how to deal with learner frustration in computing. We believe that e-learning systems should ensure that their learners are aware of how to deal with frustration and pay attention to and intervene in situations when learners need extra help dealing with frustration. Only when designers remember that users’ emotions and feelings are an integral part of how they use and learn from an e-learning system will e-learning fulfill its promise and truly earn its place among proven learning tools.
Rafanan & Tobin raf@stanford.edu
8

Frustration in E-Learning
Controlling anger- before it controls you. Retrieved December 2, 2002 from PsychNET
( http://www.apa.org/pubinfo/anger.html
).
Gershaw, D. (1997). Dealing with Frustration. Retrieved from A Line on Life
(http://www.azwestern.edu/psy/dgershaw/lol/Frustration2.html).
Klein, J., Moon, Y., & Picard, R.W. (2002). This computer responds to user frustration: Theory, design, and results. Interacting with computers, 14, pp. 119-139.
Lawson, R. (1965). Frustration: The development of a scientific concept. Macmillan, NY.
Reeves, B. & Nass, C. (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places.
Cambridge: Cambridge University Press.
Scheirer, Fernandez, Klein, & Picard. (2002). Frustrating the user on purpose: a step toward building an affective computer. Interacting with computers, 14, pp. 93-118.
Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity . Cambridge:
Cambridge University Press.
Rafanan & Tobin raf@stanford.edu
9