Running head: LIMITATIONS OF SELF AND PEER REPORTING
Limitations of Self and Peer Reporting
Gloria Lowe
Yorkville University
August 9, 2015
1
LIMITATIONS OF SELF AND PEER REPORTING
2
Abstract
The study of personality poses many difficulties. Thoughts, feelings, and personality
traits are not easily measurable. We interpret behavior and words as indicators of personality. In
this way, then self-reporting and peer-reporting are the two most common assessment
techniques. Date can be collected through questionnaires or interviews. This paper discusses the
limitations of self-reporting and peer-reporting with respect to response set bias, and cultural and
ethnic bias along with the issue of cost, time and content validity. Data collection techniques are
also compared with respect to their limitations.
LIMITATIONS OF SELF AND PEER REPORTING
3
Limitations of Self and Peer Reporting
The personal construct theorist George Kelly said “If you don’t know what is wrong with
a person, ask him; he may tell you.” (Kelly, G, as cited in Barker, Pistrang, & Elliott, 2015, p. 2)
Personal construct theory brought with it the realization that people are able to examine and
report on their own psyche, and ever since, the self-report method is the most widely used
assessment tool in modern psychology. (Barker et al., 2015) Self-report not only provides the
therapist or researcher with information, but also empowers the subject to play an active role in
assessing themselves and their problems, thereby becoming an active participant in their quest
for mental health and self-improvement. (Kearney, 2013)
Self-report has the advantage of a
direct answer that includes the feelings, motives, thoughts and mood that accompany behaviour.
Self-report is inexpensive and simple to acquire. Technology opens the door for online
assessments that can be done at a convenient time.
Despite the many advantages, there are, of course, disadvantages to self-report methods.
Questions arise concerning validity issues arising from the design of the questions and technique
used, construct, content and face validity, (Friedman & Schustack, 2011) and the validity of the
inferences made of results due to common method variance. (McDonald, 2008) There are so
many areas in which bias may affect data. Response set biases can include acquiescence bias,
social desirability bias, extreme responding bias, self-serving bias, self-deception, selective recall
bias, ethnic and gender bias. (Barker et al., 2015; Churchill, 2000; Friedman & Schustack, 2011;
Paulhus & Vazire, 2008)
Common method variance can occur when only one method of assessment is used. Peerreporting, or informant reporting, is often combined with self-reporting. This offers another
viewpoint: that of a friend, relative, colleague, therapist, or sometimes a non-connected
LIMITATIONS OF SELF AND PEER REPORTING
4
informant chosen by the researcher. These observers can infer personality factors by observing
behaviors and interactions. (McDonald, 2008) With peer-reporting, however, there can be issues
with time and expense coupled with the difficulty of recruiting the right informants. If the
informant knows the subject, there is the possibility of responses being colored by love for or a
dislike of the subject. There is also the possibility of fundamental attribution error or the actorobserver effect. (Nisbett & Ross, 1991 as cited in McDonald, 2008)
This paper will discuss limitations that occur in relation to validity and bias of self and
peer reports along with problems associated with the different delivery options of self and peer
reports.
Bias in self-report
Sample Bias
The self-report process is open to many sources of bias, the first of which is sample bias.
Recruiting participants for research studies can be difficult. Pagan et al noted that certain
personality types are more likely to agree to participate, and certain types are more apt to delve
into details about their personality. (Pagan, Eaton, Turkheimer, & Oltmanns, 2006)
A response set describes attributes about a respondent that are not directly from the
questions but from the overall response pattern. (Friedman & Schustack, 2011) There are many
types of response set bias.
Social Desirability Bias
A social desirability response set bias comes from the human tendency to show
themselves in a good way. (Friedman & Schustack, 2011; Donaldson & Grant-Vallone, 2002)
This is common in employment related studies. For example, a worker will answer questions in
such a way as to impress the employer by downplaying negative traits. (Shore, Shore, &
LIMITATIONS OF SELF AND PEER REPORTING
5
Thornton, 1992) When social desirability response bias is conscious it is called impression
management. Studies on impression management call the ability to lie effectively test-wiseness.
(Stricker, 1969) Test-wiseness involves the ability to interpret the questions with an
understanding of the kind of answers that will produce the desired impression given the culture
and community involved. (Stricker, 1969) Studies show that the structure and format of the
assessment is important. The use of indirect question formats (taking the perspective of another
person) can reduce the possibility of social desirability bias. (Neeley & Cronley, 2004)
When inaccurate responses are unconscious, it is termed self-deception.
Psychoanalysts, for example, do not believe that we can fully understand our unconscious, and
that self-deception is unavoidable. (Barker et al., 2015) This can also be attributed to the
tendency of people to describe experiences in light of their perception of the results or outcome
rather than the process itself. (Nisbett & Wilson, 1977, as cited in Churchill, 2000) Some
theorists go so far as to say that people are “predisposed to self-enhancement” (McDonald, 2008,
p. 4) Perceived success or failure of events or situation will influence their description of
themselves. Critics of the “self-deception” concern point out that researchers are aware of this
tendency, and that valid questionnaires design questions to take this into account. A
phenomenological approach looks more at intention and tendencies throughout questions, and
traits can be interpolated from this data. (Churchill, 2000)
Social psychologists would agree that unconscious motives are not always clear in that
people have a tendency to explain the behavior of others in terms of personality traits while they
explain their own behavior as a result of situational factors. This is called the actor-observer
effect. (Barker et al., 2015)
LIMITATIONS OF SELF AND PEER REPORTING
6
When the outcomes of the self-report involve rewards or punishments, as in an
employment situation, there is possibility of self-serving bias. Self-serving bias involves taking
credit for successes while denying responsibility for failures. (Barker et al., 2015) The
probability of self-serving bias is linked to the perceived positive or negative outcomes of the
situation. (Nisbett & Wilson, 1977)
Acquiescent Responding
An acquiescence response set bias comes from a tendency of some people to always.s
agree with statements. (Paulhus & Vazire 2008; Friedman & Schustack, 2011) While analyzing
results the question arises as to whether the answers indicate a strong personality trait or merely a
strong acquiescent tendency. (Paulhus & Vazire, 2008)In general interactions, people with this
tendency will agree or go with the flow rather than take a stand against something. (Paulhus &
Vazire, 2008) On a questionnaire containing closed ended items, they may often just agree rather
than thinking deeply about their true opinion. The opposite tendency that of always disagreeing
or going against suggestions, is referred to as reactant responding bias. (Barker et al., 2015)
Paulhus and Vazire point out that a high number of “agrees” or “disagrees” in itself can point out
personality traits such as a desire to conform. Answers pointing to high anxiety, for example,
could indicate high anxiety or high acquiescence. (Paulhus & Vazire, 2008) Acquiescence bias
can be decreased by careful wording of questions to ensure that some are worded affirmatively
and some negatively. (Barker et al., 2015) Open ended questions where there is no option to
agree or disagree are another option, but these have other limitations, to be discussed later in this
paper.
LIMITATIONS OF SELF AND PEER REPORTING
7
Extreme Responding Bias
Extreme Responding Bias occurs with individuals who are more likely to choose the
outside scores on a rating scale. (There can be a tendency to always choose central ratings, as
well). (Paulhus & Vazire, 2008) “Situational factors such as ambiguity, emotional arousal, and
rapid responding induce temporary increases in extreme responding.”(Paulhus & Vazire, 2008,
p. 231)
Extreme responding bias can be avoided by using agree / disagree questions, but these
may not allow the subject to answer in a meaningful way. The Q-sort process is an often used
method to avoid extreme responding. (Paulhus & Vazire, 2008) Open ended questions or an
interview method may be effective as well. (Barker et al., 2015)
Selective Bias
Selective bias (Sato & Kawahara, 2010) can also be referred to as recall bias. (Barker et
al., 2015) “People do not have an infinite ability to recall all information relevant to a posed
question.” (Paulhus & Vazire, 2008, p. 232) High stress, anxiety and negative mood states can
tend to cause exaggeration of these states in retrospective reports. (Kawahara & Sato, 2010)
Because reporting retrospectively gives time for analysis of situations, memory can be colored
by impressions of the outcome of the situations and coping mechanisms that have been activated.
(Paulhus & Vazire, 2008)
Gender and Ethnic Bias
Gender and Ethnic bias is of concern in self-reporting, particularly in the interpretation of
results. (Friedman & Schustack, 2011) If examples and situations used in the questions are set
in contexts that are not a part of the subject’s normal life experience, answering may be difficult
or contrived. Where very different cultures are involved there may even be questions that are
LIMITATIONS OF SELF AND PEER REPORTING
8
completely irrelevant. Language barriers can also be an issue. Gender and ethnic bias are a risk
with the interpretation of data as well. As discussed in Friedman and Schustack, if the subject
comes from a culture that prizes “cooperation, humility and modesty” (Friedman & Schustack,
2011, p. 32) the scorer could interpret their scores as “excessively shy” (p. 32) if they do not take
into account the cultural background of the subject. The same bias can occur with gender
stereotyping, although this has lessened in the past few decades. Women who come across as
aggressive, dominant, and extroverted may be labelled differently than a man who has the same
scores. (Paulhus & Vazire, 2008)
Common Method Variance Problem
A Common Method Variance can affect the validity of the interpretation of data. When a
study relies only on self-report data and is analyzed by one person, there can be an effect that is
not picked up as error, because the criterion, wording, reading, and style of understanding is
common to the two people involved. Studies that rely only on self-report tend to be criticized for
having Common Method Variance. (Donaldson & Grant-Vallone, 2002) A popular way pf
combatting this problem is to include peer-reports in the study. While Peer-reports offer a
different perspective to help with the Common Method Variance Problem, they in themselves
have limitations. (Barker et al., 2015; Donaldson & Grant-Vallone, 2002; Lynch, McNamara, &
Seery, 2012; Rodebaugh, Gianoli, Turkheimer, & Oltmanns, 2010)
Limitations of Peer Reporting
We make assumptions about the personality of others through observing their words,
gestures, and behaviors. By including peer reports, a study can be more reliable, as information
LIMITATIONS OF SELF AND PEER REPORTING
9
comes from a variety of sources. (Rodebaugh et al., 2010) The more informants involved, the
more reliable the results. (Eysenck, 2004) Peer reporting has its own limitations, however.
Time and Cost
Informants for peer reports can be family members, friends, colleagues, or hired
participants in academic studies. Using hired participants, while having the benefit of
objectivity, can be costly and time consuming. (Rodebaugh et al., 2010; Paulhus & Vazire, 2008)
Often participant drop out of the study or miss sessions. (Pagan et al., 2006) Advertising can be
time consuming, and as certain personality types are more prone to participate in studies, there
can be sample bias. (Pagan et al., 2006)
Social Bias
It has also been noted that people with social anxiety have an exaggerated negative view
of how they appear to others.(Rodebaugh, Gianoli, Turkheimer, & Oltmanns, 2010) While peer
reports avoid the bias of self-deception and distortion that can occur with self-reports in
assessing personality disorders, there can be social bias here as well. (John & DiLalla, 2013)
Because peer reports are working only on expressions and behaviors, there is a danger of
misinterpreting feelings and motives. One example could be a subject experiencing anxiety and
the observer interpreting gestures and facial expressions as shyness or a cold, distant personality.
(John & DiLalla, 2013; Rodebaugh et al., 2010) John et al discuss how emotion recognition is an
important factor in the accuracy of their report. Emotion recognition describes the “ability to
recognize the feelings of those around us”. (John & DiLalla, 2013, p. 474) Interpretations of
negative expressions and gestures (for example fear, anger, and frustration) tend to result in an
exaggerated description. If the subject has a personality disorder, their behaviors and
mannerisms can be misinterpreted by even mental health professionals. (Rodebaugh et al., 2010)
LIMITATIONS OF SELF AND PEER REPORTING
10
Ethnic and gender bias are a strong possibility with peer reporting. An informant is of a
different culture or gender can misinterpret behaviors. For example, some aboriginal cultures in
Canada see eye contact as rude or defiant. An informant not knowing this could rate a subject as
defiant, rude or avoidant when in fact the subject was being respectful and polite. (Baydala,
Birch, Muth, & Rutten, 2009)
Personal Bias
“Peer reports can offer a unique perspective on an individual’s personality”
(McDonald, 2008, p. 5) but if the informants know the subject personally, there can be bias.
Informants can unconsciously allow their like or dislike of the subject affect their impressions by
incorrectly assuming thoughts or motives behind expressions or gestures. An interesting
example is in a study by John et al, where parents are likely to downplay situations where their
children are reporting feeling victimized. (John & DiLalla, 2013) Studies have also found that
informants may themselves have personality disorders that colour their interpretation of others.
(Pagan et al., 2006)
Reporting Methods
Questionnaires
Self and peer reports can take the form of a questionnaire or an interview. Questionnaires
can be on paper or online. (Barker et al., 2015) While online questionnaires are easy and
inexpensive to administer, it is not possible to guarantee the identity of the respondent. Paper
questionnaires are usually in the form of well written, standardized questions. Questions must be
carefully designed to ensure validity. True-false questions, for example, eliminate the possible
of “sometimes” along with potentially important qualifying information. True-false questions
LIMITATIONS OF SELF AND PEER REPORTING
11
are prone to acquiescence bias, as well. (Paulhus & Vazier, 2008) This applies to all closedended questions, where the number of choices limits the subject who isf forced into choosing one
qualitative answer even though it might not fit exactly. (Barker et al., 2015) While scale rated
questions give more choices, they are prone to extreme responding bias.
Open ended questions, on the other hand, allow the subject the freedom to express
feelings with their own descriptors. The danger is that there can be exaggeration or “fabrication”
(Barker et al., 2015, p. 5) It can be difficult for the researcher to interpret the information from
the same viewpoint as the subject, resulting in interviewer bias. Ethnic bias is also a concern
here. Of note is that in employment situations peer reports have been found to be very accurate
when it comes to predicting job placement and ability. (Shore et al., 1992)
While questionnaires offer confidentiality, privacy and lack of time restraint, there is a
risk of the respondents not understanding, omitting, or misinterpreting questions, particularly
when language or culture is different. (Barker et al., 2015)
Interviews, conversely, allow for the respondent to ask clarification and allow the
interviewer to ensure that all questions are answered. Impression management may be an issue,
especially if the interviewer is someone that the respondent wants to impress. (Barker et al.,
2015) The time and cost limitations with interviews are evident. The personalities and comfort
level of the interviewee and the interviewer are very important, as if there is a discomfort, the
interviewee may not respond truthfully or openly. (Barker et al., 2015) This is particularly true in
health care situations where there must be a sense of respect and trust between parties. (Baydala
et al., 2009)
LIMITATIONS OF SELF AND PEER REPORTING
12
Conclusion
As the study of personality develops, methods become more refined and bias and validity
issues are identified and addressed. (Eysenck, 2004) Both methods are at risk of social
desirability bias, extreme responding bias, acquiescence bias, and ethnic and cultural bias. With
self-reports there is a possibility of self-deception; when combined with peer reports greater
accuracy will result. (McDonald, 2008) Peer reporting can be costly and time consuming, and
finding appropriate participants can be difficult. (McDonald, 2008) Despite the limitations of self
and peer reports, when used together they provide valuable insight into personality, enabling
researchers and counsellors to gain an understanding of the subject and indications of methods to
improve mental health.
LIMITATIONS OF SELF AND PEER REPORTING
13
References
Barker, C., Pistrang, N., & Elliott, R. (2015). Research Methods in Clinical Psychology: An
Introduction for Students and Practitioners (3rd ed.). Hoboken, N.J.: J. Wiley.
Baydala, L., Birch, J., Muth, T., & Rutten, L. (2009, december). Women: A case for culturally
competent care for aboriginal women giving birth in hospital settings. Journal of
Aboriginal Health, 24-34. Retrieved from www.naho.ca
Churchill, S. D. (2000). “Seeing through” self deception in narrative reports: Finding
psychological truth in problematic data. Journal of Phenomenological Psychology, 31(1),
44-62.
Donaldson, S. I., & Grant-Vallone, E. J. (2002). Understanding self-report bias in organizational
behavior research. Journal of Business and Psychology, 17(2).
Eysenck, M. J. (2004). Psychology: An International Perspective. Retrieved from
www.googlebooks.com
Fletcher, J. S. (1999). Using exercise to increase self-efficacy and improve health behaviors.
(Master’s thesis). Retrieved from https://research.wsulibs.wsu.edu
Friedman, H. S., & Schustack, M. W. (2011). Personality: Classic theories and modern
research. Boston: Allyn and Bacon.
John, S. G., & DiLalla, L. F. (2013). Explaining differential reporting of victimization between
parents and children: A consideration of social biases. Behavioral Sciences, 473-491.
http://dx.doi.org/10.3390/bs3030473
Kawahara, J., & Sato, H. (2010, November 23). Selective bias in retrospective self-reports of
negative mood states. Anxiety, Stress, and Coping., 24(4), 359-367.
LIMITATIONS OF SELF AND PEER REPORTING
14
Kearney, S. (2013). Improving engagement: the use of ‘authentic self-and peer-assessment for
learning’ to enhance the student learning experience. Assessment & Evaluation in Higher
Education, 38(7), 875-891. http://dx.doi.org/10.1080/02602938.2012.75196
Lynch, R., McNamara, P. M., & Seery, N. (2012). Promoting deep learning in a teacher
education programme through self- and peer-assessment and feedback. European Journal
Of Teacher Education,, 35(2), 179-197. http://dx.doi.org/10.1080/02619768.2011.6433
359 -367
McDonald, J. D. (2008, June). Measuring Personality Constructs: The Advantagesand
Disadvantages of Self-Reports, Informant Reportsand Behavioural Assessments.
ENQUIRE, 1(1). Retrieved from www.nottingham.ac.uk
Neeley, S., & Cronley, N. (2004, March). When Research Participants Don’t Tell It Like It Is:
Pinpointing the Effects of Social Desirability Bias Using Self vs. Indirect-Questioning .
Advances in Consumer Research, 31. Retrieved from www.acrwebsite.org
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on
mental processes. Psychological Review, 84(3), 231-259. http://dx.doi.org/10.1037/0033295X.84.3231
Pagan, J. L., Eaton, N. R., Turkheimer, T. F., & Oltmanns, E. (2006). Peer-reported personality
problems of researchnonparticipants: Are our samples biased? Personality and Individual
Differences, 41, 1131-1142. Retrieved from www.sciencedirect.com
Paulhus, D. L., & Vazire, S. (2008). The self-report method. In R. W. Robbins, R. C. Fraley, &
R. F. Kreuger (Eds.), Handbook of research methods in personality psychology (pp. 224239). New York: Guilford.
LIMITATIONS OF SELF AND PEER REPORTING
15
Powell, R. W. (2008). The effects of negative affectivity on self-reported activity limitations in
stroke patients: Testing the symptom perception, disability and psychosomatic
hypotheses. Psychology & Health, 23(2), 195-206.
Rodebaugh, T. L., Gianoli, M. O., Turkheimer, E., & Oltmanns, T. F. (2010). The interpersonal
problems of the socially avoidant: Self and peer shared variance. Journal of Abnormal
Psychology, 119(2), 331-340.
Shore, T. H., Shore, L. M., & Thornton, G. (1992). Construct validity of self and peer
evaluations of performance dimensions in an assessment center. Journal of Applied
Psychology, 77(1), 42-54.
Stricker, L. J. (1969, June 2). “Test Wiseness” on personality scales. Journal Of Applied
Psychology, 1.