2005-06 Hunt Grant Research Report Perceptions of Graduate Admission Directors: Undergraduate Student Research Experiences: "Are All Research Experiences Rated Equally?" Judith Correa Kaiser, PhD, Andrew J. Kaiser, PhD, Nick J. Richardson, and Edward J. Fox St. Ambrose University (IA) Application and acceptance rates for PhD and PsyD clinical and counseling programs in psychology show that these programs are highly competitive. Knowing which variables graduate admissions committees deem important is critical for undergraduate faculty advisors to inform these students of what graduate programs want from their applicants in order to be chosen for this field of study. The primary goal of this study was to find information on how different types of undergraduate research experiences and students' other activities were rated by 237 programs' graduate admissions directors, chairpersons, or their designees. Four of 39 variables were shown to be statistically significant and highly ranked by these participants. All 4 variables have one thing in common: publication or presentation of research in a referred journal or regional conference. Examination of the application and acceptance rates for PhD and PsyD clinical and counseling programs in psychology shows that these programs are highly competitive (American Psychological Association [APA], 2005; Mayne, Norcross, & Sayette, 2006). As faculty members who teach and advise undergraduate psychology majors who are primarily interested in these competitive graduate programs, knowing which variables graduate admissions committees deem important is critical. To inform these undergraduates of the factors graduate programs look for in applicants and the importance these programs give to these factors, each year the APA publishes the guidebook, Graduate Study in Psychology (APA, 2005). In addition, books, research articles, and other publications (e.g., Eye on Psi Chi) provide undergraduate students interested in applying to graduate schools recommendations on what to do to increase the likelihood of being seriously considered for these very competitive programs (APA, 1997; KeithSpiegel, Tabachnick, & Spiegel, 1994; Mayne et al., 2006; Walfish & Turner, 2006). In general, students who apply to graduate programs in psychology are evaluated on various objective and subjective criteria. Objective criteria include Grade Point Average (GPA), scores on the Graduate Record Examination (GRE), and in some cases, the curriculum of study completed (APA, 1997; Keith-Spiegel et al., 1994). Subjective criteria evaluated are letters of recommendation, research experience, and autobiographical statements or statements of purpose (APA, 1997; Blashfield, Keeley, Burgess, & Everett, 2005; Keith-Spiegel et al.; Norcross, Hanych, & Terranova, 1996; Walfish & Turner, 2006). A body of research has shown that among the subjective, or second-order, criteria used by graduate selection committees to rate applicants, students' research experiences are rated as some of the most important (APA, 2005; Cashin & Landrum, 1991; Keith-Spiegel et al., 1994; Landrum, Jeglum, & Cashin 1994; Mayne et al., 2006; Walfish & Turner, 2006). Because of these findings, researchers suggest that to increase a student's competitiveness for acceptance to psychology graduate programs, undergraduate programs should provide more research opportunities (Keith-Spiegel et al.). In a study conducted in 2005, Perlman and McCann assessed students' research experiences in undergraduate psychology program curricula. They found of the 203 undergraduate psychology programs surveyed, 72% had a research experience course as part of their program requirements. Moreover, these researchers found that 88% of departments offered a class with structured research experiences, and 90% of the departments allowed students to conduct individual research projects. Given the high percentages of undergraduate programs requiring research experience, one would expect that most psychology students with hopes of attending graduate school would have presented or published some research by the time they graduated with their bachelors' degrees. However, it seems that most students did not take advantage of their experiences to publish or present. Ferrari and Appleby (2005) surveyed a random sample of Psi Chi undergraduate students who graduated with psychology degrees and found that although the majority of the students had completed research projects, most had not presented any research in conferences nor published. It is possible that part of the reason for not publishing their undergraduate research is the high rejection rates from most psychology journals. Powell (2000) reported that between 50% to 90% of all research articles submitted for publication are rejected. However, according to this author, acceptance rates for undergraduate conferences and undergraduate journals are higher; nevertheless, these venues are not viewed as prestigious as the ones for professional researchers. Differences in prestige should not be surprising because they are found even among state, regional, and national conferences and peer-reviewed journals. A question that might influence advisors' and their students' decisions to submit research to undergraduate journals is "How do graduate selection committees rate this type of research experience when evaluating students for entrance into their programs?" Several studies have been conducted exploring this question. Ferrari and Davis (2001) found that faculty members from research universities did not view these undergraduate publications favorably. Similarly, Ferrari and Hemovich (2004) found that directors of graduate psychology programs did not view publications in undergraduate student journals as being a significant factor for acceptance to their graduate schools. Thomas, Rewey, and Davis (2002) posed a similar question to faculty advisors of students who had published in the Psi Chi Journal of Undergraduate Research. The results of this study indicated that faculty advisors believed publications in the Psi Chi Journal were viewed positively by graduate school selection committees. Furthermore, these researchers found that the faculty advisors indicated that if they were part of a graduate school selection committee, this type of publication would increase a student's likelihood of being accepted into their graduate program. Thus, the literature consistently shows it is important for undergraduate students to possess research experiences (e.g., Ferrari & Davis, 2001; Ferrari & Hemovich, 2004; Keith-Spiegel et al., 1994; Mayne et al., 2006; Walfish & Turner, 2006). However, when it comes to applying to competitive graduate programs, graduate school selection committees seem to rate the value of undergraduate research experiences for students' admittance to their program in different ways (Ferrari & Davis; Ferrari & Hemovich). The primary goal of this study was to examine how different types of undergraduate research experiences were rated by graduate program directors (or the appropriate admission persons) when making decisions about students' admittance to their programs. Additionally, we were interested in obtaining information about how students' other activities (subjective criteria) were rated by program directors from these competitive programs. METHOD Questionnaires about admission to their graduate programs were sent to the directors or chairpersons (or their designees) of the 237 APA-approved clinical or counseling PhD or PsyD programs. These psychology graduate schools were selected from the APA's Graduate Study in Psychology, 2006 Edition (2005). After the 237 graduate schools had been identified using this book, the current director, or chairperson, and the school's current address were confirmed by accessing the graduate school's website. The questionnaire used in this study was developed by the researchers and followed a format similar to the questionnaire developed in the research conducted by Keith-Spiegel et al. (1994). It consisted of the following scenario and a series of responses which were rated on a Likert-type scale of 1 (not important) to 5 (extremely important): Imagine that you have an excellent pool of applicants for your graduate program. These applicants all have high GPAs (3.7 or above on a 4.0 scale), high GRE scores, very good interviews, and strong letters of recommendation. Please rate the items listed below in terms of their importance in assisting you to make your final determination of who gets accepted to your program. Each director or chairperson (or appropriate designee) received a letter explaining the research and an informed consent form along with the questionnaire. Included in this letter was the link that program officials could access if they preferred to complete the questionnaire online. The online questionnaire was made available on a secured website attached to the St. Ambrose Psychology Department's homepage. Both the physical and the online versions of the questionnaire contained identical content. After 4 weeks, all 237 graduate school representatives received an email message explaining the purpose of the survey with the link to access it online or thanking them for their participation. Because our study allowed for anonymous responses, we could not know which individuals had completed the questionnaires; thus, all program directors received the email. Though it was not guaranteed that directors would not respond to the paper version and the Internet version, we believed that these graduate personnel were too busy to fill in more than one questionnaire. Data collection ended two weeks after the reminder email message was sent. RESULTS Of the 237 surveys sent, 71 questionnaires were completed: 40 by mail and 31 by use of the website. The return rate was 30%. The means and standard deviations of the ratings for each item are presented in Table 1 in order of most important item considered for program admission to least important. A Chi Square Goodness of Fit Test was performed on each variable using as the comparison distribution an equal distribution across the five levels of importance. To compensate for inflated Type 1 error rate due to multiple analyses, an alpha of .01 was used. All but nine of the variables reached statistical significance (see Table 1). The low mean ratings that were statistically significant for many of the variables suggested that program directors do not view these experiences as important for entrance in graduate school. Graduate school officials did rate some experiences as important, however. More importance was given for program admittance to four of the 39 variables. Only findings that were both statistically significant at an alpha of .01 or less and had a mean rating of 3.5 or larger (where 5 is the highest rating) were considered as the most important experiences that could lead to graduate school acceptance. Table 1 shows, in order of importance, the four undergraduate research experiences that met the criteria of being rated as very important or extremely important (and statistically significant) for the respondents to make a final decision to accept the students into their program. First was undergraduate research experience that led to a publication in a refereed journal, χ2(4) = 36.95, p <.01; M = 3.99. Thirty percent of the participants considered that experience as very important and 42% viewed it as extremely important for entrance in their graduate programs. The other three undergraduate experiences that met both statistical significance and received a mean score of 3.5 or larger were: publishing of their senior thesis, χ2(4) = 27.73, p <.01; M = 3.84 (35% of the participants rated it as very important and 33% as extremely important); being a first author in a publication in a refereed journal, χ2(4) = 37.80, p <.01; M = 3.83 (rated by 21% as very important and 47% as extremely important), and giving a paper presentation at a regional conference, χ2(3) = 17.29, p <.01; M = 3.54 (rated by 19% as very important and 23% as extremely important). Post-hoc Chi-Squared Tests of Independence were performed on each item to examine the importance respondents from different program types gave to the different undergraduate experiences. That is, were experiences rated differently depending on the type of the raters' program (PhD clinical vs. PhD counseling vs. PsyD clinical)? Although the analysis produced some interesting preliminary findings, these tests could not be interpreted due to variables having numerous cells with less than five observations. The second goal of this study was to assess how graduate program directors viewed student applicants' other activities or experiences obtained in undergraduate school. Table 1 shows that although many of these variables reached statistical significance, none of them met the mean values given to research activities. Only two of the variables, number of statistical and research courses taken and prestige of undergraduate institution, received a mean value of 3.0 or larger. For the "number of statistical and research courses taken," 43% of the respondents rated it as very important or extremely important for admittance into their program. For the variable "prestige of the undergraduate institution from where the student graduated," 31% of the respondents rated it as very important or extremely important. DISCUSSION The purpose of this study was to find information on how graduate program directors rated different types of undergraduate experiences, especially research, when making decisions about students' admittance to their programs. Though research experience for undergraduate students is highly valued when admission decisions are made, apparently not all experiences are equally important. Given the return rate of 30%, the advice given to students should be taken with caution. However, the four highest ranked variables from our respondents do seem to be quite impressive for an undergraduate student to complete. All four variables have one thing in common: publication or presentation of research in a refereed journal or regional conference. Additionally, it seems that being primary author will make you stand out from your undergraduate peers. While research experiences were expected to be most important, we were also interested in obtaining information about how graduate program directors rated other subjective criteria. It appears that these other experiences were rated lower than research publications and presentations. The reason for these ratings is not clear. One possibility is that these experiences are not important because graduate directors expect all applicants to have had these experiences. To answer the question "Are all research experiences rated equally?"the answer appears to be "No." However, a few experiences, research or otherwise, appear to impact graduate admissions decisions. Based on these findings, faculty advisors need to let students know that although publications in undergraduate student journals help improve their research skills, gain insight into the scientific research process, and learn more about the publication process (Thomas et al., 2001), they do not give students an advantage for entrance in competitive graduate programs. Thus, students should aim for those four experiences, however, credit is still given for other research work and experiences. References American Psychological Association (1997). Getting in: A step-by-step plan for gaining admission to graduate school in psychology. Washington, DC: Author. American Psychological Association (2005). Graduate study in psychology, 2006 Edition. Washington, DC: Author. Blashfield, R., Keeley, J., Burgess, D., & Everett, D. (2005, Fall). An apocryphal email exchange about admission to clinical psychology graduate programs. Eye on Psi Chi, 10(1), 16-17; 34. Cashin, J. R., & Landrum, R. E. (1991). Undergraduate students' perceptions of graduate admissions criteria in psychology. Psychological Reports, 69, 1107-1110. Ferrari, J. R., & Appleby, D. C. (2005, Winter). Alumni of Psi Chi: Does "membership have its advantage" on future education/employment? Eye on Psi Chi, 9(2) 34-37. Ferrari, J. R., & Davis, S. F. (2001, Winter). Undergraduate student journal: Perceptions and familiarity by faculty. Eye on Psi Chi, 5(1), 13-17. Ferrari, J. R., & Hemovich, V. B. (2004). Student-based psychology journals: Perceptions by graduate program directors. Teaching of Psychology, 31, 272-275. Keith-Spiegel, P., Tabachnick, B. G., & Spiegel, G. B. (1994). When demand exceeds supply: Second-order criteria used by graduate school selection committees. Teaching of Psychology, 21, 79-81. Landrum, R. E., Jeglum, E. B., & Cashin, J. R. (1994). The decision-making processes of graduate admissions committees in psychology. Journal of Social Behavior and Personality, 9, 239-248. Mayne, T. J., Norcross, J. C., & Sayette, M. A. (2006). Insider's guide to graduate programs in clinical and counseling psychology. New York: Guildford Press. Norcross, J. C., Hanych, J. M., & Terranova, R. D. (1996). Graduate study in psychology: 1992-1993. American Psychologist, 51, 631- 643. Perlman, B., & McCann, L. I. (2005). Undergraduate research experiences in psychology: A national study of courses and curricula. Teaching of Psychology, 32, 5-14. Powell, J. L. (2000, Winter). Creative outlets for student research or what do I do now that my study is completed? Eye on Psi Chi, 4(20), 28-29. Thomas, J., Rewey, K. L., & Davis, S. F. (2002, Winter). Professional development benefits of publishing in the Psi Chi Journal of Undergraduate Research. Eye on Psi Chi, 6(2), 30-35. Walfish, S., & Turner, K. (2006, Summer). Relative weighting of admission variables in developmental psychology doctoral programs. Eye on Psi Chi, 10(4), 20-21. Judith Correa Kaiser, PhD, has been a professor of psychology at St. Ambrose University (IA) in Davenport for 12 years. She received her BS in psychology from Florida State University, MS from University of Central Florida, and her PhD in clinical psychology from Florida State University in 1994. She is currently licensed to practice psychology in Iowa and Illinois. Dr. Correa Kaiser is on the board of the Iowa Psychological Foundation and is the advisor to the Psi Chi chapter at St. Ambrose. She has been nominated twice as St. Ambrose University's "Professor of the Year," winning that honor in 2001. Andrew J. Kaiser, PhD, has been an associate professor of psychology and education at St. Ambrose University in Davenport for 12 years. He received his BS in psychology from the University of Oklahoma and his MS and PhD in clinical psychology from Florida State University. He is a member of Phi Beta Kappa, Psi Chi, and the APA. Dr. Kaiser also acts as St. Ambrose University's ADA compliance officer and was a founder and past-president of the Illinois/Iowa Association for Higher Education and Disability. He also developed the curriculum for St. Ambrose University's Forensic Psychology Degree Program. Nick J. Richardson, BA, is a 2006 graduate of St. Ambrose University's Forensic Psychology Program. He participated in numerous volunteer and research opportunities studying the Scott County Iowa Jail Programs. He won the award for St. Ambrose University's 2006 Outstanding Psychology Major. He hopes to complete his doctoral degree in forensic psychology and is currently working for the Center for Alcohol and Drug Services and the Seventh Judicial Circuit's Department of Correctional Service's residential center. Edward J. Fox, will complete his BS in psychology and English at St. Ambrose University in 2007. He is a veteran of the United States Army, currently working for the Department of Homeland Security, and hopes to complete his doctorate in Neuroscience. ____________________________________________ Winter 2007 issue of Eye on Psi Chi (Vol. 11, No. 2, pp. 22-24), published by Psi Chi, The National Honor Society in Psychology (Chattanooga, TN). Copyright, 2007, Psi Chi, The National Honor Society in Psychology. All rights reserved.