MPI Mission Perception Inventory Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment North Carolina State University Undergraduate Assessment Symposium Aligning Pedagogy, Curriculum & Assessment April 24-26, 2009 Ellen Boylan, Ph.D. and Sister Jane Wakahiu, LSOSF • Marywood University 1 What’s in a mission? 2 scope of research Purpose Develop an instrument to measure student perception of institutional mission. Test instrument reliability. Uncover constructs (factors). Observe constructs longitudinally. 3 GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS scope of research Purpose Develop an instrument Research Questions 1. Is the Mission Perception Inventory (MPI) a valid and reliable reliable measure of student perception of institutional mission? Test instrument reliability. Uncover constructs.2. What are the factors in the MPI? Observe constructs longitudinally. 3. Do the factors recur in repeated administrations of the revised MPI? 4. Are the factors equally reliable over time? 4 leaders of public and private institutions alike are thinking about spirituality these days, as the data suggest that's what their students are thinking about, too (Inside Higher Ed, 2009). background connection between institutional programs and student learning environment (Pascarella, 2001). There is strong …institutions influence levels of engagement on campus as a result of structural features, programs, policies, and organizational culture (Kuh et al., 2005). 5 design Select the National Survey of Student Engagement (NSSE) as the vehicle for inserting research questions. Assemble a NSSE consortium to jointly engage in research to explore student perception of mission. Develop question items; administer them to the consortium as a NSSE attachment. Test the questions: reliability, factor analysis, and correlation analysis. Repeat annually. 6 GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS developing the question items… Concepts common to mission statement of consortium colleges, like this one, were drawn out, compared, and distilled into 20 questions. 1. 2. 3. 4. Conduct a qualitative analysis of mission statements Distill common constructs Draft question items based on mission statement constructs Vet with participating institutions, experts, prospective subjects 7 Concepts become question items... Leadership Community/ Service Professional Development Spirituality/ Ethical Develop. Prepare for leadership Sense of community Develop professionally Develop ethically Respect & understanding Develop Acedemically Critical Thinking Sense of Community Professional Growth Personal Growth Research & Expertise Scholarship Independent Learning Public Service Specialized Expertise Personal Advisement Diversity Truth & Social Justice Care and Respect Academic Excellence Develop intellectually Rigorous & Engaging Compassionate Service Other Develop Intellectually Prepare for Leadership Lives of Service Prepare for Prof. Careers Develop Spirit & Heart Values-Based Education Scholarship Analysis & Comunication Sense of Community Diciplines & Professions Morally Responsible Freedom & Diversity Educational Excellence Responsibility as Leaders Care & Dignity for Community Career Development Ethical Presence Historical Remembrance Service to Society Professional Competence Leadership Skills Involvment in Community Develop Whole Person Cultural & Social Growth Develop Intellectually Commitment Compassion & Empathy Develop Professionally Promote Ethics & Spirituality Human Dignity Truth & Justice Diversity & Dignity Personal Growth Diversity Comitment to Teach & Learn Academic Excellence Lifelong Learning Sense of Community Intellectual Inquiry Responsibility for Decisions Commitment to Service Career Preparation GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS Make Connactions 8 A2. Every institution has a mission statement. We would like to know how familiar you are with your school's mission. Please indicate your agreement with each of the following statements: Strongly agree Agree Neither agree/ disagree Disagree Strongly disagree a. The mission of this institution is widely understood by students. b. Ethical and spiritual development of students is an important part of the mission at this institution. This institution offers opportunities for volunteering and community service. Social and personal development of students is an important part of the mission at this institution. This institution offers opportunities for developing leadership skills. The heritage of the founding religious community of this institution is evident here. The faculty, staff, and students here are respectful of people of different religions. The faculty, staff, and students here are respectful of people of different races and cultures. c. d. e. f. g. h. i. People of different sexual orientations are accepted socially here. j. Students feel free to express their individual spirituality here. k. The environment here encourages students to develop an appreciation of diversity. l. At this institution, there are opportunities for students to strengthen their religious commitment. m. The professors at this institution discuss the ethical implications of what is being studied. See consortia questions by year: http://nsse.iub.edu/html/consortia-list_2009.cfm 9 administration The NSSE survey with attached Mission Perception Inventory (MPI) questions has been administered 155 times to more than 44,198 first-year and senior students at 112 unique institutions across the United States every year since 2004. 10 GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS Reliability analysis of the mission questions (20 items) is performed. Low-loading items kicked out. • A Mission Perception Inventory (MPI) is produced (17 to 19 items). • 11 Factor analysis of MPI is conducted each year. • By year, three or four subscales are produced: •Sense of mission. •Respect for diversity. •Individual values. •Religious practice/Spirituality. • 12 derivation of the MPI and subscales Sense Of Mission Administer Consortium mission questions (20 items) Mission Perception Inventory Respect for Diversity (MPI) (~19 items) Individual Values Spiritual Practice 13 • The mission of this institution is widely understood by students. • Social and personal development is an important part of the mission. • Ethical and spiritual development of students is important. This institution offers opportunities for volunteering and • This institution offers opportunities for developing • There are opportunities for students to their religious commitment. • • • • • sense of mission (10 items = .90) community service. leadership skills. strengthen This institution’s religious heritage is evident. Professors here discuss the ethical implications of what is being studied. As a result of my experience here, I am more aware of my own personal values. The mission of this institution is reflected in course offerings. 14 respect for diversity (5 items = .878) The faculty, staff, and students here… • respect different religions respect different races and cultures ……………………………………………………………………………… • • Students feel free to express individual spirituality. • Different sexual orientations are accepted. • The environment encourages appreciation of diversity. 15 Consortia institutions and respondents by year Respondents Institutions First Year Senior 2004 15 2,000 1,827 2005 16 1,279 1,332 2006 24 2,684 2,854 2007 36 4,533 4,331 2008 35 4520 5063 Indepen’t 2008 29 6805 6970 155* 21,821 22,377 Catholic *(112 unique institutions) 16 2004 2005 2006 2007 2008 Number of items in factors Sense of Mission 8 10 10 10 10 Respect for Diversity 5 5 5 5 5 Individual Actions 2 0 0 0 0 Religious Practice/Spirituality 2 3 3 2 2 Mission Perception Inventory 17 18 18 19 19 17 (MPI) NSSE 2008 Mission Perception Inventory Report Mission Engagement Consortium for Independent Colleges My University- Sample Sense of Mission Reliability and factor analysis of the instrument based on results from administrations of consortium questions have consistently produced a highly reliable scale measure called the Mission Perception Inventory (MPI)*. Further analysis reveals three highly correlated and reliable subscales within the MPI: First Year Mean 0.00 Median 0.00 25%ile 0.00 75%ile 0.00 Sense of Mission ( 11 items) Respect for Diversity (5 items) Spirituality (2 items) (Cronbach’s alpha=__) 4.6 Also presented are mean comparisons on another scale, NSSE Beliefs and Values (NSSE/BV). Senior YearThe Mean a priori, 0.00 the Consortium administrator created the NSSE/BV scale for this study by a process of selecting, Median 0.00 question items from the NSSE instrument that seemed most related to mission and then grouping them. 25%ile 0.00 Scale mean comparisons between My University and Consortium schools are presented in the0.00 bar graphs. 75%ile The graphs compare My University first year and senior means compared to Consortium group means on the (out of to x) the study’s five scales or benchmarks. The mean scores are shown on the vertical or “y” axisRank according First Year:were rated on Likert scale range used for items in that scale. Sense of Mission and Respect for Diversity items Senior Year: a scale from 1 – 5 (1=Strongly disagree, 5=Strongly agree). In the column to the left of each graph, the rank (high to low) and semi-interquartile range are given for both first year and senior scale means. Note carefully that the y-axis interval is 0.5 for some scales, and 0.1 for others. In the columns under and/or to the right of the graph, the items contained in that scale are listed. When comparing first year and senior mean scores, recall that the results of the NSSE 2008 produce a snapshot in time of student responses. Results do not indicate growth or increase in scale means from first to senior years, but simply a comparison of those cohort classes at one moment. Sense of Mission Score Mission Perception Inventory Report In spring 2008, My University participated in the National Survey of Student Engagement (NSSE) Mission Engagement Consortium for Independent Colleges, and administered additional questions on mission the 2008 MissiontoPerception Inventory Report survey’s first year and senior student respondents. Of ___ total survey respondents at My University, My Universitythere - Sample were ___ first year students and ___ seniors. 4.4 This institution offers opportunities developing leadership skills. for As a result of your experience here, you are more aware of your own personal values. 4.0 The heritage of the founding religious community of this institution is evident here. 3.8 First Year Senior My University 4.69 4.56 Consortium 4.02 3.97 The mission of this institution is widely understood by students. The mission of this institution is reflected in its course offerings Social and personal development of students is an important part of the mission at this institution. At this institution, there are opportunities for students to strengthen their religious commitment. Ethical and spiritual development of students is an important part of the mission at this institution. The faculty at this institution discusses the ethical implications of what is being studied. As a result of my experience here, I am more aware of my own personal values. 2.0 (Cronbach’s alpha=__) Within the past week, have you participated in a religious service? Within the past week, have you spent time in private prayer or meditation? Respect for Diversity 1.5 First Year Mean 0.00 Median 0.00 25%ile 0.00 75%ile 0.00 1.0 0.5 0.0 First Year Senior My University 1.96 1.92 Consortium 2.07 2.08 2008 Mission Perception Inventory Report Senior Year Mean 0.00 Median 0.00 25%ile 0.00 75%ile 0.00 Rank (out of x) First Year: Senior Year: (Cronbach’s alpha=__) The faculty, staff, and students here are respectful of people of different religions. 4.1 Respect for Diversity Score Rank (out of “x” schools) First Year: ___ Senior Year: Religious Practice Score Senior Year Mean 0.00 Median 0.00 25%ile 0.00 75%ile 0.00 for 4.2 Spirituality First Year Mean 0.00 Median 0.00 25%ile 0.00 75%ile 0.00 This institution offers opportunities volunteering and community service. The faculty, staff, and students here are respectful of people of different races and cultures. 4.0 3.9 Students feel free to express their individual spirituality here. 3.8 3.7 People of different sexual orientations are accepted socially here. 3.6 3.5 First Year Senior My University 3.92 3.60 Consortium 3.98 3.87 The environment here encourages students to develop an appreciation of diversity. 18 GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS 2004 2005 2006 2007 2008 Catholic 2008 Indept cronbach cronbach cronbach cronbach cronbach cronbach Sense of Mission .87 .88 .88 .90 .91 .88 Respect for Diversity .85 .84 .87 .86 .88 .87 Individual Actions .67 n/a n/a n/a n/a n/a Religious Practice/Spirituality .62 .55 .54 .62 .62 .64 MPI scale .88 .89 .90 .91 .91 .90 Subscales 19 Teagle Foundation Research Continues • Develop the Mission Engagement Index. • • • • • • Obtain NSSE 2008 consortia data Test using reliability analysis to produce factors Distribute MPI reports Compare consortia results Conduct regression analysis to create the new index Produce Mission Engagement Index (MEI) Reports for consortia institutions. 20 GRANTS IN HIGHER EDUCATION FOR IMPROVED ASSESSMENT METHODS • Dependent variable (Institution score) Mission Perception Inventory • Sense of Mission scale • Respect for Diversity scale • • Independent variables (need 15 cases per)* • Selectivity • Enrollment • Urbanicity • Resident % • Female % • Another? 15 institutions 30 institutions 45 institutions 60 institutions 75 institutions 90 institutions *…a recommended ratio of subjects to IVs of at least 15 to 1 will provide a reliable regression equation (Stevens,21 1992). Selecting IVs for regression analysis Institution region Carnegie Class % resident % female % ethnic/ cau UG enroll % Parttime Setting % Accept a b c d e North U Master's 39 61 95 2119 25 Suburb 74 Midwest U Master's 33 67 70 928 40 Suburb 57 South National U 32 68 39 4130 42 Suburb 54 South Bacc.. Coll 43 57 76 1011 21 Rural 66 Midwest U Master's 43 57 70 1692 23 Suburb 85 f g h Midwest Bacc. Coll 42 58 89 978 83 Urban 70 Midwest U Master's 31 69 78 2985 37 Suburb 93 West U Master's 32 68 23 1096 60 Urban 94 i j North Bacc. Coll 92 58 62 8568 16 Suburb 42 Midwest U Master's 28 72 91 1502 37 Urban 79 l North U Master's 24 76 79 1320 815 Suburb 60 22 participating institutions by region West 9% Midwest 31% South 17% Is there sufficient variability? North 43% 2008 = 2 consortia, 54 institutions 23 “urbanicity” of participating institutions 2008 Urban 31% Is there sufficient variability? Rural 15% Suburban 54% 24 considering elegance Consistency of factors is affirmed. Institution sizes reasonably varied. Data sufficient to conduct analysis. Variables, initially selected βeta juice on an a priori basis, tested. Variability investigated. Useful. 25 MPI Scales: Comparison of means by institution type Scale Sense of Mission Respect for Diversity Spiritual Practice MPI NSSE Beliefs & Values Mean Score Independent Catholic 3.77 4.04 4.03 4.00 1.37 1.46 3.46 3.63 2.57 2.62 t(df) -2.637(14.356) 0.330(40.000) -1.310(14.856) -2.296(14.756) -0.882(15.876) 2-tailed sig .019* NS NS .037* NS *p≤.05 26 Correlation of regression variables with MPI mean Correlations religious_affliation usn_setting usn_setting pct_residence mpi_mean -.203 -.258 .429 Sig. (2-tailed) NS NS .005 N 42 42 42 Pearson Correlation .590 Sig. (2-tailed) .000 .001 42 42 Pearson Correlation Sig. (2-tailed) N -.484 ** Pearson Correlation N pct_residence ** ** -.456 ** .002 42 **. Correlation is significant at the 0.01 level (2-tailed). 27 Deriving coefficients for the Mission Engagement Index (MEI) Model R Model Summary Adjusted R Std. Error of R Square Square the Estimate 1 .484a .234 .215 b 2 .591 .349 .315 a. Predictors: usn_setting b. Predictors: usn_setting, religious_affliation .163 .153 28 Correlation of regression variables with Mission scale mean Correlations usn_setting religious_affliation usn_setting Pearson Correlation mission_mean -.258 Sig. (2-tailed) NS NS .001 N 42 42 42 Pearson Correlation N Pearson Correlation Sig. (2-tailed) N .590 ** .486 ** -.203 Sig. (2-tailed) pct_residence pct_residence -.429 ** .000 .005 42 42 -.268 .086 42 **. Correlation is significant at the 0.01 level (2-tailed). 29 Predictive Equation* Institution Predicted MPI Score = (B1)*(Value of USN_setting)+ (-0.414)*(1, 2, or 3) (B2)*(Value of Religious Affiliation)+ (0.345)*(1 or 2) Constant (3.687) *Mortenson, T. (1997). Actual Vs Predicted institutional graduation rates for 1100 Colleges and universities. Opportunity, 58. 30 Mock Mission Engagement Index Report The MEI will show an institution’s actual versus predicted scores on mission constructs. Progress on mission effectiveness can be assessed by comparing MEI outcomes to institutional goals. Sense of Mission Respect for Diversity Values and Beliefs Mission Perception Inventory 31 References Inside Higher Education (2009). Spiritual accountability. Retrieved from http://www.insidehighered.com/layout/set/pri...assessment/01/02/2007 /News Kuh, D. G., Kinzie, J., Schuh, J. H., and Whitt, E. J. (2005). Never let it rest: lessons about student success from high-performing colleges and universities. Change, 37(4), 44-51. Mortenson, T. (1997). Actual Vs predicted institutional graduation rates for 1100 colleges and universities. Opportunity, 58. Pacarella, E. T. (2001). Identifying excellence in undergraduate education. Change, 33(3), 18-27. Stevens, J. P. (1992). Applied multivariate statistics for the social sciences (2nd edition). Hillsdale, New Jersey: Erlbaum. 32 Chestnut Hill College Philadelphia NorthPA Bennington College Christ the King Seminary East Aurora SouthNY Berry College Walsh University Our Lady of Corpus Christi Christendom College Front Royal SouthVA Birmingham-Southern College Our Lady of Holy Cross College Bennington College Christian Brothers University Memphis NorthTN Clark University Our Lady of the Lake College Clarke College Dubuque The United NorthIA College Clarkson University of San Antonio Berry College of the Lake University College of Mount Saint Joseph Cincinnati Thomas OH Our Lady NorthAquinas Daemen College College Birmingham-Southern College Collegeof Mount Saint Vincent Riverdale NY Presentation College SouthMore Faulkner University Thomas College Clark University College of New Rochelle New Rochelle NY Providence College SouthMore Florida Hosp College of Health SciThomas College ofRosary Liberal Queen of the Holy CollegeAr Our Lady of University Corpus Christi College of Notre Dame of MarylandManchester College Baltimore MD Midwest Clarkson Quincy University Trinity University Our Lady of Holy Cross College College of Saint Benedict NorthMN Regis College Manhattanville College St. Joseph Trocaire Daemen College College College of SaintDonnelly Catherine St. Paul Kansas CityNorthMN Our Lady of the Lake College College KS University Marlboro College University of Regis Dallas Faulkner University College of SaintDuquesne ElizabethUniversityMonmouth of the Holy Spirit PA College Our Lady of the Lake University University MorristownPittsburgh NorthNJ University of Rivier Dayton College of Saint Joseph in Vermont Rutland VT D'Youville College Buffalo NY Florida Hosp College of Health Sci Midwest Rockhurst Mountain State University Presentation College University University of Detroit Mercy College of SaintEdgewood Mary Omaha NE College Madison WI West Rosemont College of the Holy Child Naropa University Providence College Manchester College College of Saint Rose University of Great Falls Albany NY Elms College Chicopee MA Sacred Heart School of Theology North Paul Smiths College Queen of the Holy College RosaryCollege Collegeof Saint Scholastica Manhattanville University of Mary Duluth MN Emmanuel College Springfield College Boston MA Heart University Sacred North Quincy University College of SaintFairfield ThomasUniversity More Fort Worth TX Fairfield CT Ambrose University of Notre Dame du Lac Marlboro College Saint University South Stetson University Regis College College NJ Anselm College College of SantaFelician Fe Santa Fe Lodi NM University of Portland Saint South Webster University Worldwide Monmouth University College of the Holy Fontbonne Louis MO Anthony College of Nursing Regis University Cross University WorcesterSt.University MA Sacramento North of Saint Yeshiva University Mountain State University Fordham University Bronx NY Bernard's Creighton University Omaha NE School of Theology a Rivier College University of Saint Saint Francis Franciscan School of Theology Berkeley CA Bonaventure University De Paul University Chicago University IL of Saint Naropa University Rockhurst University Saint Francis Franciscan University of Steubenville Steubenville OH Catharine College University Center Valley PA Saint Rosemont College of the DeSales Holy Child Jesus University of Saint Mary Paul Smiths College Gannon University Erie PA Charles Saint Borromeo Seminary Divine Word College Epworth IA Sacred Heart School University of Saint Saint Mary of the Lake Springfield Collegeof Theology Georgetown University Washington DC Edward's University Dominican College Orangeburg NY University of Saint Thomas Sacred Heart University Dominican House Georgian Court University Lakewood NJ Elizabeth College of Nursing of Studies Washington DC Saint Stetson University Our Lady of Corpus Christi University of Saint Thomas Saint Ambrose University Dominican School Gonzaga University Spokane WA Elizabeth Saint School of Nursing of Philosophy and Theology Berkeley CA Webster University Our Lady of Holy Cross College Gwynedd Gwynedd-Mercy College Valley PA University of San Diego Saint Francis College Saint Anselm College Worldwide Dominican University River Forest IL Our Lady of the Lake College Hilbertof College Hamburg NY Francis Medical Center College Yeshiva University University San Francisco Saint Anthony College of Nursing Dominican University California San Rafael CAof Saint Our Lady of the Lake University Cromwell Holy Apostles College & Seminary CT Saint Francis University University of Scranton Saint Bernard's School of Theology and Ministry Presentation College Holy Cross College Notre Dame IN Saint Gregory's University University of the Incarnate Word Saint Bonaventure University Providence College Holy Family University Philadelphia PA Ursuline College Saint Catharine College Queen of the Holy Rosary College Holy Names University Oakland CA Villa Maria College of Buffalo Quincy University Saint Charles Borromeo Seminary Immaculata University Immaculata PA Villanova University Regis College Iona College New Rochelle NY Saint Edward's University Regis University Viterbo University Jesuit School of Theology Berkeley CA Saint Elizabeth College of Nursing Rivier College Walsh University John Carroll University Cleveland OH Saint Elizabeth School of Nursing John Paul the Great CatholicRockhurst University University San Diego CA Washington Theological Union Saint Francis College Rosemont College of the Holy Child Jesus King's College Wilkes-Barre PA Weston Jesuit School of Theology Sacred Heart School of TheologyPittsburgh Saint Francis Medical Center College of La Nursing Roche College PA University Wheeling Jesuit Sacred Heart University Saint Francis University Wyoming Catholic College Saint Ambrose University Saint Gregory's University Xavier University Saint Anselm College Saint Anthony College of Nursing Xavier University of Louisiana Discussion 33 Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment Ellen Boylan, Ph.D. Director of Institutional Research and Assessment eboylan@marywood.edu Sister Jane Wakahiu, LSOSF, MA Graduate Assistant jwakahiu@marywood.edu Office of Planning and Institutional Research http://cwis.marywood.edu/instresearch/activity.stm Marywood University Supported by a grant from http://www.teaglefoundation.org/grantmaking/grantees/assessmentmethods.aspx 34 35 MPI Mission Perception Inventory Assessing Institutional Effectiveness with the Mission Perception Inventory (MPI): Linking Mission Goals and Learning Environment North Carolina State University Undergraduate Assessment Symposium Aligning Pedagogy, Curriculum & Assessment April 24-26, 2009 Ellen Boylan, Ph.D. and Sister Jane Wakahiu, LSOSF • Marywood University 36