Paul D. Camp Community College Shock-Tucker Assessment of General Education (STAGE) 2011 Degree Graduates vs. 1st Time Students INTERVENTION: The intervention is to assess all general education goals annually and to identify areas of value-added skills and/or knowledge, as well as, areas needed for improvement. INTRODUCTION: The search for a viable measurement of the general education of college students began more than a decade after the Virginia General Assembly mandated the measurement of student outcomes. Often, multiple measures are being used and may include student opinion surveys, analysis of writing samples, and objective tests. One measure of general education that was adopted by many of Virginia’s community colleges is an objective test call Shock-Tucker Assessment of General Education (STAGE). This measurement was first developed by faculty at Mountain Empire Community college in response to many colleges’ displeasure with commercial objective tests. External validation and analysis of STAGE’s reliability turned this test into a collaborative effort among many of the VCCS colleges. Over the past few years the Virginia Community College System (VCCS) has modified its general education goals. As a result, PDCCC has made some slight modifications to its quantitative reasoning and information literacy items. The College’s general education goals include: (1) communication, (2) critical thinking, (3) quantitative reasoning, (4) information literacy, (5) cultural/social, (6) scientific reasoning, and (7) personal development. The STAGE test is used to assess all of the gen-ed objectives. It is, however, only one of the assessments instruments used to analyze general education skills. The purpose of this assessment is to show value-added. Unlike nationally developed tests, STAGE was based on the VCCS definition of general education. METHODOLOGY: Paul D. Camp Community College plan is to engage in a cross-sectional approach to determine the value that is added to degree graduates in all seven general educations areas identified by the Virginia Community College System (VCCS). These include: (1) communication, (2) critical thinking, (3) quantitative reasoning, (4) information literacy, (5) cultural/social, (6) scientific reasoning, and (7) personal development. Value added will be measured through pre and post assessments of the STAGE test. A cross-sectional approach, which permits for college-level flexibility in administration, was selected to demonstrate the value added to PDCCC’s degree graduates while accommodating the non-linear course taking and enrollment patterns, and varied times-to-degree of community college students. The cross- sectional approach will be used to draw inferences about the differences, between pre and post populations assessed and not changes in individual student learning. In spring 2011, Paul D. Camp Community College (PDCCC) administered the STAGE test to the 2011 degree graduates. It began testing new student in the SDV classes fall term as a pre-test group consisting of program-placed students in their first semester of college work. The same STAGE instrument was used as the post-test instrument each spring term to the PDCCC degree-seeking students prior to graduation. The pre-test was administered to first-time students enrolled in SDV classes. The post-test was administered spring term to upcoming degree graduates at the college. The use of a larger absolute value of students (i.e. first-time students versus graduates) will reflect a larger number of first-time students relative to graduates. Independent samples t-tests is planned to compare the pre- and post-test mean scores in order to determine where significant differences might be found. RESULTS: The STAGE instrument contained items pertaining to all of the general education goals for Paul D. Camp Community. For the analysis of each general education area, a number of the Virginia community college met and evaluated each test items and categorized them into the general education areas. Paul D. Camp Community College revised some of the items for quantitative reasoning and information literacy in 2010 to better evaluate each of the sub scores. Using the blueprint created by VCCS faculty members participating in the summer 2008 backward translation workshop, sub scores for each student learning outcomes objective were calculated by summing the number of items mapped to that objective that were answered correctly. Some test items mapped to more than one objective. In addition, an overall score was calculated by adding the number correct on each item reflected in one or more of the sub scores. For the overall score, each item was counted once as opposed to adding each of the seven sub scores together. The results definitely showed value-added in all areas when comparing new 1st time students with graduates. The strongest areas of general education were in personal development (6.48), communication (6.24), and information literacy (6.17). The weakest areas were in quantitative reasoning (3.99) and cultural/social (4.41). The weakest sub-score in quantitative reasoning was objective 6 (Graduates will recognize and communicate the appropriate applications of mathematical and statistical models when solving problems). The weakest sub-score in cultural/social was objective 1 (The graduate will demonstrate the ability to assess the impact that social institutions have on individuals, and culture, past, present, and future). As for the motivational scale, 65.3% agreed and 2.6% disagreed that they gave their best effort on the STAGE test. Paul D. Camp Community College STAGE Testing 2011 Degree Graduates vs. 1st Time Students General Education Goal 2011 Graduates 1st Time Students N=82 N=166 Communication 6.24 5.75 Critical Thinking 4.70 4.39 Quantitative Reasoning 3.99 3.57 Information Literacy 6.17 4.92 Cultural/Social 4.41 3.54 Scientific Reasoning 5.16 4.48 Personal Development 6.48 4.97 Composite Score 37.15 31.62 Test Items 5-14 15-24 25-34 35-44 45-54 55-64 65-74 5-74 Note: Composite Score maximum=70. Each general education score ranges from a minimum of 0 to a maximum of 10. Motivational Score: To what extent do you agree or disagree with the following statement: I gave my very best effort on this test. Motivational Scale 65.3% agree or strongly agree 2.6% disagree or strongly disagree SA=19 25.3% A=30 40.0% Neutral=24 32.0% D=1 1.3% SD=1 1.3% Note: Score used to show test motivation. Objective 1. 2. 3. 4. 5. 6. Mapping of STAGE Test Items to Communication SLO Objectives Communication Goal Test Items % Correct 2011 Graduates The graduate will demonstrate the ability to understand and interpret complex materials. The graduate will demonstrate the ability to assimilate, organize, develop, and present an idea formally and informally. The graduate will demonstrate the ability to use standard English. The graduate will demonstrate the ability to use appropriate verbal and non-verbal responses in interpersonal relations and group discussions. The graduate will demonstrate the ability to use listening skills. The graduate will demonstrate the ability to recognize the role of culture in communication. 7, 9 30.8%, 73.1% 5, 7, 9 85.9%, 30.8%, 73.1% 7, 8, 9, 10, 12 30.8%, 90.9%, 73.1%, 73.7%, 32.1% 6, 9, 11, 13 73.1%, 73.1%, 70.1%, 91.0% 6, 9, 11 73.1%, 73.1%, 70.1% 9, 14 73.1%, 34.6% Objective 1. 2. 3. 4. 5. 6. Mapping of STAGE Test Items to Critical Thinking SLO Objectives Critical Thinking Goal Test Items % Correct 2011 Graduates The graduate will demonstrate the ability to discriminate among degrees of credibility, accuracy, and reliability of inferences drawn from given data. The graduate will demonstrate the ability to recognize parallels, assumptions, or presuppositions in any given source of information. The graduate will demonstrate the ability to evaluate the strengths and relevance of arguments on a particular question or issue. The graduate will demonstrate the ability to weigh evidence and decide if generalizations or conclusions based on the given data are warranted. The graduate will demonstrate the ability to determine whether certain conclusions or consequences are supported by the information provided. The graduate will demonstrate the ability to use problem solving skills. 18 16, 22, 24 17, 22 73.3% 47.4%, 80.3%, 14.5% 57.9%, 80.3% 17, 19, 21 57.9%, 52.6%, 13.5% 20, 21, 23 66.7%, 13.5%, 36.0% 15, 23 67.1%, 36.0% Mapping of STAGE Test Items to Quantitative Reasoning SLO Objectives Objective Quantitative Reasoning Goal Test Items % Correct 2011 Graduates 1. 2. 3. 4. 5. 6. 7. The graduate will use logical and mathematical reasoning within the context of various disciplines. The graduate will interpret and use mathematical formulas when instructed. The graduate will interpret mathematical models such as graphs, tables, and schematics, and draw inferences from them when they appear on test. The graduate will use arithmetic, algebraic, geometric, and statistical models in problems solving. The graduate will estimate and consider answers to mathematical problems in order to determine reasonableness. Graduates will recognize and communicate the appropriate applications of mathematical and statistical models when solving problems. Graduates will represent mathematical information numerically, symbolically, and visually using graphs and charts when given standardized test. 29 39.2% 25, 27 34.7%, 50.7% 30 66.7% 26,31,34 44.6%, 20.8%, 56.0% 28 42.7% 32 20.0% 33 65.3% Objective 1. 2. 3. 4. 5. Mapping of STAGE Test Items to Information Literacy SLO Objectives Information Literacy Goal Test Items % Correct 2011 Graduates The information literate student determines the nature and extent of the information needed. The information literate student accesses needed information efficiently and effectively. The information literate student evaluates information and its sources critically, and incorporates selected information into his or her knowledge base and value system. The information literate student, individually or as a member of a group, uses information effectively to accomplish a specific purpose. The information literate student understands many of the economic, legal, and social issues surrounding the use of information resources and accesses and uses information ethically and legally. 42 35, 37, 41, 42, 43, 44 36 73.3% 54.7%, 73.3%, 56.8%, 73.3%, 57.3%, 61.3% 60.0% 39 61.3% 38, 40 89.3%, 88.0% Mapping of STAGE Test Items to Cultural and Social Understanding SLO Objectives Objective Cultural and Social Understanding Goal Test Items % Correct 2011 Graduates 1. 2. 3. 4. 5. The graduate will demonstrate the ability to assess the impact that social institutions have on individuals and culture—past, present, and future. The graduate will demonstrate the ability to describe their own as well as others’ personal ethical systems and values within social institutions. The graduate will demonstrate the ability to recognize the impact that arts and humanities have upon individuals and cultures. The graduate will demonstrate the ability to recognize the role of language in social and cultural contexts. The graduate will demonstrate the ability to recognize the interdependence of distinctive world-wide social, economic, geo-political, and cultural systems. 45, 46,47, 48, 49, 52, 53, 54 53.3%, 30.7%, 29.3%, 82.7%, 64.9%, 58.7%, 5.3%, 28.4% 49,54 64.9%, 28.4% 49, 50 64.9%, 62.2% 51, 45, 47, 48, 51, 52, 53 69.3% 53.3%, 29.3%, 82.7%, 69.3%, 58.7%, 6.7% Mapping of STAGE Test Items to Scientific Reasoning SLO Objectives Objective 1. 2. 3. 4. 5. Scientific Reasoning Goal The graduate will demonstrate the ability to generate an empirically evidenced and logical argument. The graduate will demonstrate the ability to distinguish a scientific argument from a nonscientific argument. The graduate will demonstrate the ability to reason by deduction, induction and analogy. The graduate will demonstrate the ability to distinguish between causal and correlational relationships. The graduate will demonstrate the ability to recognize methods of inquiry that lead to scientific knowledge. Test Items % Correct 2011 Graduates 59 28.0% 61, 63 62.2%, 74.7% 56, 58, 60, 64 48.0%, 57.3%, 28.0%, 77.3% 62 70.7% 55, 57 88.0%, 30.7% Mapping of STAGE Test Items to Personal Development SLO Objectives Objective Personal Development Goal Test Items % Correct 2011 Graduates 1. 2. The graduate will demonstrate the ability to develop and/or refine personal wellness goals. The graduate will demonstrate the ability to develop and/or enhance the knowledge, skills, and understanding to make informed academic, social, personal, career, and interpersonal decisions. 68, 69, 70 68.9%, 57.3%, 64.0% 65, 66, 67, 71, 72, 73, 74 89.3%, 49.3%, 81.3%, 67.1%, 82.4%, 81.1%, 72.0%