Assessment User Guide for Colleges and Universities January 2015 Prepared for Smarter Balanced Assessment Consortium by: Contents Introduction ................................................................................................................................1 Purpose of the Guide...............................................................................................................3 Coordination with the PESC: P20W Education Standards Council Transcript Schema Working Group ......................................................................................................................................3 States Implementing the Grade 11 Summative Assessment in 2014-15 ...................................4 Timeline ..................................................................................................................................4 Assessment Score Definitions ..................................................................................................5 Assessment Data Importing and Use Recommendations .............................................................7 Course Placement ...................................................................................................................8 Admissions and/or Scholarship Decision Rubric .......................................................................8 Retention Analysis and Predictive Modeling ............................................................................8 New Test Score Implementation Guidance by Technology Provider ............................................9 Campus Management .............................................................................................................9 Common Application Group ..................................................................................................11 Credentials eSCRIP-SAFE .......................................................................................................11 Ellucian ..................................................................................................................................12 Hobsons CRM (Radius) ..........................................................................................................12 Oracle (PeopleSoft) ...............................................................................................................13 Populi ....................................................................................................................................15 Three Rivers Systems (CAMS) ................................................................................................16 Introduction The Smarter Balanced Assessment Consortium is one of two multi-state consortia that have built new assessment systems aligned to the Common Core State Standards. The Common Core is “. . . a set of high-quality academic standards in mathematics and English language arts/literacy (ELA). . . . The standards were created to ensure that all students graduate from high school with the skills and knowledge necessary to succeed in college, career, and life, regardless of where they live. Forty-three states, the District of Columbia, four territories, and the Department of Defense Education Activity (DoDEA) have voluntarily adopted and are moving forward with the Common Core.” (http://www.corestandards.org/about-the-standards/) The Smarter Balanced Assessment Consortium is composed of 18 states and the U.S. Virgin Islands that have worked together to develop a comprehensive assessment system aligned to the Common Core State Standards (Iowa, North Carolina, and Wyoming participate as affiliate members but do not yet offer the assessments). The assessments were field tested with 4.2 million students in spring 2015 and operational testing will begin in spring 2015. End-of-year summative assessments will replace existing tests used in Grades 3-8 and 11 for state and federal accountability reporting. Optional interim assessments are available to inform school decision-making throughout the year. Finally, the assessment system also includes a digital library of instructional and professional development resources to help teachers implement the formative assessment process in their classrooms. For more information on the assessment system, visit www.smarterbalanced.org. Most member states will offer the Smarter Balanced summative assessment in Grade 11 (see list on page 4). The Grade 11 assessments will provide information to students and families, schools, and colleges and universities about student readiness for entry-level, credit-bearing courses for English Language Arts/Literacy (ELA) and mathematics. To provide students, parents and teachers with clear and consistent signals about student progress toward college readiness while respecting the diversity of higher education, representatives from K-12 and Assessment User Guide for Colleges and Universities 1 higher education in member states crafted a College Content-Readiness Policy that defines content-readiness in English and math and includes a policy framework for how colleges and universities in each state may use Smarter Balanced score bands, or “achievement levels,” as part of a multiple measures approach. See http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2014/06/College-Content-Readiness-Policy.pdf for the complete College Content- readiness Policy. (Note: Because college readiness is a broad construct that includes an array of academic and non-academic knowledge and skills, Smarter Balanced uses the term “content-readiness” to distinguish the narrower parameters of its assessments in English language arts and mathematics.) Colleges and universities can use Smarter Balanced scores as an early indicator of readiness and for course placement by signing a participating college agreement. Participating colleges are defined as follows: Colleges and universities with signed Smarter Balanced participation agreements will be so identified in a list of institutions that Smarter Balanced will maintain and make available through its web site to high schools, parents, and students. The signed agreement indicates a higher education institution will follow the Smarter Balanced college content-readiness policy, using Smarter Balanced grade 11 achievement levels and indicators of 12th grade performance as the basis for exempting newly admitted students from developmental or remedial courses. As specified in the college content-readiness policy, higher education institutions within each state will consult with each other and with state K–12 officials in determining appropriate indicators of grade 12 performance. The college content-readiness policy applies only to students who matriculate directly to higher education and does not preclude the use of other measures for exemption from remediation as part of a multiple-measures approach. Colleges and universities also may choose to use Smarter Balanced score information as part of the admission and/or scholarship decision rubrics and for longitudinal retention analysis and predictive modeling. Assessment User Guide for Colleges and Universities 2 Purpose of the Guide This guide is designed to serve two purposes: 1) provide practice recommendations on how to use the new assessment data; and 2) outline specific guidance shared by technology solution providers for how to incorporate the new test scores into the various technology systems. Each state in the Consortium retains control of its testing data and will make its own arrangement for sharing that information with colleges and universities. However, the data elements will be the same, regardless of source. Colleges and universities should prepare as soon as possible to receive and use these data. For many institutions, this preparation will include updating test score validation tables in one if not several systems. For example, if an institution uses an electronic transcript solution, an integrated or stand-alone application for admission solution (which may or may not also be the CRM), and a data warehouse/reporting solution all in addition to the primary SIS/ERP, all of these systems will need to be updated in some manner to be able to receive and use the new assessment scores. In addition, if any of these systems share test score data with each other, the new assessment data will need to be mapped and built into the data sharing solution(s) for these systems. Coordination with the PESC: P20W Education Standards Council Transcript Schema Working Group The AACRAO SPEEDE Committee and the Smarter Balanced Assessment team are working with PESC, the coordinating body for all electronic standards in education. Their goal is to ensure that by the time the first test is administered all of the test score information will be incorporated into the PESC transcript schema. For more information on PESC and the transcript schema, visit the PESC website at http://www.pesc.org/. 3 Assessment User Guide for Colleges and Universities States Implementing the Grade 11 Summative Assessment in 2014-15 California Connecticut Delaware Hawaii Idaho Maine Michigan Montana New Hampshire North Dakota Oregon South Dakota Vermont Washington* West Virginia* * As of December 2014, the public higher education systems in these states have announced plans to use Smarter Balanced scores as outlined in the College Content-Readiness Policy. Smarter Balanced anticipates that one or more additional states may make similar announcements in early 2015. Timeline Colleges and universities should be prepared to receive and use these data as soon as summer 2015. The Smarter Balanced assessment will be offered to high school juniors starting midspring 2015. These students may consider dual enrollment in summer or fall 2015 college-level courses; many will submit fall 2016 applications with transcripts that may contain these data. Initially, states are concentrating on transmitting score information from each state’s K-12 education agency to that same state’s participating public colleges and universities. Over time, states will expand these systems to accommodate in-state independent institutions as well as participating public and independent institutions in other states. 4 Assessment User Guide for Colleges and Universities Assessment Score Definitions Achievement Levels Achievement levels are groupings of overall scale scores in ELA and math into four categories of performance. At grade 11, the levels have the following meaning: Level 1: Has not met the performance standard and needs substantial improvement to demonstrate the knowledge and skills in [math/ELA] needed to be ready for success in college coursework or career education after high school. Level 2: Has nearly met the performance standard and requires further development to demonstrate the knowledge and skills in [math/ELA] needed to be ready for success in college coursework or career education after high school. Level 3: Has met the performance standard and demonstrates progress toward mastery of the knowledge and skills in [math/ELA] needed to be ready for success in college coursework or career education after high school. At participating colleges, student is conditionally exempt from developmental coursework, contingent on successful completion of Grade 12 requirements established by colleges and universities. Level 4: Has exceeded the performance standard and demonstrates the knowledge and skills in [math/ELA] needed to be ready for success in college coursework or career education after high school. At participating colleges, is exempt from developmental coursework. Colleges may establish Grade 12 requirements for students to retain this exemption. (Note that these achievement level descriptors are subject to further review and revision and that states may alter the language used in score reports. Nonetheless, these descriptions provide an accurate general depiction of the four achievement levels.) The academic content described by these achievement levels and the threshold scores associated with each level were determined by expert panels composed primarily of K-12 teachers and higher education faculty. These expert panels sought to align their judgments to the definitions of the achievement levels. For more information, see Assessment User Guide for Colleges and Universities 5 http://www.smarterbalanced.org/achievement-levels/. The Grade 11 threshold scores for the four achievement levels are as follows: English Language Arts Level 1: Level 2: Level 3: Level 4: There is no lower bound score for Level 1 2493 2583 2682 Mathematics Level 1: Level 2: Level 3: Level 4: There is no lower bound score for Level 1 2543 2628 2718 Detailed Achievement Level Descriptors that define the level of knowledge and skill students must display at each achievement level are available at http://www.smarterbalanced.org/achievement-levels/. Claim Levels These are performance categories for a particular content area with either ELA or math (referred to as a “claim”). ELA includes the following claims: Reading Writing Listening Research Math Claims: Concepts & Procedures Problem Solving and Modeling & Data Analysis Communicating Reasoning 6 Because of the larger standard errors associated with these sub-scores, student reports include 3 achievement levels rather than the sub-score itself. Those levels are: Assessment User Guide for Colleges and Universities below standard (more than one standard error of measurement below the overall level 3 threshold score); at/near standard (within one standard error of measurement of the overall level 3 threshold score), and; above standard (more than one standard error of measurement above the level 3 threshold score). Assessment Data Importing and Use Recommendations Ideally, colleges and universities will import all of the assessment test results and not just the aggregate “Achievement Level” which represents a student’s readiness for college level creditbearing courses. The “Claim” results are included in the data file and can be used to further disaggregate retention analyses and predictive student success models. Test Results of Interest to Higher Education FieldName Width (chars) Data Element Description Acceptable Values Data Type One or more printable ASCII xsd:token characters TestName 250 Name of the test. TestSubject 10 Subject of the test. E.g. ELA or One or more printable ASCII xsd:token MATH. characters AcademicYear 4 Academic year of the assessment 1900 <= YYYY <= 9999 xsd:integer StandardError 8 Standard error of the measure. Float, null allowed xsd:float The level of achievement represented by the Scale Score on a scale of 1 to 4. 1-4 xsd:integer 1-3 xsd:integer AchievementLevel 1 Achievement level on a scale of 1 to 3 for sub-areas. ELA includes: ClaimLevel 1 Reading Writing Listening Research Math includes: Concepts & Procedures Problem Solving and Modeling & Data Analysis Communicating Reasoning Assessment User Guide for Colleges and Universities 7 Course Placement The College Content-Readiness Policy describes parameters for use of the assessments to exempt students from developmental coursework. Use of the Smarter Balanced assessments may eliminate the need for placement testing for many students, but participating colleges may continue to administer placement tests to determine the most appropriate course placement for students who are ready for credit-bearing courses or for students whose Smarter Balanced results suggest that developmental coursework may be needed. The assessments results also could supplement or replace the course placement scores used to determine a high school student’s eligibility to enroll in dual-enrollment courses. Admissions and/or Scholarship Decision Rubric It is not uncommon for standardized assessments to be used as part of the admissions and/or scholarship decision rubric. It is important to note that these assessments were not designed for those purposes. If consistent with state and/or institutional policy, institutions may choose to include Smarter Balanced scores in admission or scholarship decision rubrics. As such, institutions’ online application for admission solutions, electronic transcript solutions and test score receipt solutions will need to be updated to accommodate these new data. Retention Analysis and Predictive Modeling Over time, as institutions collect multi-year Smarter Balanced Assessment score data, the detailed test scores can add richness, depth and accuracy to existing retention and predictive modeling analyses. In these studies, institutions would compare the information they received as part of the Smarter Balanced Assessment score data against variables such as grades in English and math courses in the first year, term and cumulative GPA, percent of courses completed and retention/persistence toward degree. These are just a few of the possible statistical tests that could help colleges and universities understand how the assessment data help them identify students who need academic support and create meaningful interventions and programs to increase the likelihood of persistence and credential/degree attainment. Assessment User Guide for Colleges and Universities 8 New Test Score Implementation Guidance by Technology Provider AACRAO partnered with several technology solution providers in the development of this guide. The information below contains specific information provided by vendors, directly or through product documentation, at the time of publication. If your information system vendor is not listed below, please contact the company directly for any information or guidance you may need to implement the use of Common Core assessment data. Campus Management Introduction Campus Management Corp’s CampusNexusTM Student solution provides clients with a tool to import any test score file layout. Once the file layout has been defined, users can import test scores for students based on incoming files from the test provider. Tests in the student information system have a variety of uses. They can be configured for admissions acceptance; have thresholds for scholarship or placement purposes; be utilized as pre-requisites for coursework; and can be incorporated in longitudinal reporting to find trends over time in student performance. Additionally, test scores can trigger documents and workflows, or waive other admission requirements. Test File Layout Entry The Test File Layout Entry User Interface allows clients to add file layouts for any student test that needs to have test scores imported into CampusNexus Student. Users may also import test scores for additional tests, match the import results, conduct test mapping, and then run the Entrance Test reports, including the Entrance Test, Entrance Test Detail Report, and Entrance Test Batch Detail Report. Adding File Layouts via the User Interface: 1. In CampusNexus student, a user would navigate to Tools > Admissions > Test File Layout Entry. The Test File Layout Entry form appears. Assessment User Guide for Colleges and Universities 9 This form consists of two sections. The top section, entitled Test Types, contains a grid which displays the available Test Types. The grid consists of three columns: Code, Description, and Active. The content for the three columns is obtained from the bottom section of the screen entitled Test Type Details. The Test Type Details section allows you to edit an existing test type or to add a new test type. If you add a new test type, the Code for a new test type is automatically capitalized and checked for uniqueness when you save it. The Code and Description fields are required and cannot consist of spaces. 2. Users may also set the system to create new leads when importing a test score for a prospective student who is not yet been entered into the system. This proactively identifies any students new to your system and eliminates the need for any dual data entry. Import Test Scores Within the Import Test Scores module, users can select the test type they want to import into CampusNexus Student. Test files must be a flat file with the current file layout provided by the official testing centers. After test files are imported into the system, the file name will automatically be archived by renaming the file with a ".imp" file extension. Entrance Tests for a Program The list of tests for programs is used to indicate that certain entrance tests are required of students who wish to enroll in a particular program (major or minor). Previous education can be used to qualify which tests and scores are required. Clients may select one or more tests from the list of tests and enter a required minimum score for each test selected. Once a test or tests are selected for the program, the tests will be tracked for any lead or student for whom the program is selected. This means that if the program is selected for a new prospective student, the program-related tests will be listed in that student's test file. Enrollment in the program is not a requirement for the tests to be listed in lead/student file. 10 When selecting a test, all sub-fields will also be available for selection, so that the system can compare the imported student's test score with the Program's or Program Versions' Entrance Assessment User Guide for Colleges and Universities Test Requirements. Only sub-fields that are of 'Score' type can be assigned a minimum score and remedial course requirement. Other types of sub-field will appear disabled. The 'Available Tests' list will contain only Main Tests. The "Selected Tests" list will contain the hierarchical order of the tests, which will include the sub-fields. The field name is followed by its field type such as score, text, date, or numeric value. Reporting As mentioned, standardized reports are delivered with the functionality. Additionally, test types, dates and scores, along with student demographic information and course and program completion information, can be reported on through the embedded ad-hoc reporting tool (Crystal Reports), via SSRS, and through the use of other report writing tools in conjunction with the data warehouse provided with CampusNexus Student. Common Application Group The Common Application is fully committed to responding to the needs of its members. To that end, they maintain a comprehensive process for evaluating suggestions regarding changes and additions to the application. As the members evaluate what role, if any, Common Core aligned assessments should play in the college application process, the Common Applications group stands ready to support and implement any solutions that meet their members’ needs, either individually or collectively. Credentials eSCRIP-SAFE If participating high schools decide to add the Smarter Balanced Assessment information to their PDF or EDI/XML transcripts, Credentials eSCRIP-SAFE will add the ability to extract the Smarter Balanced Assessment data elements from the delivered transcript to the receiving institution as needed. It will be the responsibility of the high school to add the information to their transcript. The receiving institution will need to import the data into their admission Assessment User Guide for Colleges and Universities 11 decision database. For PDF transcripts the receiving institution may choose to make this a manual/keyboard process, or they could elect to develop data-capture-update scripts on their side. For EDI/XML transcripts Credentials eSCRIP-SAFE will deliver to EDI/XML posts to the intended receiver, where the data will be unpacked and updated into the admission decision database. Ellucian If you have any questions about how to receive and process new test scores for any of the Ellucian suite of products, please contact: Vinay Nilakantan, Director Product Management Vinay.Nilakantan@ellucian.com (703) 261 2301 Hobsons CRM (Radius) The test score module is very flexible. Users can create new test types and either manually enter the data per student or automatically import it from an FTP location (Figure 1). Figure 1: Hobsons Radius Screen Shot of New Test Score Development 12 Assessment User Guide for Colleges and Universities Oracle (PeopleSoft) Defining Test Transfer Equivalency Rules Access the Test Credit Rule/Component page (Records and Enrollment, Transfer Credit Rules, Test Transfer Rules, Test Credit Rule/Component). Setting up test transfer equivalency rules is similar to setting up course credit transfer equivalency rules. For each test equivalency rule that you define, describe the rule, select the test and test component for the rule, and specify course equivalencies for the test component. Test ID Enter the identification number of the test for which you are defining this test equivalency rule. Test Component Enter the component of the test for which you are defining this test equivalency rule. Equiv. Component (equivalent component) The equivalency component number is the numeric counter that distinguishes each row of the test equivalency rule apart from other rows. By default, the system enters the first equivalency component of the test equivalency rule as 0001 and increases the number by one as you add rows. Description (lower) This field describes the row of the equivalency test rule. By default, the system displays the description of the test component according to the description on the Test Component Table page. You can override this default value. Transfer Priority Enter the transfer priority number for this row of the test equivalency rule. The Transfer Credit process evaluates the rows within the test equivalency rule according to the transfer priority of each row. The test component within the row that has the highest value takes priority. If an individual's transfer test meets all conditions of the test equivalency rule, then the Transfer Credit process uses the equivalent course as defined on the row with the highest transfer priority. However, if the individual's transfer test does not meet the conditions of the test equivalency rule, then the Transfer Credit process evaluates the row with the next highest transfer priority. Min/Max Score (minimum Enter the minimum and maximum score of applicable transfer tests and maximum score) for this row of the test equivalency rule, or enter the minimum percentile. Assessment User Guide for Colleges and Universities 13 Minimum Percentile Enter the minimum percentile of the applicable transfer tests for this row of the test equivalency rule, or enter the minimum and maximum score. Begin/End Date By default, the system sets the begin and end date of the transfer test to 01/01/1900 and 12/31/9999. You can override these default values. These dates inform the Transfer Credit process when the applicable transfer test must be taken for this row of the test equivalency rule. Maximum Age This field defines, in years, the maximum age of the transfer tests for this row of the test equivalency rule. This prevents an individual from transferring test credit into your academic institution if the individual took the test more years ago than the number of years that you specify here. By default, the system sets the maximum age of a transfer test to 99, but you can override this default value. For instance, you might want restrict an individual from receiving transfer credit for a test if the individual took the test more than 4 years prior to the date that your institution processes the individual's transfer credit. Course ID Enter the course to which the given test component is equivalent. The system prompts you with courses from your academic institution's course catalog. You can add rows to create a one-tomany test equivalency rule. You are prompted from your course catalog. The system automatically populates the Course Offering Number and Units Taken fields with values from the course catalog definition. You can enter a different offering number and enter different units. Course Offering Number By default, the system displays the course offering number of the specified course according to the value in your academic institution's course catalog. You can override this default value. Units Taken By default, the system displays the units taken value of the specified course according to the value in your academic institution's course catalog. You can override this default value. 14 Assessment User Guide for Colleges and Universities Populi SAT and ACT scores can be added to the prospect profiles in admissions and you can add other test scores to admissions using custom fields. The screenshot below shows how one can add those scores. 15 Assessment User Guide for Colleges and Universities Three Rivers Systems (CAMS) To Import Test Scores: 1. From the CAMS Enterprise Home page, click Tools >Import >Admissions >Test Scores Import. The Test Scores Import page opens. 2. Click Browse to select the file to be imported. You must use a UNC path to designate the import file. The file must also be accessible by the SQL server. In this example an ACT test scores file will be imported. 3. Select the File Type. There are a standard set of importable test files. The years after each file type defines what file layout is being used. 4. Click Create New Students/Prospects for Unmatched Records if new student or prospect records should be created for records that do not match an existing student or prospect record. If this is checked then select Insert To Prospects or Students. Select an Address Type if necessary. Note: Address type does not have to be selected. If a type is selected and unmatched records are going to Prospects then CAMS Enterprise will create the address type selected and a Home address type. If unmatched records are going to Students then CAMS Enterprise will create the address type selected and a Home, Billing and Local address type. If a type is not selected CAMS Enterprise will create a Home address in Prospects and a Home, Billing and Local address in Students. 5. Click Next. The New Records or Create New Students/Prospects for Unmatched Records window displays depending on what was selected on the first window. If Create New Students/Prospects for Unmatched Records was selected then the first screen displays all individuals who did not match any criteria. 6. Select the records to import by checking the corresponding Import checkbox. Those records not checked will not be imported. If there are two records for the same individual only one prospect/student record will be created but both test scores will be imported. 7. Click Next. The Exact Matched Records window displays. The Exact Matched Records window displays those records that met the criteria for an Exact Match. This window will normally not be changed. Warning: If changes are necessary it is important to make the change in the correct data grid. Assessment User Guide for Colleges and Universities 16 The upper data grid displays those records that are in the file being imported. If you uncheck the Import checkbox in the upper grid then that record will not be imported. The lower grid displays information that is being imported related to the record highlighted in the upper grid. If you uncheck the Selected checkbox in the lower grid and the Create New Students/Prospects for Unmatched Records checkbox on the first window was checked then this record will be imported into CAMS Enterprise as a new prospect or student record. 8. Click Next. The Partial Exact Matched Records window opens. In this window a record will have a match in both the prospect and student modules. One will be an Exact match and one will be a Fuzzy match. A decision will need to be made on which record to associate the imported score(s) with. 9. Select the correct record in the lower grid. Warning: If changes are made it is important to make the change in the correct data grid. The upper data grid displays those records that are in the file being imported. If you uncheck the Import checkbox in the upper grid then that record will not be imported. The lower grid displays information related to the record highlighted in the upper grid that will help in determining which record to associate the score(s) with. If you uncheck both Selected checkboxes in the lower grid and the Create New Students/Prospects for Unmatched Records checkbox on the first window was checked then this record will be imported into CAMS Enterprise as a new prospect or student record. 10. Click Next. The Fuzzy Matched Records window displays. This window displays those records that can possibly match records in CAMS Enterprise. The upper grid displays those records that can be imported. In the lower grid it displays the record that there was a fuzzy match with. Warning: If changes are made it is important to make the change in the correct data grid. The upper data grid displays those records that are in the file being imported. If you uncheck the Import checkbox in the upper grid then that record will not be imported. The lower grid displays information related to the record highlighted in the upper grid that will help in determining which record to associate the score(s) with. If you uncheck both Selected checkboxes in the lower grid and the Create New Students/Prospects for Assessment User Guide for Colleges and Universities 17 Unmatched Records checkbox on the first window was checked then this record will be imported into CAMS Enterprise as a new prospect or student record. 11. Select the correct record to associate the imported score(s) with. 12. Click Next. The Stored Procedures window opens. The Stored Procedures window allows the selection and running of custom stored procedures that the institution may have created. They can be run before and/or after the scores import. 13. In the left window all stored procedures will be displayed. Highlight a stored procedure and use the double right arrows to select it to run the stored procedure before the import or after the import. It will display in the upper and lower windows on the right side. You may add more than one stored procedure in each window. Use the double up or down arrows to prioritize the stored procedures’ running order. If there are numerous stored procedures in the list they can be filtered by using the Stored Procedure Name Prefix field. End in the beginning letters of your stored procedure and click the ReQuery button to filter the list. 14. Click Next. The Review and Process window displays. This window allows you to verify what imports will take place. The first column is Import To. You will see it display New for those records that will be created in CAMS Enterprise. If it lists a prospect or student’s name then the third column, RecordType, will display what type of record the score(s) import will associate with. The SelectedUID field will ensure that it is going to the correct record. 15. After the records are reviewed click Finish. A dialog box will appear when all the records have been imported. 18 Assessment User Guide for Colleges and Universities