Ivy Tech Community College
Blackboard Instructor Training
Course (BBRD101)
Evaluation Plan
Prepared by:
Michelle Pittman
October 13, 2013
Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 2 Executive Summary To: Ivy Tech Community College -­‐ Center for Instructional Technology and Excellence Instructional Designers; Dr. Kara Monroe; Matthew Pittman (North Central Region, Online Technologies Coordinator) From: Michelle Jackson Subject: Evaluation Plan for Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) Date: October 6, 2013 CC: Dr. Marisa Exter; Ji Hyun Yu Purpose of the Evaluation of BBRD101 The Blackboard Instructor Training Course (BBRD101) is a statewide training. Every Ivy Tech Community College (ITCC) faculty member statewide is required to complete this training prior to the start of his or her first teaching assignment. This training assists faculty with the tools of Blackboard and how to implement them in accordance to ITCC’s standards and policies. Objective The primary objective of this evaluation plan is to assist ITCC to measure the effectiveness of the training to prepare faculty to use Blackboard and their implementation of Blackboard in their classrooms. Summary of the Evaluation Plan Due to size of the ITCC system, the North Central region will be used as the audience and context. The North Central region, which is outlined in this evaluation study and plan, is comprised of South Bend, Elkhart County, and Warsaw campuses. The North Central region has 500 faculty. Results from the study could be generalizable to other regions, but an evaluation for those regions should also be completed, as there may be several differences in the population of the regions. The evaluation strategy is built upon Kirkpatrick & Kirkpatrick’s (2006) four-­‐level evaluation model. The four levels are (1) Reaction, (2) Learning, (3) Behavior, and (4) Results. The following evaluation procedures will be used to address each of the levels: • See Appendix G for the recommended timeline for the evaluation. • To evaluate reaction: Faculty are surveyed prior to the beginning of the workshop and at the end (pre and post surveys). The purpose of the survey is to gather reactions to the training. In addition, the surveys will gather data about the training in regards to relevance of the materials, what should be eliminated, and what needs to be added. The complete survey can be found in Appendix B. The survey will be incorporated into Blackboard. A link to the survey will be made available within Session 1 and Session 8. • To evaluate learning: Kirkpatrick & Kirkpatrick (2006) suggest that a control group be used to evaluate learning. However, because all faculty are required to be certified to teach on Blackboard, a control group cannot be used in the evaluation. Therefore, a pre-­‐test and post-­‐
test will be used to evaluate learning. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 3 A pre-­‐test will be found in session 1 and is required to assess the faculty’s current knowledge of Blackboard. How they score will not affect their certification. o A post-­‐test, also known as the certification exam, will be found in session 5. Faculty is required to pass this certification exam with a 90% or better. They will have three opportunities to pass. If they do not pass within three attempts, they will need to meet with a representative in their Online Technologies Department for further training and review. o Self-­‐assessments are provided to faculty in each session. They are not required for faculty to complete and they do not affect their certification. They are also welcome to take the self-­‐assessments more than once. The results of each attempt will be recorded in the gradebook and could be used to show additional data as necessary to backup recommendations or needs. To evaluate behavior: The faculty’s supervisor will complete observations on the faculty’s Blackboard usage in their courses at 30 days and 60 days. The observation will look at how the faculty member utilizes the Blackboard tools and how they perform using Blackboard in their classrooms. The observation will also make sure that they meet the Blackboard requirements. This observation evaluation will be used to show the transfer of knowledge from the training to the work environment and the increase or decrease over time. Due to the number of faculty and the number of courses, a representative from the Online Technologies department will complete a random sample for observations. The observation tool can be found in Appendix D. o
•
Data Findings Since this training is new, there currently are no initial data findings. This evaluation is scheduled to begin on January 9, 2014. The data gathered through the evaluation of BBRD101 will assist the Online Technologies department and the Center for Instructional Technology and Excellence to (1) determine the needs of faculty who use Blackboard and (2) areas of improvement of the training to help faculty implement Blackboard in their classrooms. They will also be able to review the reactions of faculty to the training and the average understanding of the Blackboard tools. Recommendations Based on the evaluation, possible recommendations from the evaluation would be: (1) areas of improvement in the training and (2) how faculty are utilizing Blackboard in their classrooms. From this data and analysis, areas for additional trainings could be discovered and implemented for faculty to further their Blackboard knowledge. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 4 Evaluation Goals and Scope Description Ivy Tech Community College (ITCC) is a singly accredited statewide community college system in Indiana and the largest in the nation. ITCC offers Associate degrees, technical certificates, and workforce training and certification. Approximately 200,000 students per year attend ITCC. The statewide system is comprised of 14 regions covering the entire state. Each region has one or more campuses or sites. For instance, the North Central region, which outlined in this evaluation study and plan, is comprised of South Bend, Elkhart County, and Warsaw campuses. The North Central region serves 8,500 students and 500 faculty and adjuncts. At Ivy Tech, students and faculty interface with Blackboard Learn 9.1 (Bb 9.1) as the learning management system (LMS) for traditional, hybrid, and online courses. All faculty are required to utilize Blackboard to communicate basic course information in all course sections regardless of delivery method. At minimum, each faculty member will post his or her syllabus, calendar, contact information, and post grades through Blackboard. They are required to complete training and become certified on the basic use of Blackboard prior to the start of their first teaching assignment. To become certified, faculty need to pass a proficiency test with a 90% or better score in the Blackboard Instructor Training Course (BBRD101). They can complete the training by either attending a three-­‐hour face-­‐to-­‐face format or through a self-­‐paced online course (completed in a one week timeframe). Faculty can choose which training that they want to participate in. All faculty are required to pass the same certification test found in the online training regardless of the modality in which they receive the information. At the end of the training, if they pass the certification test with a 90% or better, faculty are awarded with a certificate of achievement and can begin teaching at ITCC. Audience and Context: Learner Analysis All ITCC faculty throughout the statewide system are required to be certified to teach with Blackboard. Faculty chose which modality that they want to participate in the training – either face-­‐to-­‐face or online. Regardless of the modality in which they interact with the materials and activities, all faculty are required to pass the same certification test with a 90% or higher. This evaluation specifically will be looking at the North Central cross-­‐section of faculty. This evaluation specifically will analyze the online workshop rather than the face-­‐to-­‐face because the post-­‐test in the online workshop is required of each person regardless of modality taken and the material presented is the same. In addition, since approximately 80% of the faculty at ITCC is adjunct faculty and they do not get paid to participate in this training, it is expected that a majority of the faculty will participate in the training from home, online. There are approximately 500 current faculty members in the North Central region for Ivy Tech. The demographics of the faculty within the region are a mix of female and males. Full-­‐time faculty have a Master’s degree or terminal degree; adjunct faculty hold at least a Bachelor’s degree or higher. Approximately 80% of faculty is adjuncts. They come to the training with varying degrees of technical knowledge from novice to expert. This training is the first interaction that the faculty has with Blackboard at ITCC. However, it is possible that new faculty may have previous exposure to Blackboard through other avenues such as, being a faculty member at another college or as a student. Therefore, they will have varying degrees of knowledge of Blackboard, ranging from none to expert. Faculty are notified about this training through e-­‐mail from their regional Online Technologies department and other notification strategies of the requirement statewide to pass the certification. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 5 They are notified that they may either take the training online at their own pace or come to one of the Ivy Tech locations to take a three-­‐hour certification in a face-­‐to-­‐face format. Also noted to faculty that regardless of modality they must pass the final certification with a 90% or better in Class Session 5: Certification Exam. They are sent instructions on how to access the training through Blackboard. The certification test is completed online through Blackboard and faculty will receive their results immediately upon completion. A copy of the sample questions in the certification exam can be found in Appendix E. The online training consists of eight modules presented to faculty within the Bb 9.1 environment, which include: • Session 1: Course Expectations • Session 2: Blackboard Navigation • Session 3: Adding Content • Session 4: Grade Center Basics • Session 5: Certification Exam Learning activities included in the training are audio, handouts, and videos. The learning activities, assignments, and assessments are further outlined in Appendix A. Also found in Appendix A are screenshots of the online learning environment for the training. Looking further, faculty is the main stakeholder in this training because they are required to become certified to teach for ITCC. In addition to faculty, another major stakeholder are the instructional designers charged with developing the faculty introduction training to Blackboard. They will be using the data collected in this training to revise and improve. Another stakeholder that would be interested in the results of this study would be the administrative staff of ITCC North Central region. These administrators would be the Chancellor, Vice Chancellor of Academic Affairs, Deans, and Program Chairs. They could use the results of the evaluation to help determine further needs for training relating to Blackboard. Evaluation Procedures, Strategy & Timeline The evaluation strategy is built upon Kirkpatrick & Kirkpatrick’s (2006) four-­‐level evaluation model to determine the effectiveness of the training to prepare faculty to use Blackboard and their implementation of Blackboard in their classrooms. The four levels are (1) Reaction, (2) Learning, (3) Behavior, and (4) Results. Evaluation Timeline • See Appendix G for the recommended timeline for the evaluation. o Design and implementation of the plan will begin on December 2, 2013. o Faculty will have access the training on January 6, 2014. It is estimated that it will take 2-­‐
4 hours to complete. They are required to complete the training in one week. Therefore, data will be collected during that week. This date was chosen due to the requirement that all new faculty complete this training prior to their class beginning. o Data analysis will be completed within two weeks. Recommendations will also be presented. o The evaluation will by completed by February 3, 2014. Evaluation Procedures Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) •
•
•
6 Faculty are surveyed prior to the beginning of the workshop and at the end (pre and post surveys). The purpose of the survey is to gather reactions to the training. In addition, the surveys will gather data about the training in regards to relevance of the materials, what should be eliminated, and what needs to be added. The complete survey can be found in Appendix B. The survey will be incorporated into Blackboard. Blackboard has a survey feature where faculty or students can leave feedback anonymously and not have to leave the Blackboard learning environment. A link to the survey will be made available within Session 1 and Session 5. o In order to ensure a 100% response rate, the surveys are required. Faculty will not be able to continue to the next session without entering information into the survey. It is possible that using this method will also create data that will be skewed because some faculty may hurry to finish the survey to just be able to move on to the certification test. o Faculty will also be surveyed at 30 days and 60 days after completing the training to ensure that their satisfaction did not diminish over time or perhaps increased over time. o The faculty’s supervisor will complete observations on Blackboard usage at 30 days and 60 days. The observation will be looking at how the faculty member utilizes the Blackboard tools and how they perform using Blackboard in their classrooms. The observation will also make sure that they meet the Blackboard requirements. Kirkpatrick & Kirkpatrick (2006) suggest that a control group be used to evaluate learning. However, because all faculty are required to be certified to teach on Bb 9.1, a control group cannot be used in the evaluation. Therefore, a pre-­‐test and post-­‐test will be used to evaluate learning. o A pre-­‐test will be found in session 1 and is required to assess the faculty’s current knowledge of Blackboard. How they score will not affect their certification. o A post-­‐test, also known as the certification exam, will be found in session 5. Faculty is required to pass this certification exam with a 90% or better. They will have three opportunities to pass. If they do not pass within three attempts, they will need to meet with a representative in their Online Technologies Department for further training and review. Self-­‐assessments are provided to faculty in each session. They are not required for faculty to complete and they do not affect their certification. The results will be recorded in the gradebook and could be used to show additional data as necessary to backup recommendations or needs. Evaluation Instruments There are four evaluation instruments used in this evaluation plan, including: 1. Surveys 2. Interviews 3. Observations 4. Performance-­‐based tests These evaluation instruments are discussed in further detail below and outlined in Appendices B through E. Please note that this evaluation will be focusing specifically on the online version training. Therefore, questions about training instructor will be eliminated because it is a self-­‐paced training and certification. Questions will be more focused on activities, materials, and assessments in each level. Observations of the faculty members will be completed to evaluate their knowledge gained from the training. Level 1: Reaction Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 7 Kirkpatrick & Kirkpatrick (2006, p. 27) state, “Evaluating reaction is the same thing as measuring customer satisfaction. If the training is going to be effective, it is important that trainees react favorably to it.” Learning and transfer of learning is unlikely to occur unless the faculty has positive attitudes toward the training. Faculty reactions to the training are very important. We want the faculty to be charged up and excited to implement Blackboard in their classrooms. Blackboard can incorporate different interactions (student-­‐content, student-­‐student, student-­‐instructor) to their course experience, which could aid students to be more successful in their courses. If faculty are not excited or motivated to use Blackboard in their classrooms, they will not explore the Blackboard environment to incorporate tools within their classroom to further student engagement with the course. Evaluation Instruments: Survey, Interview The instrument used to gather faculty reaction to the training will be in survey form. The survey questions will gather data on the satisfaction, quality, and relevance of the learning materials and activities. The survey will be completed online through Blackboard. Blackboard has a survey feature where faculty or students can leave feedback anonymously and not have to leave the Blackboard learning environment. A link to the survey will be made available within Session 1 and Session 8. Survey questions will be grouped accordingly. In order to have 100% participation for the most data, they will be required to complete the survey for the next session to open. See Appendix B for the complete survey. Approximately 25-­‐30 faculty will be interviewed to gather additional information. The evaluation team may feel that it is necessary to interview additional faculty depending on the results found in the 25 interviews. At least a minimum of 2-­‐3 faculty per discipline at each campus will be interviewed to show a cross-­‐section of demographics per campus and per discipline. Interviews will be conducted by a representative of the Online Technologies Staff. See Appendix C for the complete interview. Rationale: Kirkpatrick & Kirkpatrick (2006, p. 28) recommend the following guidelines for evaluating reaction: (1) Encourage written comments and suggestions, (2) Get 100% immediate response, and (3) Get honest responses. Data will be collected electronically so that it is easier to disseminate the data and analyze. In addition, the survey has open-­‐ended responses so that faculty can leave additional information that they want us to know in the survey. Blackboard will also show that in fact they completed the survey, but it will not show what survey data is directly from a particular faculty member. Data collected from this level could also be used to provide reasoning for the data collected in level two. Reaction to the training can positively or negatively affect learning. Level 2: Learning Kirkpatrick & Kirkpatrick (2006, p. 50) state, “Evaluating learning is important. Without learning, no change in behavior will occur.” The purpose of the learning evaluation is to determine learning and the transfer of learning from the training to the work environment. It is important for faculty to correctly implement the Blackboard tools in their classrooms. Tools that are improperly used or implemented can cause student frustration with the course, which could affect their overall success in the course. Evaluation Instruments: Self-­‐assessments, Certification test & Observations In order to help faculty assess their own knowledge of Blackboard and to prepare for the certification test in Session 5, self-­‐assessments are provided at the end of each session, except for Session 1 and Session 5. These self-­‐assessments are not required for faculty to complete and Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 8 they do not affect their certification. The results will be recorded in the gradebook and could be used to show additional data as necessary to backup recommendations or needs. To eliminate bias in the observation, a representative of the Online Technologies staff will be conducting an observation of the course in Blackboard. The observation will verify that they met the Blackboard requirements. This observation evaluation will be used to show the transfer of knowledge from the training to the work environment. The observation tool can be found in Appendix D. Kirkpatrick & Kirkpatrick (2006, p.43) suggest that knowledge, skills, and attitudes are evaluated before and after. The focus for evaluating learning in the training will be a certification test. In Session 1, there will be a pre-­‐test required for the faculty member to complete. The pre-­‐test is completed online through Blackboard. There is no minimum score that a faculty member needs to achieve in order to move on. They are not able to take this pre-­‐test more than once. They will receive their score immediately after submission. The certification test requires faculty to have knowledge of how to use Blackboard. For instance, after the training they should know how to place their syllabus and calendar in their course. In addition, they should understand how to update their gradebook. They should also understand how to post lecture notes and assignments. Faculty may take the certification test up to three times to achieve the required 90%. After three times, faculty will be required to contact their local Online Technologies department for additional training and review. The certification test is completed online through Blackboard. Faculty receives their results immediately after submission. The certification test and pre-­‐test is 25 questions pulled randomly from a test bank. This means that faculty may not receive the same questions the second time that they had for the first time. A sample of the questions from the certification test is in Appendix E. Rationale: Kirkpatrick & Kirkpatrick (2006, p. 53) recommend the following guidelines for evaluating learning: (1) Use a control group, (2) Evaluate, knowledge, skills, and/or attitudes both before and after the training, (3) Use a performance test to measure skills, (4) Use a paper/pencil test to measure knowledge and attitudes, and (5) Get a 100% response rate. All faculty are required to pass the certification test and they are required to post their documents on Blackboard. Therefore, there will be a 100% response rate. Observations will evaluate the knowledge, skills, and attitudes after trainings. Faculty are measured by the certification test to measure the skills. Level 3: Behavior Kirkpatrick & Kirkpatrick (2006, p. 50) state, “What happens when trainees leave the classroom and return to their jobs? How much transfer of knowledge, skills, and attitudes occur? That is what level 3 attempts to evaluate.” Evaluation Instruments: Observation To eliminate bias, an Online Technologies representative will complete observations on the faculty’s Blackboard usage in their courses at 30 days and 60 days. The observation will look at how the faculty member utilizes the Blackboard tools and how they perform using Blackboard in their classrooms. The observation will also make sure that they meet the Blackboard requirements. This observation evaluation will be used to show the transfer of knowledge from the training to the work environment and the increase or decrease over time. The observation tool can be found in Appendix D. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 9 Rationale: Kirkpatrick & Kirkpatrick (2006, p. 53) recommend the following guidelines for evaluating behavior: “(1) Use a control group, (2) Allow time for change to take place, (3) Evaluate both before and after the training, (4) Survey or interview one or more of the trainees, their immediate supervisor, their subordinates, and other who observe their behavior, (5) Get 100% response rate, (6) Repeat the evaluation at appropriate times, and (7) Consider cost versus benefits.” Since these faculty members are new to ITCC, it is not feasible to evaluate their Blackboard usage before training. Due to the number of faculty, a re-­‐certification or administering another test would be time-­‐consuming and beyond the scope of this evaluation. In addition a control group cannot be used in this evaluation due to the requirement that all new faculty members must complete the certification. Level 4: Results Kirkpatrick & Kirkpatrick (2006, p.63) state, “Now comes the most important and perhaps the most difficult part of the process, you decide-­‐determining what final results occurred because of attendance and participation in a training program.” The data gathered through the evaluation of BBRD101 will assist the Online Technologies department and the Center for Instructional Technology and Excellence to (1) determine the needs of faculty who use Blackboard and (2) areas of improvement of the training to help faculty implement Blackboard in their classrooms. In addition, faculty are required to pass the certification with 90% or higher. Evaluation Instruments: Survey The instrument used to gather faculty follow-­‐up reaction to the training will be in survey form. The survey will be completed online through Blackboard. Blackboard has a survey feature where faculty or students can leave feedback anonymously and not have to leave the Blackboard learning environment. They will be asked to complete this survey after 30 days from the training. See appendix H for the full follow-­‐up survey. Rationale: Kirkpatrick & Kirkpatrick (2006, p. 65) recommend the following guidelines for evaluating results: “(1) Use a control group, (2) Allow time for results to be achieved, (3) Measure before and after the program, (4) Repeat the measurement at appropriate times, (5) Consider cost versus benefits, and (6) Be satisfied with evidence if proof is not possible.” After 30 days of being in the classroom utilizing Blackboard after completing the training, faculty should have more of an idea what they need to learn more about to be successful using Blackboard. This follow-­‐up survey to faculty will show further results into how they are using Blackboard in their classroom and what additional trainings they need and want. The instruments developed for levels 1, 2, and 3 will also be used to measure and serve as a baseline for the results in level 4. Data Collection & Analysis Level 4 of the Kirkpatrick & Kirkpatrick model is results. This level determines whether the goals of the training were met. In addition the results of the evaluation can be used to assist with the necessary improvements of the training. The primary objective of this evaluation plan is to assist ITCC to measure the effectiveness of the training to prepare faculty to use Blackboard and their implementation of Blackboard in their classrooms. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 10 Stakeholders will be involved from the beginning of the evaluation to determine the needs of the evaluation and what their goals are for the training to achieve. There is one clear-­‐cut requirement of the training to determine effectiveness. Faculty must complete the certification test with a 90% or better. However, faculty have three chances to meet this requirement. Although they may pass the test, this does not mean mastery. They could pass the certification with assistance from another or just sure luck in some cases. A discussion with the stakeholders to determine what is the definition of “effective” in this case is needed as well. Level 1: Reaction Level one data will be collected and aggregated by category for analysis. Reactions are averaged in relation to quality of materials and activities, satisfaction, and relevance. Data will be presented to the stakeholders in charts. Since data is collected at the beginning and at the end of the training, data will be compared as well using charts. The anecdotal information collected through the interviews and the other comments area in the survey will be presented in a list format accompanying the charts by common themes found in the interviews. There are some challenges that a positive reaction may be influenced by to keep into account when reviewing the results of the reaction. Faculty are required to pass this certification. They may have a negative view on being forced or required to do anything relating to Blackboard. They may feel that participating in this training is wasting their time. This may be particularly true if a faculty member has a great deal of working knowledge of Blackboard already. It may just seem like another hurdle to just “get through.” It also may be possible that the training will encounter faculty that are resistant to using Blackboard. They are upset that they have to use Blackboard in their classes. They just want to keep using their old paper and pencil teaching techniques. Level 2: Learning Faculty are required to pass the certification test with a 90% or higher to show proficiency using Bb 9.1. The results of this test will be aggregated and displayed with charts. It will also be noted how many attempts it took faculty on average to get the 90%. This will show an area of improvement to see if the learning materials or activities need to be improved to help faculty better prepare for the certification. The data collected from the pre-­‐test and the certification test will also be compared to see if there is an increase in knowledge. The certification test results will be analyzed to find those areas that are weakest for faculty as well as strengths. This data will be presented in a bar chart for easy comparison. This data will also be presented side by side the pre-­‐test information for comparison to see if faculty improved and in what areas. There are some challenges that may affect learning, including (1) the faculty’s technical skills and expertise. Since the material is being presented on Blackboard, if the faculty member lacks basic computer skills, they may have trouble locating the training or completing the test online. They may also have issues completing the required Blackboard tasks such as uploading a document to Blackboard not because they don’t know how to do it, but other technical issues such as locating the document or creating documents electronically. This training specifically just trains faculty to upload documents, not how to create them. Results for the observations will be collected and aggregated in the form of charts and anecdotal comments. The results from the certification test will be aggregated down to the number of attempts Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 11 and compare with the initial score from the faculty’s pre-­‐test. This data will be reviewed to find areas of improvement in the training. Level 3: Behavior Faculty will be evaluated by course observations in Blackboard. This data will be aggregated in the form of charts by timeframe. Further a line chart will be used to show an increase or decrease in Blackboard usage. There are some challenges that may affect behavior, including (1) time constraints on the faculty’s supervisor and the Online Technologies department to complete the observations. Behavior is also influenced by the faculty’s reaction to the training and learning. If they do not truly learn the material (like the learn the material for the certification test), they may have difficulty implementing the Blackboard material. Level 4: Results The data collected at this level will be aggregated by demographics and in the form of bar charts and list of responses in the open-­‐ended questions. The challenge faced in this level would be the number of respondents to the survey. Faculty may not respond to the follow-­‐up survey due to the nature of an electronic survey. Some additional follow-­‐up may be required to gather the follow-­‐up data we need. See Appendix F for an example of how the data will be presented. Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) References Ivy Tech Community College. (2013). Blackboard Instructor Training Course. Retrieved from Ivy Tech Community College, Blackboard website. Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs. San Francisco: Berrett-­‐
Koehler Publishers, Inc. 12 Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) Appendices Appendix A: Instructional Materials & the Learning Environment…………………………………………. p. 13 Appendix B: Reaction Survey………………………………………….……………………………………………………… p. 14 Appendix C: Interview Tool………………………………………….………………………………………………………… p. 16 Appendix D: Observation Tool………………………………………….…………………………………………………… p. 17 Appendix E: Sample Questions from the Pre-­‐test and Certification Test………………………………… p. 18 Appendix F: Sample Data Results Presentation………………………………………….………………………….. p. 19 Appendix G: Gantt Chart………………………………………….……………………………………………………………. p. 20 Appendix H: Follow-­‐up Survey………………………………………….…………………………………………………… p. 23 13 Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 14 Appendix A: Instructional Materials & the Learning Environment Class Instructional Materials, Activities, Assignments & Assessments Session 1 Assessment, Survey, Handouts, Audio 2 Assessment, Handouts, Audio 3 Assessment, Handouts, Audio, Video, Flickr 4 Assessment, Handouts, Audio 5 Online Test Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 15 Appendix B: Reaction Survey Name (optional): ………………………………………………………………………………………….. About You Department: Number of Years Teaching: Number of Years using Blackboard: Quality of the Content 1. What learning activities were most beneficial to you? Please rank the learning activities (1=Not Useful; 5=Very Useful) Handouts 1 2 3 4 5 Comments: Videos 1 2 3 4 5 Comments: Audio 1 2 3 4 5 Comments: 2. What information (handouts, materials, written text) were you provided that you felt assisted you in using Blackboard? Presentation & Delivery of the content, assignments, assessments 3. How easy was the content to navigate? (1=Not Easy; 5=Very Easy) 1 2 3 4 5 Comments: 4. Was the amount of information provided in each section appropriate? (1=Not Enough; 5=Too Much) 1 2 3 4 5 Comments: 5. I understand what is required of me to utilize Blackboard in my courses. Yes No Comments: Relevance of the Instructional Content 6. How relevant were the sessions for your job? (1=Not Relevant; 5=Very Relevant) 1 2 3 4 5 Comments: Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 16 7. How confident do you feel about applying what you learned in this training to your job? (1=Not Confident; 5=Very Confident) 1 2 3 4 5 Comments: 8. I will utilize what I learned in this workshop into my course on Blackboard. Yes No Comments: 9. What tools do you plan on implementing in your classroom? (pick all that apply)  Blogs  Wikis  Journals  Mashups  Discussion boards  Tests  Surveys  Other 10. What other items or training do you need to make Blackboard more beneficial for you, your students, and your job? Overall levels of satisfaction 11. Would you be interested in attending additional trainings on Blackboard? Yes No Comments: 12. Were there any topics or specific content missing that you feel should be added in the workshop? 13. Were there any topics or specific content that you felt were unnecessary? 14. Please rate how beneficial the training was to you. (1=Not Beneficial; 5=Very Beneficial) 1 2 3 4 5 Comments: Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 15. If you have any further comments about the content of the training, please add them here. 17 Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 18 Appendix C: Interview Tool Name (optional): Instructions to the Interviewer: Ask the following 7 open-­‐ended questions and record the interviewee’s responses. Please provide as much detail as possible. About the Interviewee Department: Number of Years Teaching: Number of Years using Blackboard: 1. What was the most valuable part of the training? Why? 2. Has completing the training helped you to do any of the following tasks in Blackboard? a. Uploading your syllabus or other materials in your course b. Adding your instructor contact information c. Implementing any of the following tools: blogs, wikis, journals, discussion boards d. Other 3. What in your opinion could be added to improve the training? 4. What in your opinion could be changed to improve the training? 5. What in your opinion could be removed to improve the training? 6. What knowledge from the training have you employed in your classroom on Blackboard? Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 7. Is there anything else you would like to tell us? 19 Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 20 Appendix D: Observation Tool Name: Evaluator: Instructions: Please check off all of the following that applies. Please leave any notes in the comments section that needs to be addressed by the faculty member. Any of the items starred are the minimum Blackboard usage requirements. These items are required and will need to be addressed if missing from the course. Yes No Review Comments Contains a Syllabus* Instructor Information* Contains a Calendar* Grades in the Grade Center* Course in the QM template Unused tools turned off Blogs Wikis Journals Lectures uploaded Contains Resources for students (videos) Discussion boards Tests Surveys Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 21 Appendix E: Sample Questions from the Pre-­‐test and Certification Test 1. In Blackboard 9.1 there is no way to reorder announcements – they will be listed in the order in which they are posted. a. True b. False 2. Which of the following content types in Blackboard 9.1 allows faculty to add content from both external websites and some other Blackboard tools directly Text Editor in Blackboard? a. Mashups b. URLs c. Files 3. Which of the following is a content type in Blackboard 9.1 that allows faculty member to post attached files to a course without adding a description? a. Files b. Items c. URLs d. Mashups 4. Which area of the Control Panel allows faculty to access instructor features of course tools? a. Course Tools b. Tool availability c. Customization 5. It is best practice in course design to leave course tools available even if you are not planning to use those tools. a. True b. False 6. Which interactive tool provides a self-­‐reflective tool where students post their personal reflections about the course content? a. Journals b. Blogs c. Wikis d. Discussion boards 7. The Message tool allows faculty and students to send messages to external Ivy Tech email accounts a. True b. False 8. The Grade Center now allows faculty members to color code columns based on rules that have been created for specific assignments. a. True b. False 9. A faculty member must use only the Blackboard grade center and cannot maintain any offline view of their grades. a. True b. False Evaluation Plan: Ivy Tech Community College Blackboard Instructor Training Course (BBRD101) 22 Appendix F: Sample Data Results Presentation Usefulness 5 4.5 4 3.5 3 2.5 Usefulness 2 1.5 1 0.5 0 Handouts Audio Video