Quality of Learning Management Systems towards ELearning Effectiveness: A Study based on Blackboard 7.1, Moodle 1.9 and Blackboard 9.1 Presented By: Nadeera Ekanayake, Travis Fisher , Brandon Molton Presented To: Dr. Ahmad Zargari Department of Applied Engineering and Technology Morehead State University March 2013 2 Table of Contents Chapter 1: General Area of Concern............................................................................................................. 3 Introduction ............................................................................................................................................... 3 Problem Statement .................................................................................................................................... 5 Significance............................................................................................................................................... 6 Assumptions.............................................................................................................................................. 7 Limitations ................................................................................................................................................ 7 Definition of Terms................................................................................................................................... 8 List of Acronyms ...................................................................................................................................... 9 Chapter 2: Literature Review ...................................................................................................................... 10 Background ............................................................................................................................................. 10 ELearning............................................................................................................................................ 10 Learning management systems ........................................................................................................... 14 Blackboard .......................................................................................................................................... 19 Moodle ................................................................................................................................................ 21 Moodle Compared to Blackboard 9.1 ................................................................................................. 22 A brief historical review ......................................................................................................................... 24 Rationale ................................................................................................................................................. 26 Chapter 3: Methodology ............................................................................................................................. 28 Research Design...................................................................................................................................... 28 Instrumentation ....................................................................................................................................... 30 Data Collection Methods ........................................................................................................................ 32 Data Analysis Methods ........................................................................................................................... 32 Appendix A – Sample Survey Questionnaires ............................................................................................ 34 References ................................................................................................................................................... 40 3 Chapter 1: General Area of Concern Introduction E-learning programs and utilities have become important mainstays for a large variety of educational institutions ranging from secondary schools through collegiate and university programs. E-learning programs help to augment the learning services institutions can provide by enabling a method for students and faculty to communicate among each other outside of lecture and discussion, as well as, allowing distance learning for students who are incapable to being present for traditional personal learning techniques. For many instructors, e-learning tools have become vital in order to successfully communicate important information to students, including various texts and media related to the course’s current topic, important dates and times for events such as exams, and coursework for students to complete and turn-in. Instructors even often have the option to facilitate class discussion through course forums, and host video seminars. With these tools, instructors easily have the option to teach courses in a hybrid model, utilizing both in-class lecture and at-home online studying in order to enhance the learning experience, or even teach courses completely outside of lecture by providing all of the necessary tools students require to complete the course so long as all participating students and faculty have access to a computer and internet access. Although it is possible for an institution to create their own e-learning system, there currently exist programs which can be implemented and used across an entire institution which can automatically setup and maintain all of the background functionality required to establish a successful e-learning system. Currently, the two most popular programs are Blackboard and Moodle. Although both are developed and distributed with the same goal in mind (to provide a direct method of communication between students and instructors outside of the classroom 4 environment) both systems do function somewhat different from both a student’s and a faculty/instructor viewpoint. Prior studies have also shown that both students and faculty have shown personal preference and varying levels of success using one system over the other. Traditionally, Morehead State University relied solely on the Blackboard e-learning system in order to encourage the application of an online learning environment within its courses. However, in 2011, Morehead State experimented with a pilot program which permitted some courses to use Moodle in its courses, in a trial run which would ultimately help to decided whether the university would continue with Blackboard, or overhaul the e-learning processes within the university to switch to a Moodle based program. When Morehead started the evaluation process, it was using Blackboard Learn 7.3. Although, Moodle was successfully implemented, the decision was made by Morehead State to continue on with the use of Blackboard. Although, the stint with Moodle was short lived, there still exists the ability to collect valuable data from those who took part in this pilot program and most closely research the effects both Blackboard and Moodle have on both students and facilitators. This proposal will provide background information behind Blackboard and Moodle and present relevant past research which will demonstrate the importance of continuing the investigation of the comparison between the effects of Blackboard 7.3, Moodle 1.9 and Blackboard 9.1 have on Morehead State University’s e-learning programs. 5 Problem Statement There are various online learning management systems available. As discussed in the Introduction section Blackboard and Moodle are one of the most discussed and widely used systems throughout the US universities. Over the past years, MSU online course and hybrid course offering was increased. And most of the courses were using online components to engage with the students. Due to this trend MSU had to select or improve existing LMS system to a newer system, which would improve the e-Learning effectiveness and user satisfaction. In 2011, MSU decided to switch from old Blackboard 7.3 version to newer Blackboard 9.1 version. This decision was not one sided, MSU teachers and students were involved in deciding the best LMS system for them through number of trails between Moodle 1.9 and Blackboard 9.1. This research will try to identify if the selected (Blackboard 9.1) LMS system quality has improved the online learning effectiveness. There are many components in online LMS systems that directly improve the learning/teaching experience. Each new version of LMS has improved in quality compared with previous versions. The purpose of this study is to examine how the quality of LMS systems affects the eLearning effectiveness. Specifically, the study will focus on one primary objective: To determine how the quality of LMS systems affects e-learning effectiveness. In addition, this research will try to identify which LMS components have helped students and instructors to enhance e-learning experience. 6 Hypothesis Null hypothesis: Quality of LMS does not have an effect on eLearning effectiveness. Alternative hypothesis: Quality of LMS has an effect on eLearning effectiveness. Significance Online education has become widely used within the last 10 years by children as young as preschool age. It’s being implemented into basic curricula throughout the nation, from elementary schools to college campuses. By using a learning management system (LMS) some schools now have the ability to better serve their students through an integrated web-based platform of online education courses. Everything is all there from educational records, documentation of teacher appraisals, and even personalized content per student. There are webinars, assessments, discussion forums, and many more components in a LMS system that a student/teacher can take advantage of. From MSU point of view, this research will help the university departments to identify if there has being significant improvement in online course enrollments after moving to new LMS system from previous version. This research will also help to identify which components are widely used by teachers to interact with students. Information gathered from this study will help MSU instructors to identify and incorporate most important LMS components needed for students when creating online course contents. This will help to maximize the productivity of online courses. 7 These findings will be helpful for LMS vendors to identify major components that are helpful for students and what improvements should be integrated to those to improve the quality of services they are providing. Assumptions For this study, the research would have to be based on few assumptions. The following assumptions are made about this research and the circumstances in which it will be collected: 1. The data and information that will be obtained from the MSU personnel are unbiased and accurate. 2. The participants in this study, understand the surveyed questions and answers, and will follow the instructions to complete the survey. 3. Participants have sufficient technological skills to complete the survey. 4. Participants will volunteer to take the survey and will provide the most accurate answers based on their understanding of the questions. 5. Some participants may not have got the chance to participate in the pilot project for Moodle. 6. The treatments of all groups (Department, Instructors and Students) will be comparable in regards of methodology of the study. Limitations This study has some limitations. The study is limited to Morehead state university and it will also be limited to Blackboard Learn 7.3, Moodle 1.9 and Blackboard Learn 9.1, since these are the only Learning Management Systems used by Morehead state university. 8 Definition of Terms E-Learning : eLearning - short for electronic learning, is an umbrella term that refers to all types of training, education and instruction that occurs on a digital medium, like a computer or mobile phone. Distance Learning: Educational situation in which the instructor and students are separated by time, location, or both. Education or training courses are delivered to remote locations via online, including written correspondence, text, graphics, audio- and videotape. Learning Management systems (LMS) : Learning Management System (LMS) is a web based software application that is used to automate the administration of the E-Learning activities such as managing, tracking and reporting on course events. Blackboard(Bb) : Virtual learning environment and course management system developed by Blackboard Inc. It is a Web-based server software which features convenient course management, customizable open architecture, and accessible design that allows for integration with student information systems and authentication procedures. Moodle : Moodle (Modular Object Oriented Developmental Learning Environment) is a learning management system which allows for facilitators to create online dynamic websites which can be installed on either an established web server, a personal computer, or a web hosting company Online Course: Courses that are taught completely via Internet. Hybrid Course: Course uses both classroom and online instructional modes and meets face-toface for 25%-50% of the course contact hours prescribed by the course type and units. 9 List of Acronyms CMS : Course Management System LMS : Learning Management systems Bb : Blackboard Learn Moodle : Modular Object Oriented Developmental Learning Environment MSU : Morehead State University 10 Chapter 2: Literature Review Background ELearning Electronic learning covers a wide set of applications and processes, such as web-based learning, computer-based learning, virtual classrooms, and digital collaboration. It includes the delivery of content via internet, intranet/extranet (LAN/WAN), audio- and videotape, satellite broadcast, interactive TV, and CD-ROM. Thus, this definition defines eLearning as basically anything that is electronic. The global market for corporate eLearning will grow nearly 27%, compounded annually (McGee, 2004). IDC forecasts that the eLearning market, which was about $6.5 billion in 2003, will increase to more than $21 billion by 2008, and hit $52.6B by 2010. Figure 1Projected Corporate e-Learning Growth 11 Almost 3.5 million students were taking at least one online course during the fall 2006 term; a nearly 10 percent increase over the number reported the previous year (Allen, Seaman, 2007). In addition, the 9.7 percent growth rate for online enrollments far exceeds the 1.5 percent growth of the overall higher education student population (nwlink, 2013). Strengths of Online Learning There are many reasons why online programs have become a popular form of distance learning in higher education today. The online environment offers unprecedented opportunities for people who would otherwise have limited access to education, as well as a new paradigm for educators in which dynamic courses of the highest quality can be developed. Here is a list of some of the major benefits of online programs (ION, 2013): Anywhere…. The main advantage of asynchronous online learning is that it allows students to participate in high quality learning situations when distance and schedule make on-ground learning difficultto-impossible. Students can participate in classes from anywhere in the world provided they have a computer and Internet connection. In addition, the online format allows physically challenged students (and teachers) more freedom to participate in class. Participants access the Virtual Classroom through their computers instead of having to "go to class" physically. Anytime, Any Pace…. The Virtual Classroom is accessible 24 hours a day, seven days a week. Time efficiency is another strength brought by the online learning format. Asynchronous communication through online conferencing programs allows the professional juggling work, family, and study schedules 12 to participate in class discussions. There is no question about doing the work; just do it at the times that are more convenient. Students can access their courses at any time of day or night. Further, they have continuous access to lectures, course materials, and class discussions. This is particularly convenient for those who may need to reread a lecture or take more time to reflect on some material before moving on. Synergy The online format allows a dynamic interaction between the instructor and students and among the students themselves. Resources and ideas are shared, and continuous synergy will be generated through the learning process. Each individual can contribute to the course discussions and comments on the work of others. The synergy that exists in the student-centered Virtual Classroom is one of the most unique and vital traits that the online learning format possesses. High Quality Dialog Within an online asynchronous discussion structure, the learner may reflect on comments from others before responding or moving on to the next item. This structure allows students time to articulate responses with much more depth and forethought than in a traditional face-to-face discussion situation where the participant must analyze the comment of another on the spot and formulate a response or otherwise lose the chance to contribute to the discussion. Level Playing Field In the online environment, learners have a certain measure of anonymity. Discriminating factors such as age, dress, physical appearance, disabilities, race and gender are largely absent. Instead, 13 the focus of attention is clearly on the content of the discussion and the individual's ability to respond and contribute thoughtfully and intelligently to the material at hand. Access to Resources It is easy to include distinguished guest experts or students from other institutions in an online class. Further, today's students have access to resources and materials that may be physically located anywhere in the world. An instructor can compile a resource section online with links to scholarly articles, institutions, and other materials relevant to the course topic for students to access for research, extension, or in depth analysis of course content material. Weaknesses of Online Learning Equity and Accessibility to Technology Lack of access whether it be for economical or logistics reasons will exclude eligible students from the course. Computer Literacy Both students and facilitators must possess a minimum level of computer knowledge in order to function successfully in an online environment. Limitations of Technology User friendly and reliable technology is critical to a successful online program. However, even the most sophisticated technology is not 100% reliable. The Students 14 While an online method of education can be a highly effective alternative medium of education for the mature, self-disciplined student, it is an inappropriate learning environment for more dependent learners Lack of Essential Online Qualities Successful on-ground instruction does not always translate to successful online instruction. If facilitators are not properly trained in online delivery and methodologies, the success of the online program will be compromised. Levels of Synergy Online learning has its most promising potential in the high synergy represented by active dialog among the participants, one of the most important sources of learning in a Virtual Classroom. However, in larger classes (20 or more students), the synergy level starts to shift on the learning continuum until it eventually becomes independent study to accommodate the large class. At this point, dialog is limited as well as interaction among participants and the facilitator. The medium is not being used to its greatest potential. Learning management systems A Learning Management System (LMS) is a web based software application that is used to automate the administration of the E-Learning activities such as managing, tracking and reporting on course events. The types of learning delivered could be 100% online and a 15 combination of traditional and online learning, which is also called hybrid. Figure 1 illustrates the learning management system with its typical functionalities. Security Administ ration Tools Reportin g Assesme nt Capabiliti es Learning Management System (LMS) Skill Manage ments Content Accessibi lity Content Develop ment Contenet Integrati on Figure 2: Learning Management System and Its Functional Elements (Amirkhanpour, 2010) The main functional characteristics of a learning management system include administration tools, content accessibility, content development, content integration, skills management, assessment capabilities, reporting and security (Amirkhanpour, 2010). Administration tools: Enables administrator to manage user registration and define roles, manage content. Content Accessibility: LMS allows to access course contents through online. 16 Content development: Includes authoring, maintaining and storing learning materials. Content integration: Provide support to a wide range of external systems such as content management systems and registration modules. Skill Management: assessing the competency skills of the E-Learners using peer reviews and other feedback tools. Assessment capabilities: Provides assessment capabilities for the delivered learning content. Examples could be Assignments, Quizzes and Exams. Reporting: tracks all the learning and training efforts within the LMS and creates customized reports based on the courses offered, course completion/incompletion status. Security: Passwords and encryption methods are typical security features of an effective LMS. Examples could be user authorization, authentication, and data protection Over the past years LMS has become a powerful tool for higher education institutes. Its impacts have been felt mostly outside of traditional education institutions. LMS software is able to add a level of efficiency to higher education institutes’ learning systems, with a number of other benefits emerging as well, such as: Highly Flexible and Customizable Learning management systems are extremely flexible and customizable to meet your specific needs. Easily Create Courses 17 The integrated online registration software lets anyone create courses and training programs over the internet. You can instantly publish the course materials online for people to view and register for your upcoming classes. Anytime Access Since the whole learning experience is created and managed virtually, the system lets an individual (student/instructor) access the course materials, 24x7. This will allow students to go through their classes and courses in a more relaxed environment, wherever they wish. Robust Reporting and Tracking Capabilities Instructors can easily track and evaluate a course. The market for Learning Management Systems has grown into a billion dollar industry with hundreds of offering. 2012 annual Campus Computing Survey held by Campus Computing ProjectTM confirms the increasingly important role of Learning Management Systems as a core instructional resource. Overall, the percentage of college courses that use a LMS tool has risen from 15% in 2000 to more than 75% in 2012 (The National Survey of Computing and Information Technology, 2012). Figure 3 represents a profile of the LMS Market for Fall 2012 and Table 1 provides information regarding Institutional Demography of LMS providers for Fall 2012. 18 Figure 3: A Profile of the LMS Market, Fall 2012 (The National Survey of Computing and Information Technology, 2012) Table 1 : Institutional Demography of LMS Providers, 2012 (The National Survey of Computing and Information Technology, 2012) 19 Blackboard The Blackboard Learning System is a virtual learning environment and course management system developed by Blackboard Inc. It is a Web-based server software which features convenient course management, customizable open architecture, and accessible design that allows for integration with student information systems and authentication procedures. It may be installed on local servers or hosted by Blackboard ASP Solutions. Its main purposes are to add online elements to courses traditionally delivered face-to-face and to develop completely online courses with few or no face-to-face meetings (Education Week, 2013). The Blackboard Learning System provides users with a platform for communication and sharing content. Communication Announcements: Professors and teachers may post announcements for students to read. These can be found under the announcement tab, or can be made to pop-up when a student accesses Blackboard. Chat: This function allows those students who are online to chat in real time with other students in their class section. Discussions: This feature allows students and professors to create a discussion thread and reply to ones already created. 20 Mail: Blackboard mail allows students and teachers to send mail to one another. This feature supports mass emailing to students in a course. Content Course content: This feature allows teachers to post articles, assignments, videos etc. Calendar: Teachers can use this function to post due dates for assignments and tests. Learning modules: This feature is often used for strictly online classes. It allows professors to post different lessons for students to access. Assessments: This tab allows teachers to post quizzes and exams and allows students to access them via the internet Assignments: This features allows for assignments to be posted and for student to submit assignments online Grade Book: Teachers and professors may post grades on Blackboard for students to view. Media Library: Videos and other media may be posted under this function. Students and faculty may benefit from course management systems such as the Blackboard Learning System. Potential benefits include: Increased availability Quick feedback Improved communication Tracking 21 Skill building. Moodle Currently acting as one of the most popular e-learning systems available for use, Moodle (Modular Object Oriented Developmental Learning Environment) is a learning management system which allows for facilitators to create online dynamic websites which can be installed on either an established web server, a personal computer, or a web hosting company (“About Moodle”, 2013). Moodle is extremely customizable, as it exists as a free, open source package. This means that a e-learning instructor, so long as he or she has the necessary computer skills, can fully customize the utility as necessary. This also means that the software can be scaled to fit of needs of a variety of institutions, ranging from a single instructor course, to a full 40,000 student university setting (Beatty and Ulasewicz, p.36, 2006). Moodle comes packaged with activity modules, which are the bases with which to develop systems such as forums, databases, and wikis which can all be used to construct collaborative communities within which students and instructors can discuss course material. These modules also allow instructors to deliver course content to students and provide coursework and various assessments to students. Figure 1, below, demonstrates an example of an opening page of a Moodle based learning system. This helps to demonstrate the various types of activities and content available to students upon beginning a class session using this e-learning system. 22 Figure 4: Example of a Moodle opening page for a course. This helps to demonstrate the capabilities of what can be accessed through a Moodle-based e-learning system (Beatty and Ulasewicz, p.36, 2006). One major advantage which Moodle has gained over Blackboard is its lack of a licensing requirement. Currently, in order for any institution to make use Blackboard, they must purchase a licensing agreement. This makes Moodle a cheaper alternative to Blackboard from a business standpoint. This does not necessarily make Moodle a “free” program, due to the need to consider administration and maintenance costs for the system, however compared to Blackboard’s yearly licensing fees, this is typically much more affordable (Carvahlo et al, 2011). Moodle Compared to Blackboard 9.1 Moodle and Blackboard are both popular online learning platforms with which educators can develop complete online course that can include multimedia content. 23 Moodle is an Open Source Learning Management System that is provided freely and can be run on many operating systems. Its “free to download, change, share, improve, and customize to whatever you want it to be,” according to the Moodle website. Therefore, any educator can use it to build or supplement a course. Blackboard on the other hand is a proprietary Learning Management System and its use is typically limited to educators at institutions which pay a fee each year to take on a license agreement for its use. Blackboard Learn 9.1 is only one of many offerings from the software company Blackboard, Inc. Improvements in its uses for higher education include course wikis (Moodle improved theirs too), blogs and journals that stimulate conversation and reflection on a course, and group tools that make group collaboration and communication easier than the previous version. Its most notable feature is its Web 2.0 interface, which makes it easy for educators to navigate when adding content to an online course and for students to navigate when accessing course content. Features & Functions: Both of these tools have a lot of different functionality available, either natively, or through add-on types of functionality. If different functions are going to be the deciding factor in selecting one of these versus the other, this will have to be a decision made by the institution based on its ultimate needs. Cost: This is clearly different. As an open source product, Moodle is simply less expensive. Blackboard is sort of the “Rolls Royce” of today’s LMS, and there are users of the product who would tell you that if you want the best LMS money can buy, you should make the financial commitment to Blackboard. On the other hand, if you want a premier product for a much lower cost, Moodle is really the way to go. Another thing to be aware of is that Blackboard builds 24 substantial annual increases into their pricing model, since they are continually procuring and integration additional products into their offerings, with the intent of adding value for their users. Product/vendor model: Moodle and Blackboard are very different products with very different vendor models. One is open source, and there are many support and service vendors to choose from, while the other is proprietary and there is just the one company to work with. How that impacts your decision is up to you and your institution to determine (EmergingEdTech, 2013). A brief historical review In 2005 a grant from the US Department of Education funded Boston College (BC) to conduct a research study with many aggressive goals that would, if met by the ten collaborative state partnerships, result in a sustainable, systematic delivery of online professional development (OPD) for teachers. The e-Learning for Educators research efforts focused on assessing the effects of OPD on three areas: (1) improvements in teacher content knowledge; (2) improvements in instructional practice; and (3) subsequent improvements in academic achievement for students. Assessing improvements required the assistance of a team of researchers and evaluators from inTASC, Boston College’s Technology Assessment and Study Collaborative Center. Education Development Center (EDC) developed twelve ELA and math online courses used for the research series and provided the training for facilitators who delivered the workshops to teachers participating in the research study. This study took place in the spring of 2007 and continued through the summer of 2009. Out of the 21 participating states there included 369 teachers and 21,217 students that were subject to the studies. The scale of participation was large and the quantity of data collected and analyzed 25 was extremely large. For example, more than 21,500 open-ended responses were analyzed in addition to hundreds of multiple-choice questions. There were no existing systems that could capture the types and amount of data that this research effort required so Boston College designed an online data collection system specifically for the e-Learning research effort. Ultimately, teachers completed pre and post surveys, as well as administered student surveys and tests over a period of three semesters (Alabama Public Television, 2013). The collected data were subjected to appropriate statistical tests (ANOVA, ANCOVA, HLM2) to determine both statistical impact and the practical educational impact (effect size). These statistical analyses examine whether a measurable effect on teachers or students was happenstance or is a result of taking the OPD courses. These study results demonstrate that the e-Learning courses and process can have a positive effect on teacher knowledge, teacher instructional practices, and, most importantly, student achievement. Overall, both teachers and students made statistically significant improvements. In other words, there is some evidence of a cause-and-effect change due to the OPD instruction; improvements did not occur by chance. Most of the previous studies on LMS systems are based on comparisons between Blackboard and Moodle. Generally, these types of studies show a preference to Moodle. In a study in Hong Kong in 2005, it was reported that out of 84 education students, 45% preferred the use of Moodle over Blackboard, while 49% showed no preference between the two systems. In this particular study, Moodle was favored due to it have a superior ability to promote interaction among students and instructor, however, users tended to prefer Blackboard in terms of organization of the course’s instructional materials (Carvahlo et al, 2011). 26 This is a stark contrast to another study completed by Machado and Tao (2007) when course material organization and communication were rated higher in Moodle over Blackboard. In this particular study, involving 53 students, 71% of the students found that Moodle was easier to use that Blackboard, with 75% stating they would prefer the use of Moodle as a replacement for Blackboard in the future (Carvahlo et al, 2011). Carvahlo et al (2011) conducted further research comparing Blackboard and Moodle among students in Portuguese university and found that 46.5% of students preferred the use of Blackboard while only 34.7% preferred Moodle (nearly 20% of the students surveyed had no preference). However, student assessment of various e-learning features, including the visual appearance, organization of course materials, user interface, etc, favored Moodle. These results tended to favor Moodle, more so, when students identified themselves as being familiar with the use of both systems. Rationale Currently, not many studies exist which studies how quality of LMS affects the eLearning Effectiveness. From a faculty/instructor perspective, there is a need for more direct research behind preference between Moodle and Blackboard systems. Although there tends to be a lack of direct information pertaining specifically toward this group of users, some information has been provided which shows more personal insight into the preferences between the two systems. Beatty and Ulasewicz (2006) wrote a paper together describing their experiences with the two programs. According to Beatty, he found the intstructor interface for Moodle to be “more intuitive and easy to use than the instructor interface in Blackboard” due to the improved editing tools supplied with Moodle. Ulasewicz agreed that the instructor interface may have been superior in stating that it was easier for her to instate a gradable forum system 27 than she felt it would have been using a Blackboard system. Beatty found issues with his students being able to edit posts in discussions, while Ulasewicz found no major issues with the system on Moodle. Finally, both commented that due to the open source nature of Moodle, there are times that’s it can be preferred when it pertains to fixing and creating new features needed for their environment. Overall, more information is needed to determine if there is indeed a preference among students and faculty as to each of the features and overall use of Blackboard and Moodle. This information could be beneficial to institutions, such as Morehead State University, who have been undergoing changes within their e-learning programs and will empower them to make a more beneficial decision as to which program to instate. From MSU point of view, this research will help the university departments to identify if there has being significant improvement in online course enrollments after moving to new LMS system from previous version. This research will also help to identify which components are widely used by teachers to interact with students. Information gathered from this study will help MSU instructors to identify and incorporate most important LMS components needed for students when creating online course contents. This will help to maximize the productivity of online courses. These findings will be helpful for LMS vendors to identify major components that are helpful for students and what improvements should be integrated to those to improve the quality of services they are providing. 28 Chapter 3: Methodology In the previous section, an introduction to the proposed area of research was presented. This included a description of the study’s research problem, several research purposes, hypotheses that will serve to direct the data analysis, and an identification of several terms key to this research. In addition, a review of relevant literature related to the eLearning, Learning Management systems, Blackboard and Moodle established a background of support for the study. Previous related studies have been studied extensively, to identify any related information. Purpose of this study is to examine how the quality of LMS systems affects the eLearning effectiveness. The intent of this section is to describe the methodology that is proposed for this research effort. Included in this section will be a description of the proposed research design, study sample, and proposed data collection methods, procedures, and analysis efforts. Research Design Research plan will be divided into six main steps including: Research Planning, Survey development and deployment, Data analysis, findings, Conclusions and Implications. 29 Figure 3: Research Methodology The first step in this research will start with planning of the research. In this step the study will involve an extensive literature review. The purpose of the literature will be to study and plan the research. In the planning phase study setting and populations and sampling plan will be identified. Study Setting The proposed study will be conducted in Morehead State University. The study will consist of three settings. First setting will focus on Blackboard Learn 7.3, Second will focus on Moodle and third setting will focus on Blackboard Learn 9.1. Population and Sampling Plan Study will be conducted through all the Academic departments in Morehead State University. Colleges: Caudill College of Arts, Humanities, & Social Sciences 30 College of Business & Public Affairs College of Education College of Science & Technology University College 1 Department representative, 10 Instructors, and 20 Students from each department will be selected for the study. A sample of the population of students and instructors at each department in Morehead State University will be used to draw conclusions about the entire population of students and instructors. Instrumentation Self-administered questionnaires will be used as the instruments in this study. Following three main categories will be taken in to consideration when creating the questionnaires. A. Social Issues : Learner Perspective a. Learners Attitude toward LMS b. Learners Computer Anxiety c. Self-Efficiency d. Enjoyable Experience e. Interaction methods with other students and Instructor : Instructor Attitudes a. Responsiveness b. Enjoyable Experience c. Availability 31 d. Promptness e. Usefulness f. Fairness g. Communication Ability h. Interaction methods with students B. System Quality : Technical Quality a. Easy to use b. Security features c. User Friendly d. Stable e. Access speed f. Help manual availability : Content Quality a. Course management b. Well organized c. Content Flexibility ( Can move content from one course shell to another) d. File types supported e. Availability of editing options. : Service Quality a. Tracking options with report generating availability b. Administration 32 C. Supportive Issues a. Environment b. Trend Questionnaires will be divided in to three main sections based on the respondents. Department Instructors Students Sample Survey Questionnaire for Departments is attached in the Appendix A. Surveys will be administrated through free online questionnaire and survey provider, Survey Monkey. Data Collection Methods Data will be collected from surveys of departments, faculty and students using online survey tool Survey Monkey and later collected data will be entered into Excel spreadsheets for review. Data Analysis Methods The data analysis will be processes in six general phases. The first phase consists of parsing the gathered data into Excel spreadsheets for further analysis. Each spreadsheet will be allocated to Department, Instructor and Student respondent data. The second phase will be filtering the gathered data for further analysis. This will be filtered based on the criteria: before 2011 and after 2011.The third phase consists of removing any invalid data and filtering data for further analysis. The fourth phase consists of grouping the response data into related question categories. The fifth phase consists of calculating total and sub totals for each categories and calculating the percentages for collected response data. 33 Final phase will consist of general analysis of the results and to conduct statistical analysis for determining the acceptance or rejection of the hypothesis of Research question. 34 Appendix A – Sample Survey Questionnaires Survey Questionnaire for Departments 1. Name of the Department The next several questions ask about online distance learning courses 2. Which year did your department begin to offer online distance learning courses? 3. What is the estimated percentage of total courses offered by your department before 2011 that were online distance learning? a. Less than 5% b. 5-10% c. 10%-20% d. More than 20% 4. How did the number of online distance-learning course increase after 2011? a. Decreased or no change b. Increase 1-9% c. Increase 10-24% d. Increased 25%-50% e. Increased more than 50% 5. What is the estimated percentage of tenure-track faculty members who taught online distance-learning courses before 2011? a. None b. 1-4% c. 5-9% 35 d. More than 10% 6. What is the estimated percentage of tenure-track faculty members who taught online distance-learning courses after 2011? a. None b. 1-4% c. 5-9% d. More than 10% 7. What is the estimated percentage of students who took online distance-learning courses before 2011? a. None b. 1-9% c. 10-24% d. More than 25% 8. What is the estimated percentage of students who took online distance-learning courses after 2011? a. None b. 1-9% c. 10-24% d. More than 25% The next several questions ask about hybrid courses, where the instructor combines the elements of online distance-learning courses and traditional course to replace some classroom session with virtual sessions. 36 9. What year did your institution begin to offer hybrid course? 10. What is the estimated percentage of total courses offered by your department before 2011 that were hybrid? a. Less than 5% b. 5-10% c. 10%-20% d. More than 20% 11. How did the number of hybrid course increased after 2011? a. Decreased or no change b. Increase 1-9% c. Increase 10-24% d. Increased 25%-50% e. Increased more than 50% 12. What is the estimated percentage of tenure-track faculty members who taught hybrid courses before 2011? a. None b. 1-4% c. 5-9% d. More than 10% 13. What is the estimated percentage of tenure-track faculty members who taught hybrid courses after 2011? a. None b. 1-4% 37 c. 5-9% d. More than 10% 14. What is the estimated percentage of students who took hybrid courses before 2011? a. None b. 1-9% c. 10-24% d. More than 25% 15. What is the estimated percentage of students who took hybrid courses after 2011? a. None b. 1-9% c. 10-24% d. More than 25% The next few questions ask about students taking online distance courses, hybrid courses, and traditional courses with LMS activities. 16. How challenging was it for the department to support students' online distance courses, hybrid courses, and traditional courses with technology activities, in the following areas before 2011? Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge a. Students lack of knowledge about technology b. Network access/usage problems c. Utilizing online course technology d. Utilizing online course application tools 38 17. How significant a challenge is it for the department to support students' online distance courses, hybrid courses, and traditional courses with technology activities, in the following areas after 2011? Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge a. Students lack of knowledge about technology b. Network access/usage problems c. Utilizing online course technology d. Utilizing online course application tools The next few questions ask about instructors taking online distance courses, hybrid courses, and traditional courses with LMS activities. 18. How challenging was it for the department to support instructors' online distance courses, hybrid courses, and traditional courses with technology activities, in the following areas before 2011? Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge a. Faculty’s lack of knowledge about technology b. Faculty’s lack of confidence to use technology in teaching environment c. Unreliable technology: Network/software crashes during teaching session. d. Instructors’ lack of knowledge about how to design course contents incorporating LMS. 19. How significant a challenge is it for the department to support instructors' online distance courses, hybrid courses, and traditional courses with technology activities, in the following areas after 2011? 39 Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge a. Faculty’s lack of knowledge about technology b. Faculty’s lack of confidence to use technology in teaching environment c. Unreliable technology: Network/software crashes during teaching session. d. Instructors’ lack of knowledge about how to design course contents incorporating LMS. 40 References Amirkhanpour, M. (2010). Learning Management Systems and E-Learning within Cyprus Universities. ION. (2013). Retrieved from http://www.ion.uillinois.edu/resources/tutorials/overview/weaknesses.asp nwlink. (2013). Retrieved from http://www.nwlink.com/~donclark/hrd/elearning/tools.html (2012). The National Survey of Computing and Information Technology. The Campus Computing Project. Carvalho, A., Areal, N., & Silva, J. (2011). Students' perceptions of Blackboard and Moodle in a Portuguese university. British Journal of Educational Technology, 42(5) Despotovic-Zrakic, M., Markovic, A., Bogdanovic, Z., Barac, D., & Krco, S. (2012). Providing adaptivity in Moodle LMS courses. Journal of Educational Technology & Society, 15(1), 326-338. About Moodle. (2013, January 27). Retrieved from https://moodle.org/about/ Beatty, B., & Ulasewicz, C. (2006). Faculty perspectives on moving from Blackboard to the Moodle learning management system. TechTrends: Linking Research & Practice to Improve Learning, 50(4) Alabama Public Television. (2013). Retrieved from http://www.aptv.org/aptplus/ELearning/impact.asp Drake University. (2013). Retrieved from http://oit.drake.edu/advantages-blackboard-9/ 41 Education Week. (2013). Retrieved from http://www.edweek.org/dd/articles/2008/06/09/01moodle.h02.html EmergingEdTech. (2013). Retrieved from http://www.emergingedtech.com/2010/11/moodle-2versus-blackboard-brief-comparison/ The Blackboard Learning System. (n.d.). Retrieved from http://uupinfo.org/research/working/bradford.pdf