A Distributed Adaptive Learning Environment Qiangguo Pu2, Harris Wang1, Oscar Lin1, Nikos Mastorakis3 CCIS, Athabasca University, 1 University Drive, Athabasca AB T9S 3A3, Canada 2 Computer Center, University of Science and Technology of Suzhou 298 Binhe Road, Suzhou, Jiangsu 215011, China 3 Hellenic Naval Academy, Terma Hatzikyriakou, 18539 Piraeus, Greece Also: WSEAS, Ag.I.Theologou 17-23, 15773, Zografou, Athens, Greece mastor@wseas.com 1 Abstract: - The World Wide Web is evolving into a virtual environment where people work, shop, play, and learn. The early WWW learning systems were generic and aimed at a general class of student. DALE (Distributed Adaptive Learning Environment) is an evolving learner-centered system for learning and teaching computer science at a distance. Older generations of electronic learning systems have a central server with thin clients for computer-mediated communications. DALE adds much more functionality. DALE is a large-scale Internet-based distributed information management system with the many merits of distributed systems. DALE includes not only a central server for cooperative systems but also peer-to-peer server components and local modules similar to the emerging 3 D gaming environments for processing simulations/multimedia. Furthermore, DALE integrates agents and learning objects in the context of a distributed system. This paper covers the design and implementation of many aspects of DALE. Keywords: - Distance Education, Computer Science Education, Distributed Adaptive Learning Environments, Learner Centred 1. Introduction The opportunity for life long learning is a prerequisite for modern society. Information technology supports the creation of stimulating and complete learning experiences occurring at the time and place of the learner’s convenience. Athabasca University (AU) believes that due to the World Wide accessibility, storage capabilities, and its relatively easy to use standard ways of multimedia publishing one of its potential greatest uses is as an interactive learning environment (Holt, Gismondi, Fontaine, and Ramsden, 1995). The goal of AU (http://www.athabascau.ca) is to provide lifelong open learning (Race, 19942) by reducing barriers to education. Generally, students can take the courses at the time and place of their convenience. AU serves about 25,000 students a year. Most of the students are part-time. AU offers about 420 undergraduate courses and 25 undergraduate degrees. The University also has five graduate degrees and serves about 1,500 graduate students annually. Most of the Masters courses are paced with fixed start-dates, but most undergraduate courses can be entered at any time in the year, and students have six months at their own pace to complete all the assignments and the final examination. The Centre for Computing and Information Systems (CCIS) (http://ccism.pc.athabascau.ca) at AU offers four undergraduate credentials and an MSc IS shaped primarily by ACM curricula, professional requirements, and student needs.. Most of the 1600 undergraduate students are part-time, generating about 2400 course registrations per year. It is anticipated that in two years the Masters in IS will have about 150 students per year generating about 450 course registrations per year. CCIS has been leading AU in developing an online learning system. In the past, the separation by space and time of the AU students diminished the effectiveness of distance education approaches. Informal peer support and group work efforts were particularly restricted, with students often feeling lost in their attempts to deal with new endeavors in isolation. Now computer technology provides a new and innovative approach to open learning. E-mail, computer based conferencing, structured hypertext and the virtual reality technologies change the nature and enhance the quality of distance education (Pea, 19933). Starting in 1995 (Holt et al, 19954), CCIS has implemented 24 undergraduate and a number of graduate courses for World Wide Web (WWW) delivery. Students at home and work are supported with e-mail, chat, and computer conferencing. Led by CCIS other centers of AU have become increasingly reliant on electronic delivery. Our goal is to develop an integrated system for course development and delivery. Before describing that endeavor how we will describe some underlying concepts and technologies of our endeavor. Experiential thought is easily integrated into computer based learning systems. Students are continually interacting with the computer and with one another through the computer. But reflective thought is more 2. Technologies and Concepts Underlying difficult concept to tie directly to the computer based Design, Delivery, and Evaluation learning system. Reflective thought in our curriculum occurs in analysis and design and generating of algorithms. It is nurtured by interaction with students and Learner-Centred Design tutors and by access to appropriate computer based tools. Students who grew up with Internet gaming expect In our system design we define coordinator sophisticated use of graphics and multimedia to requirements, tutor requirements and learner supplement their courses. We have begun adding Java requirements. In this paper we focus on the learner. applet and animated simulations for explication of Curriculum and course design in CCIS is learnerdifficult material --particularly data communications, 5 centered (Norman and Spohrer, 1996 ). Thus a key computer networks, and distributed systems. A balance feature of the requirements definition are the needs of the between multimedia and electronic text is required. learner (Holt et al 20017) which we abstract from our Multimedia material places a burden on resources must experience with focus groups, case studies from fieldbe limited within current Internet bandwidth and testing, some questionnaire evaluations, anecdotal personal computers owned by average students. reports, and the literature. The systems should be designed to optimize the use of the learner’s time not the Learning Objects convenience of the institution or instructor. Learners must be involved in the conceptualization, design, Our long-term goal is to create a body of electronic development, implementation and evaluation of the instructional support tools, curriculum content, and environment. In the design of our system we begin with design strategies from which, on the basis of learner the assumption that the learning environment should needs, we can select materials for a particular course or belong to the learner and as much as possible be under module (Holt, 199814). According to the Sharable the learner's control. Users learn quickly and gain a fast sense of mastery when they are placed ‘in charge.’ The Content Object Reference Model (SCORM, 200015) a learning environment should be open to exploration but learning object is modeled as the smallest stand-alone must have some constraints or it is likely the learner will and meaningful component of a course that is not feel comfortable. It is important to optimize the interoperable, modular, and discoverable. We have learner’s privacy and security. If users control their own modified that description slightly to allow a hierarchy of information, privacy will be enhanced. As much as learning objects with any dependencies amongst objects possible, learners’ data should reside on their own specified. The highest-level learning object is the course machine. In other cases it should be behind proper itself. Our design uses the eXtensibe Markup Language firewall and guarded with other security features. All (XML) to define our learning. The IMS project is materials originating on the institutional server must be developing standards for content packaging so that virus free. Institutions should provide consultation on learning objects can be easily shared. The metadata privacy and security issues. An assumption of our describes and specifies the use of learning objects. approach is that learning environments can be Different XML schema may define what is a valid challenging but still provide a relatively low stress learning object structure at different levels of our course psychological experience for the user. Psychological and hierarchy. Much research on defining, constructing, and social concerns generated by a stressful technological building and displaying learning objects has been environment can be inimical to the learning process. conducted over the past few years. The CAREO project defines learning objects to include "simulations, tutorials, The learning environment should optimize the various drill and practice modules, content databases, multiapproaches to learning available to the learner such as media exercises" (Downes, 200016). There remains apprenticeship learning (Collins, Brown, and Newman much work on development of an ontology for subject 19898), guided discovery (Holt et al, 19959), peer domains, an ontology for instructional design, definitions 10 of the granularity of learning objects, methods for tutoring (Greer et al, 1998 ), situated learning (Brown, 11 combining learning objects into courses, and several Collins, and Duguid, 1990 ), case-based learning and other issues. 12 collaborative learning (Harasim, 1993 ). These can often be accommodated by instructional design although Distributed Adaptive Learning Environments particular approaches can be facilitated by software (e.g. peer tutoring Greer et al, 199813) and some require some We define the learning environment as consisting of all special capabilities (e.g. computer conferencing). the interaction with materials, the tools, the interactions with peers, and the interaction with tutors, access to other materials, and even the approaches to learning available. In a distributed learning environment the various resources are geographically distributed but connected by information technology. The World Wide Web offers the most successful model of a distributed learning environment. Originally web based learning was restricted to basic to text and images presented through HTML. With the rapid increment of bandwidth available on the web these have been supplemented with audio, video, animations, and simulation. The web has become a virtual environment – a place where people work, play, shop, and learn A basic premise of the CCIS distributed learning environment is that our learners are best served by an underlying distributed architecture with much of the processing occurring on the learner’s computer. Advanced Internet gaming activities are based on such a distributed model. There are services that must be offered centrally but many functions are best performed locally (URL re desktop). A local component best provides an environment supporting learner autonomy, empowerment, and privacy. Learners can provide services to other learners in a peer-to-peer network ala Napster (www.napster.com). A distributed learning environment facilitates a learner-centered educational paradigm and promotes active learning. Our model of web-based distributed learning focuses on a model of guided self-discovery in which learners engage in learning activities at the time and place of their convenience. However, where it best fits the learners’ needs we also use the technology a more traditional “paced” approach where the learners are part of a cohort group with a fixed schedule for all assignments etcetera. We also used hybrid models that fall between these two models. The term adaptive learning environment refers to technology that will adapt the environment in various ways. First, CCIS wants its content to be reusable across various modules and courses. XML combined with international standards such as IMS (see www.imsproject.org) provides a standard way of storing learning objects making them available for use in a variety of ways. Second, content must be able to be delivered across different platforms. XML, XSLT, and related Java technologies provide the means for transforming material for presentation on various devices such as desktop computers and wireless handheld devices. Third, CCIS wants the content to be dynamically adaptable to the needs of particular learners. Intelligent software agents (Lin and Holt, 200117) provide the key intelligence for a wide range of adaptations. Finally, standards such as IMS and a variety of tools help make material adaptable to the needs of the authors, instructors, and the educational institutions. Agent Based Systems The agent-based computing paradigm is particularly suitable for developing the web-based distributed adaptive learning environments (Lin and Holt, 200118). A software agent is a higher-level system component, defined around a particular function, or utility, or role in the overall system and it is more autonomous than a simple object. The autonomous nature of software agents makes them ideal for implementing a distributed learning environment. Intelligent agents make the environment adaptable. There has been considerable exploration of agent technology applications for education for example: multi-agent approach to the design of peer-help environment (Vasileva et al, 199919); agents for information retrieval (Hiltz and Wellman, 199720); agents for student information processing, distribution, and feedback collection (Huhns and Mohamed, 199921); pedagogical agents (Johnson and Shaw, 199722); teaching agents (Selker, 199423); tutoring agents (Solomos and Avouris, 199724); agents for assignment checking (McCollum, 199725); agents for student group online support (Whatley et al, 199926). An empirical study evaluating the effectiveness of intelligent agents in online instruction has suggested that the application of agent technology to online learning hold promise for improving completion rates, learner satisfaction, and motivation (Thaiupathump, 199927) From the Web-based educational experience, we know that the students need to download course materials get dynamic updates to downloaded materials connect to obtain and interact with central dynamic materials serve materials to other students/tutors submit assignment to tutors ask tutors for course-related questions interact with other students have optimal control over their own information/tools etc Tutors’ tasks are answering questions from their students; knowing student profiles, especially those that need contacting; checking students’ assignment; contacting other course tutors to answer students’ questions sometimes. The primary teaching tasks of professors are to develop and update course materials serve as course coordinators contact to know students’ performance and feedback about the courses understand student’s profiles taking their courses some course-related agents. The system includes four course-related agent classes: content agent class, collaboration agent class, tutoring agent class, and evaluation agent class. For instance, a content agent has a goal that is to keep the course content updated and fit best to the students. It realizes its goal by continuously monitoring the links for the course and notifying the students taking this course when the links have updated or changed. More importantly, the content agent is able to proactively and adaptively generate the content structure for the student according to the knowledge structure of the course and students' performance. It accomplishes this task by collaborating with a curriculum management agent that is a server agent and is responsible for determining the knowledge structure of a course. Tutor agents: Tutor agents are responsible for helping tutors to provide on-line tutoring to students. Their role is to interact with student agents to receive and answer students’ questions, to do some marking work, and to contact professor agents and other tutor agents within the collaborative learning environments. The tutor agents can further monitor student’s interactions6. Professor agents: Professor agents’ role is helping human professors do course coordination, offer course materials by creating and maintaining course material databases, make examination papers, and offer solutions to exercises of courses. A professor agent may need to inquire into students’ learning performance from tutor agents and profiles from students profile database to design adaptive learning materials. From system point of view, the requirements for the architecture are: Distributed: In fact, the Internet is the largest distributed systems. Distributed systems use multiple computers to solve a problem and provide a common, consistent global view of the database system, name space, time, and security, and access to resource. This can be done to increase performance, improve reliability and scalability, or support multiple geographical locations. For instance, to ensure system scalability, in our Figure 1: User case diagrams for distributed learning infrastructure we deploy a set of special agent environment servers rather than a single one. These servers identify agents that provide services that are the Therefore, we have three types of agents: student agents, same as or similar to a requested service. tutor agents, and professor agents. These agents’ roles are Mobile agents-based: Multi-agent systems are described as follows: subject to performance bottleneck in cases where agents cannot perform tasks by themselves due to Student agents: Student agents can be hosted by central insufficient resources. A mobile agent approach is site for more complex updates. Each student taking an much less susceptible to nagging client/server online course will have an interface agent that actually is a network bandwidth problem, network traffic, collection of more or less independent smaller agents, each transaction volume, number of clients and servers, having an associated visual presence, downloading initially and many other factors. Furthermore, the from the server, operating in the background, watching downloading of aspects of our course materials progress, measuring it against the plan, and taking remedial gives us a great environment for testing distributed actions when necessary. These smaller agents are typically agents and mobile agents. We enhance the existing course materials download system with mobile agent-based course material disseminators. These mobile agents embody the so-called "Internet push model". They can disseminate information such as news and automatic course material updates for course instructors. The agents can carry the new materials as well as installation procedures directly to the students’ personal computers. These agents can manage material on the computer creating personalized learning materials. Incorporation of Legacy-Systems: For those components in use, our strategy is to incorporate them as far as possible by giving them wrappers. Incorporating these components into different agents’ structures and integrating them with new developed components would make our system more cost-effective. Each student agent has currently incorporated a course material manager, a VHD, a WWW Conferencing system, an E-mail system, and a white board. Each tutor agent has been equipped with a VHD, a WWW conferencing system, a white board, and a TRIX, and Course material manager. Each professor agent uses a WWW conferencing, an E-mail, a white board, a TRIX, and a course material manager. 3. An Integrated Architecture for a Distributed Adaptive Learning Environment CCIS academics have focused much of their research our Distributed Adaptive Learning Environment (DALE) project. The DALE project explores the issues in creating a learner-centered platform for electronic distance learning, particular for a computer and information systems programme. The DALE project is based on a distributed processing architecture to take full advantage of the powerful student client computer (Gelernter, 200028) and investigate the potential of peer to peer computing (Oram, 200129). Course development system Course Server Course delivery system Figure 2: The whole picture - course authoring, course delivery and course server It is important that there be an overall integrating architecture for organizing the various functions and the associated technology (see Figure 2). It must be capable providing a stimulating, interactive informative environment for situated learning, peer-to-peer collaboration, and communicating with tutors (DALE Research Team, 200231). It should support tutors facilitating student learning and support professors in designing, creating, and delivering their course materials. The development and delivery systems are both critical and enhances the functionality of both systems. For example, feedback from the delivery informs timely revision of course materials. However, this paper focuses on the delivery system and the student-learning environment. Course Server As can be seen from Figure 1, the course server plays some important roles in the whole integrated system. On the development side we anticipate agent technology will be use to implement courses comprised of learning objects (Lin, Holt, Korba, and Shih, 200132). To enable computer agents to automatically and dynamically compose personalized course materials for an individual learner, we need some effective knowledge management mechanisms and to include instructional design information in the metadata. Presumably learning objects may include their own agents and our system agents would have to negotiate with these “foreign agents. For delivery it must be able to store a variety of learning objects consisting of text, image, graphs, audio and video clips, and use them for composing courses. The course server serves both general web browsing clients such as Netscape and Internet Explorer, and our specialized system. The server has a JASIG u-portal (http://www.jasig.org/), which will include standard channels for online testing, external links, agent enhanced conferencing, agent dispatch, and student white pages (and yellow pages) for peer-to-peer networking. An agent maintains the link database removing broken links and notifying students’ agents of links of particular interest to their owner. Some of these services will be presented in the browser; some may interact directly with agents on the student’s computer. Some special attention must be paid to course server support for data encryption, user authentication and some more complex security operations, as well as support for the implementation and deployment of intelligent mobile agents within the system. A course delivery module on the server is designed to link course servers to course clients running on student’s computers. This module plays four important roles. The first role is to generate individualized course contents for each individual student based on the course material and students' information in the student database. Some course materials such as assignments, external links and other dynamic course material may be kept on the central course server. The second role is to serve the centrally delivered materials. Other materials are downloaded to ICSSL. The third role is to deliver the individualized course materials to the students. The fourth role is to manage the updating of downloaded materials). To reduce dependence on a live connection and enhance mobility the bulk of the course materials are distributed to the student’s machine along with a student-learning module (SLM). A special mobile agent initially resides at the educational institution free the institution. It watches for connections from students. When there has been an update of the course materials since that last connection from a student it then makes a copy of itself, and travels to the student's computer with updating files specific for the identified student. It frees the learner from paying attention to and spending time on updating and maintaining course materials. Distributed Sub-systems There are three distributed sub-systems: the sub-system for course coordinators (SCC), the sub-system for course tutors (SCT), and the sub-system for student learning (SSL). These systems share some modules such as the course messaging module, the course conferencing module, and the shared workspace module. Then the SCC has a module to help coordinators make decisions on student’s evaluation and other academic-related matters involved in student management. The SCT has a module to help manage the duties of answering student questions in a timely fashion and demonstrating course contents to students. It includes a scheduler built in to help tutors to schedule their work. Much of the rest of this paper will deal with exploring the SSL as it is a relatively new and original concept for web-based learning environments. The SSL (see Figure 3) consists of the learning management module, the course-messaging module, the course conferencing module, and the shared workspace module. The learning management module is designed to help students download and digest course materials, and access external documents. Standards-based educational material can be processed in a variety of ways according to educational features and passed to an appropriate viewer (for instance the XML enabled mozzilla embeddable browser, a multimedia viewer, or a simulations player). The browser will support shared browsing of course materials between two or more students (or a tutor and one or more students). There are many multimedia-viewing programs but they do not include many of the functions useful for learning such as tools for testing the recognition of specific situations in a video clip. This module includes an interface to the many tools a learner might use for his course. Finally, since it supports many functions at a local level, ISSL allows mobile users to accomplish their learning on the move. With the course messaging module students can communicate with course coordinators and tutors and submit their assignments directly without having to switch to other mail readers. Instant messaging, chat, audiochat, and video chat will be optional sub-modules of this module. The course conferencing module provides a posting management system for course conferencing. Finally the workspace-sharing module allows students to share files during collaborative work. Overall the SSL will be an ideal learning environment for collaborative work and will support professional practices such as extreme programming. Student Data Course Database Tutor Data Central Conference Data Course Server Course Messaging Module Email Data Shared Files Workspace Module Learning Management Module Downloaded Course Data Learning activity data Course Conference Module Cached Conference Data Course Presentation Module Figure 3: Sub-System for Student Learning (SSL) appropriate research focus for a computer science Lin and Holt, 200133, have outlined how distributed program at a distance education university. However, it intelligent agents can be incorporated to enhance the is clear that neither a single department nor even a single functionality of DALE student profiles (for student university will be able to develop, implement, and modeling) and access to course materials. Student agents support a complete open learning environment. The other reside primarily on the student client after being options are commercial proprietary closed systems, dispatched from an agent archive on the server. proprietary open source systems, or open source systems Functions supported by agents include collaborating with developed by formal and informal collaborations of other students, generating self-tests of random items, educational institutions and government. Hence, CCIS is tailoring course material to student profile, managing involved in or is tracking a number of open source and course material presentation, managing external links related collaborative endeavors: JA-SIG (http://www.ja(reporting new links to of interest to the students). There sig.org/) is an independent organization whose mission is is no reason to restrict connections to a single institution. to enhance the flow of information among educational Learner’s agents could reside on the learner platform institutions and companies involved in the development interacting with a variety of institutions and peers from of administrative applications using Java technology. those institutions. Some agents may be mobile and visit The JA-SIG uPortal is a free, sharable portal for posta variety of hosting institutions looking for particular secondary institutions. It is an open-standard effort courses, information, or peer support. using Java, XML, JSP and J2EE and a collaborative development project. The Open Knowledge Initiative 4. Implementation Strategies (http://web.mit.edu/oki/) led by the Massachusetts Institute of Technology and Stanford University is CCIS uses open source and free software following open developing open source software for e-learning. The systems standards so we are not locked into a particular software is an alternative for institutions that want to vendor's solution. This approach providea e-learning provide online courses but do not want to invest in environments that are not under the control of commercial course management systems. The mozilla commercial interests. Also it provides CCIS with an project (http://www.mozilla.org/) is developing an embeddable xml-enable browser that can be embedded with a Java application (http://www.mozilla.org/projects/blackwood/webclient/) as part the learner’s platform in a distributed learning environment. SUN supported Project JXTA (http://www.jxta.org/) is an open source effort aimed at providing “an open, generalized protocol that interoperates with any peer on the network including PCs, servers and other connected devices” to facilitate the development of distributed applications. GNOME (http://www.gnome.org/) is an open source desktop environment for Linux.. With the emergence of XML with DTD's and schemas, X-Path, X-link, XML objects (DOM) and XSL, tools are emerging for building inheritance hierarchies for all curriculum objects that can be easily manipulated by Java. JMF allows us to easily integrate and manipulate multimedia. For foreseeable future we will continue to build on XML, Java, and Java related tools For example for agent communications we will use the Java Reflection Broker (JRB: http://andromeda.cselt.it/users/g/grasso/free.htm) which deploys a dynamic invocation model similar to the dynamic invocation interface in the Common Object Request Broker Architecture ( CORBA). We use a KQML-like message protocol and the Java Bean event model to provide the communication mechanism between agents and the facilitator and define our own event objects to handle the message content. We are considering using SOAP (http://www.w3.org/TR/SOAP/) for XML document exchange. We add security using the Java Security API, including user authentication and data encryption. Our peer-to-peer technology will be based on JXTA (http://www.jxta.org). 5. Evaluation "Situated research" is a very important concept for us. The WEB provides a means for other researchers to immediately experience students’ educational experience as the student does. Further, the students become a direct part of the research and development. We want to understand the experience of learners and the context in which learning occurs. Thus along with traditional quantitative methods for informative evaluation we use "illuminative evaluation" (Partlett and Hamilton, 1972). References Holt, P., Fontaine, C., Gismondi, J. and Ramsden, D. (1995) Collaborative Learning Using Guided Discovery on the INTERNET. ICCE95. Singapore. 2 Race, Phil. (1994) The Open Learning Handbook. NJ: Nichols Publishing Company Review of Design, Teaching, and Institutional Issues. PA: ACSDE. 3 Pea, R. (1993) Seeing what we build together: Distributed multimedia learning environments for transformative communications. The Journal of the Learning Sciences, 3 , 3, 285-299. 4 Holt, P., Fontaine, C., Gismondi, J. and Ramsden, D. (1995) Collaborative Learning Using Guided Discovery on the INTERNET. ICCE95. Singapore. 5 Norman, Donald A. & Spohrer (1996). LearnerCentered Education. Communications of the ACM, 39, 4, 24-27. Oram, A. (Ed.) ( 2001) Peer-to-Peer Harnessing the Power of Disruptive Technologies, O’Reilly Sebastapol CA 6 Shih, T. K., Chang, S. K., Wang, C. S., Ma, J., Huang, R. An Adaptive Tutoring Machine Based on Web Learning Assessment. In: Proceedings of the IEEE Inter. Conf. on Multimedia and Expo 2000, New York City, USA, July 31 – August 2, 2000. 7 Holt, O. Lin, F., Stauffer, K., Jelica, G., Shih, T. (2001). An Infrastructure for Developing Agents for Distance Education on the Internet, Journal of Computers, special issue on Distance Learning, May 2001 (in press) 8 Collins, A., Brown, J.S. and Newman, S. (1990). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics, in L. B. Resnick (Ed.), Knowing, Learning, and Instruction: Essays in the Honour of Robert Glaser. Hillsdale, N.J.: Erlbaum. 9 Holt, P., Fontaine, C., Gismondi, J. and Ramsden, D. (1995) Collaborative Learning Using Guided Discovery on the INTERNET. ICCE95. Singapore. 10 Vassileva, J.I., Greer, J.E., McCalla, G.I., Cooke, J.E., Collins, J., Kumar, V.S., and Bishop, A. (1998) The Intelligent Helpdesk: Supporting Peer-Help in a University Course, International Intelligent Tutoring Systems Conference (ITS'98), San Antonio, August 1998 (pp. 494-505). 11 Brown, J. & Collins, A. & Duguid, P. (1989) Situated Cognition and the Culture of Learning. Educational Researcher, 18, 4, 10-12. 12 Harasim, L. (1993) Collaborating in cyberspace: Using computer conferences as a group learning environment. Interactive Learning Environments, 3, 2, 119-130. 13 Vassileva, J.I., Greer, J.E., McCalla, G.I., Cooke, J.E., Collins, J., Kumar, V.S., and Bishop, A. (1998) The Intelligent Helpdesk: Supporting Peer-Help in a University Course, International Intelligent Tutoring Systems Conference (ITS'98), San Antonio, August 1998 (pp. 494-505). 14 Holt, P. (1998) Waiting for Godot. Principles Underlying an Interim Distance Education System for World Wide Web Delivery. ICCE 98, Beijing, China. 15 SCORM “Advanced Distributed Learning Initiative. Sharable Courseware Object Reference Model (SCORM)” Version 1.0. January 31, 2000. 16 Downes, S. (2000) Exploring New Directions in Online Learning, 17 Lin, F., Holt, P. (2001a) Towards Agent-based Online Learning, CATE'2001, Banff, Canada. 18 Lin, F., Holt, P. (2001a) Towards Agent-based Online Learning, CATE'2001, Banff, Canada. 19 Vassileva, J.I., Greer, J.E., McCalla, G.I., Cooke, J.E., Collins, J., Kumar, V.S., and Bishop, A. (1998) The Intelligent Helpdesk: Supporting Peer-Help in a University Course, International Intelligent Tutoring Systems Conference (ITS'98), San Antonio, August 1998 (pp. 494-505). 20 Hiltz, S. R. and Wellman, B. (1997). Asynchronous learning networks as a virtual classroom. Communications of the ACM, Vol. 40, N. 9, pp. 44-49. 21 Huhns, M., A. Mohamed. (1999). "Benevolent Agents", IEEE Internet Computing, Vol.3, No.2, pp.9698 22 Johnson., W. L. and Shaw, E. (1997). Using Agents to Overcome Deficiencies in Web-Based Courseware, AI-ED'97. 23 Selker, T., Coach: A Teaching Agent that Learns, Communications of ACM, 37(7), 1994. 24 Solomos, K. and Avouris, N. (1999). Learning from Multiple Collaborating Intelligent Tutors: An Agentbased Approach, J. of Interactive Learning Research, 10(3/4), 243-262 25 McCollum, K. (1998). How a Computer Program Learns to Grade Essays, The Chronicle of Higher Education, September 4, 1998. 26 Whatley, J.G, Staniford, Beer, G. M. and Scown, P. (1999). Intelligent Agents to Support Students Working in Group Online, Journal . of Interactive Learning Research, 10(3/4), 361-373 27 Thaiupathump, C., J. Bourne, J. and Campbell, J. (1999) Intelligent Agents for Online Learning, JALN Volume 3, Issue 2 - November 1999. http://www.aln.org/alnweb/journal/Vol3_issue2/Choon2. htm 28 Gelernter, D. (2000) The second coming a manifesto. Edge, 70, June 15. URL http://www.edge.org/documents/archive/edge70.ht ml 29 Oram, A. (Ed.) ( 2001) Peer-to-Peer Harnessing the Power of Disruptive Technologies, O’Reilly Sebastapol CA 30 Partlett and Hamilton (1972). Evaluation as Illumination: a new approach to the study of Innovatory Programs, Centre for Research in the Educational Sciences, University of Edinburgh, Occasional Paper - 9, Edinburgh 1972 31 DALE Research Team. (2002) The Design of an Integrated System for Web-based Distance Education, CCIS Report in preparation. 32 Lin, F., Holt, P., Korba, L. and Shih, T. (2001). A Framework for Developing Online Learning Systems, DMS'2001, Taipei, Tiawan. 33 Lin, F., Holt, P. (2001a) Towards Agent-based Online Learning, CATE'2001, Banff, Canada.