Software Tools for Distance Learning: The Benchmarks
Project
Abbie Brown
Indiana University Bloomington
Lisa Hansen
Indiana University Bloomington
Distance education has emerged as the latest wave in educational practice.
It is currently the fastest growing form of domestic and international
education. Course delivery via the World Wide Web is receiving a great deal
of attention from educators and administrators, but questions on how to
implement and administer Web-based coursework have been left largely
unanswered.
The major problem facing Web-based instructional development today is the
development process itself. This process is time-consuming and requires a
certain level of computer competency to be performed adequately. New tools
being developed to create Web-based learning materials are helping this
process along, but a new issue has arisen. Which software should a teacher
use to create on-line materials? Which tools work best? What elements are
required in a Web-based course, and what software tools perform that
function?
Beginnings of the Project
The Center for Excellence in Education at Indiana University Bloomington
facilitated the formation of a small group interested in addressing the
problem of categorizing and evaluating software. The group members, who
dubbed the project "Benchmarks," were self-selecting, with expertise in
software evaluation, and considerable experience in teaching and in
facilitating faculty use of technology. The group began by conducting a pilot
evaluation of on-line "chat" tools, and by developing a taxonomy of
distance-education software, a set of evaluation protocols, and a sample
Web site putting these protocols into practice.
Successive prototypes of the evaluation protocol, and the apparent
soundness of the taxonomy, led to an increased interest in the project's
completion and incorporation into a larger, university-funded Web site
developed for educators interested in emergent technologies. The project
received funding after ten months of volunteer effort, and has since
developed and tested a formal evaluation protocol.
In a combined effort to meet the needs of faculty members asked to develop
on-line instruction, Indiana University’s School of Continuing Studies and the
Center for Excellence in Education encouraged and ultimately funded a
project to document and evaluate the software necessary to Web-based
course delivery.
The Need for Web Tool Categorization and Evaluation
The need for organization and classification of software by type, as well as
evaluation of the various software packages within each type, was clearly
identified by those educators under pressure to produce distance education
courses. There has been little previous research in this area, nor have
evaluation protocols been established for distance education development
software.
There is a definite need for the organization and articulation of Web-based
development tools. First, simply understanding the variety of tools available
provides insight on what is needed or possible for a Web-based course.
Second, by providing this organization and articulation, instructional
designers can reduce the amount of time needed to develop on-line
instructional materials, by sharply reducing the sampling and discarding of
various tools by uninformed instructors. Teachers already have huge
demands upon their time; to require instructional development, modestly
estimated at 100 hours of development for one hour of instruction, may
prevent many schools and teachers from delivering courses on-line.
Having identified the need for software evaluation, expert review presents
itself as the most efficient and effective method. Expert users have little fear
of difficult or obscure interface idiosyncrasies and, as designers themselves,
are quick to recognize common mistakes made in software design. Expert
reviewers are capable of standing in the programmer's shoes, as well as the
user's, and are comfortable identifying problems that arise from faulty
software as opposed to faulty software use.
Developing the Taxonomy
Figure 1. Taxonomy of Web tools.
Originally the Benchmarks team focused its attention on review and
assessment of current on-line collaborative activities, particularly those that
involved k-12 classroom groups or individual learners in k-12 settings (e.g.
projects offered by TERC, and the Monarch Butterfly Watch project). The
focus changed when the School of Continuing Studies requested review and
evaluation of tools relevant to their instructors developing on-line courses;
additionally, it was decided that the focus would be on tools available at
Indiana University.
We began to discuss these collaborations in terms of their shared traits - we
noticed that each collaboration required a series of component applications,
and that we could categorize these application components in terms of three
major concepts: their purpose in facilitating the collaboration (purpose) their
relationship to other applications (package) and the time-frame in which
they operated (process).
Each application (for example: word processing, on-line chat, e-mail) could
be thought of as part of the purpose, package, and process of the
collaboration.
Purpose
What a software product does is its purpose. A Web tool will fall into one of
three categories: creation, communication, or administration. A fourth
category used in the Web site is multipurpose tools, those tools that
combine in some way the previous categories.
Creation (development) --creating physical Web elements
Creation tools are those that are used to create the "physical" elements of a
Web site: Web pages, graphics, sound, video, image maps, forms, etc.
Currently we provide information about the two basic development tool
types, HTML editors and graphics tools.
HTML editors are often what people think of when they think of using a tool
to develop Web pages. From the Web site, one can better understand that
HTML editors are merely part of a much larger whole. However, the uses of
an HTML editor are obvious, and using a good HTML editor makes the
process of Web development much easier.
A graphics tool can perform many or few functions. However, larger is not
necessarily better when it comes to a graphics program. If you only need to
save a graphic as a Web-compatible file (a GIF), then you really don't need a
program that is extremely complicated. On the other hand, if you are
creating or scanning images, a larger graphics program with more
functionality will be more useful.
Communication --interaction and information-sharing among class
participants
This category of tools includes chat, conferencing, e-mail, file transfer, and
newsreaders. A key element to any course is communication between
participants, including teacher-to-student (and vice versa) and student-tostudent. These tools provide a variety of communication options, including
live versus time-independent, group communications versus one-on-one,
and file and news sharing. While many tools can perform more than one
function, they often fall primarily into one of the following types.
Conferencing is a vital component to an on-line course because it provides
the means for dialogue and discussion as a group. Indiana University is
currently reviewing a variety of conferencing tools for potential adoption. File
transfer programs are used to move files from a local computer to a shared
remote computer, such as a Web server. The step of file transfer is usually
the last step in creating a Web site. Newsreaders are not as commonly used
for on-line courses; however, they do provide a low-bandwidth alternative to
chat and conferencing applications. E-mail applications enable a user to
read, compose, send and manage e-mail messages. Users send and read
messages at a time of their choosing, without needing to coordinate time
with other parties. It provides individual communications (useful for
answering and asking questions privately), but can also be used for general
announcements and informal discussion, in the form of listservs and
distribution lists, and as a convenient interface for file transfer.
Administration. Administration tools for faculty developing on-line courses
are often provided by the school -- in this case, IU Bloomington. However,
there are several commercial products as well. Currently, we have reviewed
testing and grade reporting applications.
Multi-purpose. Multi-purpose tools are those applications which are
specifically designed to do a variety of things, in order for a developer to
work more efficiently. These tools fall into two groupings. Web browsers are
just that -- Web interface programs. All-in-one course creation tools are
those applications which were created for the purpose of developing on-line
courses. These tools have much of the functionality of smaller, more specific
applications; their advantage is that they are designed to work together.
However, all-in-one tools vary widely in their functionality.
Package
Package is that feature of the program which indicates how it is used. An
independent program is a stand-alone application. An interdependent can
stand alone, but is designed to work with other applications; an example
would be Microsoft's Word, Excel, and Powerpoint. Dependent tools require
the use of a larger application; for example, using America On Line's chat
tool is only possible when using America On Line itself
Process
This element is more applicable to those on-line application tools that have
time as an important component. This is particularly important for
communications tools and collaborative development tools. Asynchronous
tools are time-independent; participation occurs at users' convenience.
Synchronous tools are time-dependent, and require simultaneous
participation by users.
Evaluation Protocol and Design
The resulting evaluations covered more than 30 creation, communication,
and administration tools. We focused primarily on applications available to
faculty at Indiana University, since they were our target audience. We
developed benchmark tests that were used to evaluate these tools. These
tasks were written such that they generated an ordinal number grade; each
task was reviewed with the same scale. The scale is defined as such:
1. Very poor capability to perform listed task; feature not available or
does not perform as intended.
2. Performs task with only limited success.
3. Performs task adequately, but improvement is
4. Performs task well and with few problems.
5. Performs task extremely well; the task feature was as well designed as
could be hoped.
The evaluation results were presented in three types of pages. Features
reviews examined basic features such as platform, availability, and interface.
These reviews included tools that were not fully reviewed but represented
popular software available. Evaluations pages were comparative results of
benchmarks tasks. Figure 2 below is an example of an evaluation page
report. Profile pages were individual reviews of software, with comments
from reviewers included.
The site was also designed to be easily expanded and new reviews included.
Because of the size of the site and the potentially large number of software
reviews that could be added, it was deemed important to design the site in
such a way that future reviewers could add material easily.
This site is available at
http://www.wisdomtools.com/benchmarks/bhome.html
Figure 2. Sample evaluation table from Benchmarks site. HTML editors.
Note: We would like to express our appreciation and acknowledge the contributions of all
Benchmarks team members -- Susan Sloffer, Tim Green, Nadine Kwok, Chandra Hawley,
and Bill Dueber; and our directors at Wisdom Tools -- Marty Siegel and Sonny Kirkley.