Adaptive Educational Technologies
TOOLS FOR LEARNING
EARNING and for
LEARNING ABOUT LEARNING
The National Academy of Education advances high quality education research
and its use in policy formation and practice. Founded in 1965, the Academy consists
of U.S. members and foreign associates who are elected on the basis of outstanding
scholarship related to education. Since its establishment, the Academy has undertaken
research studies that address pressing issues in education, which are typically conducted
by members and other scholars with relevant expertise. In addition, the Academy sponsors professional development fellowship programs that contribute to the preparation of
the next generation of scholars.
Gary Natriello, Editor
National Academy of Education
Washington, DC
NATIONAL ACADEMY OF EDUCATION 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the National
Academy of Education Board of Directors.
The National Academy of Education acknowledges grant support for this study
that was provided by the Pearson Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author
and do not necessarily reflect the views of the funder that provided support for
the project.
This report has been reviewed in draft form by individuals chosen for their diverse
perspectives and technical expertise in accordance with the review procedures of
the National Academy of Education. The following individuals are thanked for
their careful review of this report: Allan Collins, Northwestern University; Roy
Pea, Stanford University; and Lorrie Shepard, University of Colorado at Boulder.
Additional copies of this report are available from the National Academy of
Education, 500 Fifth Street, N.W., Washington, DC 20001; (202) 334-2340; Internet,
http://www.naeducation.org.
Copyright 2013 by the National Academy of Education. All rights reserved.
Suggested citation: National Academy of Education. (2013). Adaptive Educational
Technologies: Tools for Learning, and for Learning About Learning, G. Natriello (Ed.).
Washington, DC: Author.
The National Academy of Education advances high quality education
research and its use in policy formation and practice. Founded in 1965, the
Academy consists of U.S. members and foreign associates who are elected
on the basis of outstanding scholarship related to education. Since its
establishment, the Academy has undertaken research studies that address
pressing issues in education, which are typically conducted by members
and other scholars with relevant expertise. In addition, the Academy
sponsors professional development fellowship programs that contribute
to the preparation of the next generation of scholars.
ADAPTIVE EDUCATIONAL TECHNOLOGIES PROJECT
Workshop Cochairs
Susan Fuhrman, Teachers College, Columbia University
James Gee, Arizona State University
Brian Rowan, University of Michigan
Staff
Judie Ahn, National Academy of Education
Katie Conway, Teachers College, Columbia University
Gregory White, National Academy of Education
v
Contents
Executive Summary
1
1
Introduction
5
2
What Are Adaptive Educational Technologies?
7
3
The Potential of Adaptive Educational Technologies in
Education Research
11
4
Examples of Research Drawing on Adaptive Educational
Technologies
15
5
Infrastructure Needed to Support Research Drawing on
Adaptive Educational Technologies
19
6
Next Steps
25
7
References
27
Appendixes
A Workshop Agenda and Participants
31
B
37
Summit Agenda and Participants
vii
Executive Summary
W
ith the spread of adaptive technologies that customize the
user experience in response to individual users, it is not
surprising that such experiences are increasingly found in
educational settings or in tools to facilitate learning. The National
Academy of Education commissioned a background paper and held
two meetings of scholars, policy makers, and developers of adaptive
educational technologies to consider the implications of such adaptive systems for education research and education researchers. This
report highlights the issues discussed in those proceedings.
Recent progress in the development of adaptive educational
technologies builds on several decades of efforts to use computer
systems to offer tailored instructional experiences to students. Adaptive elements can be found in many forms and formats that support
learning. Adaptive hypermedia learning systems, intelligent tutoring systems, adaptive elements embedded into online courses, and
a variety of educational games and simulations can all be designed
to tailor the learning experience to the needs of individual students.
A wide array of information on learners can be used to shape the
individual learning experience, including prior achievements, preferences, interests, traits, and the immediate learning environment.
Whatever the individual learner characteristics or the dimensions of
the learning experience represented in the system, the key defining
feature of adaptive educational technologies is that one or more elements of the system are modified in response to information about
1
2
ADAPTIVE EDUCATIONAL TECHNOLOGIES
the learner. It is this adaptivity that creates the personalized learning
experience intended to maximize the learning of each student.
The growth of adaptive educational technologies offers some
unique advantages as well as some new challenges for education
research. Among the advantages are the ability to gather expanded
learner profile metadata and aggregate it to allow us to learn more
about the patterns of learning across venues, expanded data sources
to make inferences about learning, and new techniques such as data
mining and machine learning for making sense of information on
learning. In addition, adaptive educational technologies can support learner agency by extending access to learner data systems to
learners themselves, they can allow us to inquire into the impact of
learner feedback in social systems by granting learners access to data
on their own performance in relation to the performance of others,
and they can provide large-scale test beds for experimentation.
In addition to new advantages, adaptive educational technologies also present some new challenges. Adaptive systems typically
do not include contextual data beyond the system, they gather data
in ways that are not easily organized for research and analysis,
and they often produce data for which it is challenging to attribute
meaning in the absence of a prior learning theory. In addition, there
are unresolved concerns about the ownership of data on students as
well as concerns about student privacy.
Although the era of adaptive educational technologies is just
dawning, there are already several major areas of research. A number of studies have attempted to provide effective feedback to students as they utilize adaptive systems. Such studies have explored
the possibility of providing advice to students in real time using
ideas drawn from areas such as machine learning and social network
analysis. A second group of studies has attempted to determine how
to provide feedback to instructors as they utilize adaptive systems,
including information on how to improve course structure, teaching
styles, and student motivation. A third group of studies uses data
from adaptive systems to provide insights into patterns of student
learning in response to particular configurations of content. These
studies of how students learn specific subject matter offer guidance
for the development of efficient learning trajectories. A fourth group
of studies draws on data from adaptive educational technologies to
inform improvements to those same systems. A fifth and final set of
studies uses data from adaptive systems to advance general theory.
Because adaptive educational technologies represent a substantially new research opportunity, making full use of them will require
new types of infrastructure. Adaptive educational technologies pres-
EXECUTIVE SUMMARY
3
ent new kinds of data organized in fresh ways. These data exist in
new configurations that vary from system to system. Moreover,
the conditions under which data are gathered in these systems differ from those typically associated with education research activities. Dealing with the situations presented in adaptive educational
technologies will require consideration of at least new training
for researchers, development of new tools of analysis, specification of prototypes and standard protocols, and attention to the rights
of individuals whose data are collected as part of the operation of
these systems.
Education researchers will require new skills to make use of the
data generated by adaptive technologies. Elements of an effective
training infrastructure might include seminars and workshops on
handling large and complex datasets; publications such as handbooks, textbooks, and journals to build the knowledge base on techniques for dealing with data from adaptive systems; specialized
professional associations and related conferences; and courses, specializations, and degree programs in institutions that prepare education researchers. Developing prototypes and protocols to standardize the data produced by adaptive educational technologies could
also support the greater use of such data by researchers. In addition,
investments in the development of analytical tools to handle data
from adaptive educational technologies could reduce the burden on
education researchers and encourage greater use of data from such
systems. Finally, developing models for the governance of the data
generated by adaptive systems will be necessary to promote access
for education researchers.
In view of the rapid growth of the use of adaptive educational
technologies, workshop participants identified some possible next
steps to protect the interests of the education research community.
These include developing standards for data gathered through
adaptive educational technologies to support education research,
developing standards for credentials for education researchers to
demonstrate proficiency in the handling of data from adaptive educational technologies, and guidelines for human subjects committees to facilitate the review of research projects involving data from
adaptive systems.
The growth of adaptive educational technologies presents new
opportunities for education research that can advance our understanding of student learning and performance. The full participation
of the education research community is necessary to create the conditions that will guarantee that the promise of adaptive educational
technologies is fully realized for research as well as practice.
1
Introduction
W
e have reached a point where most people in modern
societies have at least some experience with adaptive
technologies, that is, systems that present themselves in
ways customized to individual characteristics. Such systems are all
around us: the lock that opens in response to your ID card, the ATM
that retrieves your account information, the car seat that returns to
the position you left it in when you insert your key (despite what
other drivers may have done in the meantime), the thumbprint
reader that allows only you to access your computer.
In the online world, the examples are even more powerful: personalized pages on Amazon that show items that may be of interest
to you given your browsing and purchasing history, Google search
results personalized in response to your search history, the personal
account information displayed by your bank after you log in.
The behavior of these and similar systems is modified or adapted
in response to individual users. The systems may respond to directions or information provided by the user, to an action or choice, or
to information in the system or in connected systems.
With the spread of adaptive technologies into all aspects of life,
it is not surprising that they are increasingly found in educational
settings or as tools to support learners and to facilitate learning. The
growth of such technologies has implications for educators, learners,
and all those interested in using them in tools for learning.
The growing use of adaptive educational technologies as important elements in the education sector also creates new opportunities
5
6
ADAPTIVE EDUCATIONAL TECHNOLOGIES
and challenges for education researchers. These tools generate new
types of robust datasets that can offer new possibilities for education
researchers. At the same time, these new opportunities suggest the
need to enhance the skills of education researchers so that they can
manage data from adaptive systems and utilize the data in a range
of basic and applied studies.
Over the course of 2011, the National Academy of Education
commissioned a background paper and convened two meetings
to discuss these and related issues. An initial planning meeting
was held in April to identify major issues and topics for a more
extensive gathering in December. The December meeting included
panels on learning, instruction, assessment, concerns, institutional
responses and innovations, and developing infrastructure as well
as demonstrations of a number of adaptive educational systems.
Appendixes A and B contain the lists of attendees and panelists
from the meetings. This report highlights the issues discussed in
those proceedings.
2
What Are
Adaptive Educational Technologies?
T
he goal of responding to the needs of individual learners has
received attention recently because of new demands on the
educational system and new possibilities for providing personalized learning support. New demands for personalized learning
stem from the growing sense that advanced economies require the
vast majority of citizens to achieve high levels of learning throughout their lives. Such a massive increase in the demand for education
may only be met with new tools, techniques, and learning resources.
Progress over the past decades in computing and communications technologies has set the foundation for a new learning infrastructure (Computer Research Association, 2005; National Science
Foundation Task Force on Cyberlearning, 2008). Many computerbased systems engage students with educational opportunities,
including instruction and resources. There has also been a shift from
stand-alone hypermedia and tutoring systems to widely available
Web-based systems. It is upon these technologies that new learning
tools are being built to support individually responsive learning
environments (Gardner, 2009; Maeroff, 2003) that promise to help
greater proportions of the population to achieve higher levels of
learning.
Adaptive educational technologies take account of current
learner performance and adapt accordingly to support and maximize learning. By design, they present personalized educational
experiences for each learner. Such technologies grow out of a long
line of work on using computer systems to offer tailored instruc7
8
ADAPTIVE EDUCATIONAL TECHNOLOGIES
tional experiences to students (U.S. Congress, Office of Technology
Assessment, 1988), beginning with Skinner’s teaching machines in
the 1950s (Skinner, 1986), and continuing on through the PLATO
project at the University of Illinois in the 1960s (Smith & Sherwood, 1976), and the work of Suppes and Atkinson at Stanford on
computer-assisted instruction in the 1970s and beyond (Suppes &
Fortune, 1985).
Adaptive elements can be found in many forms and formats that
support learning. They might involve ways of organizing resources
or might involve complex learning environments. Adaptive hypermedia learning systems organize and present resources in ways that
are tailored to individual student learning needs (Brusilovsky, 2001).
Intelligent tutoring systems attempt to achieve the kinds of positive impact on learning long associated with one-on-one tutoring
(VanLehn, 2011). The instructional elements of these systems, which
are based on domain knowledge, knowledge of typical student
learning patterns, knowledge of teaching strategies, and knowledge
of methods for communicating with students (Woolf, 2009), adapt in
response to individual students in order to maximize learning. Tutoring and other adaptive strategies can be embedded in online courses
(Lovett, Meyer, & Thille, 2008). Additionally, a small but growing
number of educational games utilize adaptive techniques to enhance
the learning of individual players (Pierce, Conlan, & Wade, 2008;
Barab, Gresalfi, & Ingram-Goble, 2010; National Research Council,
2011; Reese, 2012; Shute & Ventura, 2013).
A wide array of information about learners has been used to
drive adaptation, including the learner’s current state of knowledge
and history of learning as inferred from his or her digital educational
system interactions. Of course, both of these are considered within
the context of the learning goals established within a particular
system. More general characteristics of the learner—preferences,
interests, and traits—are also used in some adaptive systems. The
learner’s experience with online environments and the immediate
environment in which the learner is working may also be considered. Information on all of these factors can be used to generate the
optimal personal learning experience through the adaptive system
(Brusilovsky, 1996, 2001).
Brusilovsky (1996) specifies two broad techniques for adapting
content to learners: adaptive presentation and adaptive navigation.
Adaptive presentation involves tailoring the presentation of media
content, presenting different text to different learners, and adaptation of the mode of presentation. Adaptive navigation involves techniques that help learners navigate content by adapting the way links
WHAT ARE ADAPTIVE EDUCATIONAL TECHNOLOGIES?
9
are presented. Adaptive navigation techniques include providing
direct guidance to learners, sorting links, hiding links, annotating
links, generating links, and mapping links, all based on individual
learner characteristics (Brusilovsky, 2001).
In terms of the more complex learning arrangements of virtual
worlds and games, the possibilities for adaptation become greater
and less generic. In the panel on instruction, Sasha Barab talked
about “adapting whole story lines, whole worlds, whole roles, not
just conceptual ideas. . . .” Indeed, such complex learning environments allow elements such as role specifications and entire story
lines to respond to individual learners and their unique characteristics (Barab, Gresalfi, & Ingram-Goble, 2010).
Whatever the individual learner characteristics or the dimensions of the learning experience represented in the system, the key
defining feature of adaptive educational technologies is that one or
more elements of the system are modified in response to information about the learner. It is this adaptivity that creates the personalized learning experience intended to maximize the learning of each
student.
3
The Potential of Adaptive
Educational Technologies in
Education Research
B
ecause adaptive educational technologies collect data on individual students and student performance, they generate datasets that offer both new opportunities and new challenges for
education researchers.
Unique Advantages of Using Data from
Adaptive Learning Technologies
In the panel on learning, Roy Pea identified six ways in which
adaptive learning technologies can help us learn about learning:
1. Adaptive learning technologies provide expanded learner
profile metadata and aggregate them to allow us to capture
the benefits at scale of learning more about the patterns of
learning across diverse schools, districts, and states.
2. They expand the data sources we use to make inferences
about learning and its conditions.
3. Through the use of techniques such as data mining and
machine learning, we can expand our sense-making techniques for understanding learning and related conditions
as a basis for guiding more effective learning.
4. Adaptive learning technologies can extend access to learner
data systems to the learners themselves to enhance agency,
self-assessment, and self-regulation.
11
12
ADAPTIVE EDUCATIONAL TECHNOLOGIES
5. They can also extend learner access to data about their own
performance in relation to the performance of others to
support inquiry into the functioning of learner feedback in
social systems.
6. Adaptive educational technologies can also provide largescale test beds for experimentation.
These new opportunities afforded by adaptive educational technologies suggest three courses for education research. First, they
offer new possibilities for education researchers to examine problems and issues previously examined in other data. Second, they
allow new kinds of research questions that have previously eluded
empirical examination. Third, they have the potential to generate
research questions as a result of examinations of large new datasets.
However, as is discussed below, the data generated by adaptive
systems come with special challenges as well.
Unique Challenges of Using Data from
Adaptive Learning Technologies
The data generated by adaptive educational technologies present challenges for analysts trying to extract meaning and address
research questions, and also present some special issues worth
noting:
1.
Although data gathered through adaptive systems can offer
insight into important relationships among the variables
included, they typically do not include contextual data beyond the system itself. Thus, for example, data on events
preceding student engagement with the system, or data
on contemporaneous events outside the system such as
conversations between students and teachers or learning
experiences in the home or community, would not be available without additional efforts to collect data. In addition
to such traditional out-of-system events, during the meeting of the panel on learning, Jim Gee highlighted the independent growth of affinity spaces where users of systems
gather to share knowledge; activity in such spaces would
not be captured in the data generated by the adaptive system. If the adaptive system generates data over a considerable period of time, the lack of data on other aspects of the
students’ educational and life experience may compromise
a researcher’s capacity to develop a clear understanding
POTENTIAL OF ADAPTIVE EDUCATIONAL TECHNOLOGIES
2.
3.
4.
5.
13
of the impact of experience on learning. Thus, researchers
relying solely on data from the adaptive system might not
appreciate the contextual factors.
Adaptive technologies gather data in ways that are optimized for the efficient operation of the system, including adjustment of what the system presents to students. Such data
are not organized in ways that are amenable for analysis. At
the meeting, Brian Rowan described the challenge of taking
data from adaptive systems and processing them to make
them suitable for analysis. Preparing data from adaptive
systems for analysis is a very substantial task. As a result,
it may be more accurate to view the data preparation stage
as another step of data collection as the researcher selects
and reorganizes the data to address the research questions
(Romero & Ventura, 2007).
While adaptive educational technologies tend to produce
data tightly linked to specific student actions, the meaning of such data is often unclear. For example, the kind of
keystroke data typically gathered may raise questions about
the proper unit of data for analysis. Less detailed and more
meaningful actions are both easier to handle (Stephens &
Sukumar, 2006) and potentially more useful for research
purposes (Mislevy et al., 2010). Other student movements
(e.g., drawing on a tablet or making gestures) are challenging to interpret, and attributing meaning requires additional
data, assumptions, or, ideally, a framework of meaning.
Although there are ongoing efforts to address issues of data
ownership and access (Office of Science and Technology
Policy, 2012), the ownership of and access to data from
adaptive educational systems is complicated. Education
researchers are accustomed to negotiating access with
students, parents, and schools, but adaptive educational
systems introduce another player into the mix: the system
developer or provider. System providers may assert rights
to the use of data for system development, and they may
be reluctant to share proprietary data rights with education
researchers. Of course, commercial providers are not unique
in that regard as John Stemper suggested in the panel on
concerns when he explained the challenge of encouraging
researchers to share data in open repositories.
Because adaptive educational technologies gather information on individual students through online applications, they
inherently raise three types of privacy concerns. First is the
14
ADAPTIVE EDUCATIONAL TECHNOLOGIES
6.
set of concerns related to the status of students as individual
citizens or consumers in a networked world where valuable
access to networked resources requires the exchange of personally identifiable information (Nissenbaum, 2010). Second
is the set of concerns related to the status of many students
as children whose privacy may require additional protections by virtue of their youth (Pitman & McLaughlin, 2000).
Third is the set of concerns related to the role of students
within educational institutions, a role that entails the gathering of particular kinds of information in the educational
process (Glenn, 2008). Developers, providers, and adopters
of adaptive educational technologies must confront these
multiple layers of concerns for the privacy of student users
of such technologies. Education researchers intent on using
the data generated by adaptive technologies must be accountable for understanding these various privacy concerns
and the procedures for addressing them in the applications
that generate data used in their research.
Because data gathering in adaptive systems is integrated
with program delivery in a way seldom encountered in education research activities that are typically grafted on (often
over considerable resistance) to the regular business of educational programs, the opportunities for research have the
potential to expand exponentially.
The unique opportunities afforded education researchers by
data from adaptive educational technologies have been and will
be sufficient to generate the interest and effort necessary to address
the unique challenges presented by their use in research. Indeed,
some of the challenges noted can be reduced or eliminated over
time. For example, as learning scientists become more involved in
the design and development of adaptive learning systems, these
systems are likely to be designed with research and data interpretation in mind. As adaptive systems become more widely used, issues
of data ownership, handling, and privacy are likely to be resolved.
Even at this early stage, adaptive educational technologies are
supporting fruitful lines of inquiry. In the next section are some
examples of research drawing on data from adaptive systems.
4
Examples of Research Drawing on
Adaptive Educational Technologies
S
everal attempts to characterize and classify research that makes
use of data from adaptive educational technologies (Baker &
Yacef, 2009; Castro, Vellido, Nebot, & Mugica, 2007; Romero
& Ventura, 2007) highlight the breadth of possibilities. To illustrate the types and range of studies, five categories are highlighted
below with the caveat that some studies may fall into more than
one category.
Major Areas of Research on
Adaptive Educational Technologies
•
•
•
•
•
Research that informs student users of adaptive systems
Research that informs teacher users of adaptive systems
Research that informs curriculum development
Research that informs the design and improvement of adaptive systems
Research that advances general theory and practice
Research That Informs Student Users of Adaptive Systems
A number of studies have attempted to determine how to provide effective feedback to students as they use adaptive systems.
These studies have explored the possibility of providing advice to
15
16
ADAPTIVE EDUCATIONAL TECHNOLOGIES
students in real time using ideas drawn from, among other areas,
machine learning and social network analysis. Examples of this type
include the following:
•
•
•
•
Hwang (1999) investigated a system to provide learning
advice to students.
Heraud, France, and Mille (2004) used student log data to
guide students in a tutoring system.
Kelly and Tangney (2005) employed machine learning techniques to generate information on student learning styles.
Romero, Ventura, Zafra, and De Bra (2009); and Tang and
McCalla (2005) provided personalized content for students,
the former via Web-usage mining, and the latter through
social network analysis.
Research That Informs Teacher Users of Adaptive Systems
Studies in a second group have attempted to determine how to
effectively provide feedback to instructors as they utilize adaptive
systems, including information on how to improve course structure, teaching styles, and student motivation. Examples of this type
include the following:
•
•
•
•
•
Feng and Heffernan (2007) provided live reporting to teachers
on student performance in the ASSISTment System.
Romero, Venturo, and De Bra (2004); Tang, Lau, Li, Yin, Li,
and Kilis (2000); and Vialardi, Bravo, and Ortigosa (2008)
provided general insights for course development and
improvement.
Roll, Aleven, McLaren, and Koedinger (2011) drew on students’ help-seeking to infer learning.
Hurley and Weibelzahl (2007) provided insight into student
motivation.
Crespo, Pardo, Perez, and Kloos (2005); and Zakrzewska
(2008) developed information to guide group formation.
Research That Informs Curriculum Development
Some studies using data from adaptive systems provide insights
into patterns of student learning in response to particular configurations of content. These studies of how students learn specific subject matter offer guidance for the development of efficient learning
EXAMPLES OF RESEARCH
17
trajectories. Examples of this area of research discussed at the meeting include the following:
•
•
•
Baker (2007), Jong, Chan, and Wu (2007), and Muehlenbrock
(2005) worked to detect student responses to the learning of
specific content areas.
Simko and Bielikova (2009) developed concept maps of specific subject-matter areas with the goal of allowing instructors to automatically create graphs showing the relationships
among concepts and the hierarchical nature of knowledge in
those domains.
Pavlik, Cen, and Koedinger (2009) analyzed learning curves
to generate domain models.
Research That Informs the Design and
Improvement of Adaptive Systems
Other studies have attempted to draw on data from adaptive educational technologies to inform how improvements to the
design of these systems might be made most effectively. For example, studies have attempted to understand how different types of
students respond to various adaptive educational technologies.
Others have focused on the delivery models for different forms of
content or across different subject matter, and still others on the
efficacy of various pedagogical strategies. Examples of this area of
research include the following:
•
•
Chi, VanLehn, Litman, and Jordan (2010) examined pedagogical approaches that lead to effective tutoring experiences.
Superby, Vandamme, and Meskens (2006) identified factors
that predict student failure in college.
Research That Advances General Theory and Practice
A fifth and final type of study has attempted to use data from
adaptive systems to advance general theory. Self-regulation is one
area where general theory has been used in research on adaptive systems (Lajoie & Azevedo, 2006). Research on the impact of
hypermedia environments on self-regulated learning provides a
good example of work that uses data from adaptive systems to
refine understanding of theory and ultimately improve practice
beyond the immediate adaptive system. Specific examples include
the following:
18
ADAPTIVE EDUCATIONAL TECHNOLOGIES
• Azevedo,Guthrie,andSeibert(2004)examinedstudentuseof
self-regulated learning processes and the impact on learning.
• McManus (2000) analyzed the relationship between levels
of learner control in hypermedia environments and student
self-regulatory skills for their impact on learning.
5
Infrastructure Needed to
Support Research Drawing on
Adaptive Educational Technologies
B
ecause adaptive educational technologies represent a substantially new research opportunity, making full use of them
will require new types of infrastructure. These technologies
present new kinds of data organized in fresh ways. These data
exist in new configurations and in ways that vary from system to
system. Moreover, the conditions under which data are gathered in
these systems differ from those typically associated with education
research activities. Dealing with the situations presented in adaptive educational technologies will require consideration of at least
four responses: training for researchers, development of new tools
of analysis, specification of prototypes and standard protocols, and
attention to the rights of individuals whose data are collected as part
of the operation of these systems.
Training for Researchers
Education researchers will require new skills to make use of
the data generated by adaptive technologies. Such skills are necessary for the tasks that are involved in taking the data from adaptive systems and making them manageable in analyses. Education
researchers will also require skills in analyses that are more common in data mining typically conducted by computer scientists and
systems engineers (Romero, Ventura, Pechenisky, & Baker, 2011).
Moreover, if researchers wish to participate in the development of
adaptive systems and thereby have a say in the data that are col19
20
ADAPTIVE EDUCATIONAL TECHNOLOGIES
lected, they will require knowledge of the designs and the design
possibilities of the systems. Otherwise, researchers will be limited to
data available from systems designed without their research questions in mind.
With a relatively well-defined set of skills to be conveyed to
education researchers, several elements of what might become an
effective training infrastructure are essential:
1.
2.
3.
4.
Seminars and workshops devoted to handling the large and
complex datasets generated by adaptive learning technologies could provide focused training and practice opportunities. A model for such activities can be found in the
institutes and workshops sponsored by the National Center
for Education Statistics to prepare researchers to work with
national datasets. Another option would be to organize
such activities through a network of regional institutionally
based programs. Examples of this approach can be found in
the Pittsburgh Science of Learning Center’s summer school
on mining of data from adaptive learning technologies and
the Learning Analytics Summer Institutes at Stanford.
A variety of publications might contribute to a knowledge
base on techniques for dealing with data from adaptive systems. These might include publications such as the Handbook
of Educational Data Mining (Romero, Ventura, Pechenisky,
& Baker, 2011) as well as textbooks and other professional
books highlighting the evolving set of analysis issues
and techniques. Specialized journals, such as the recently
launched Journal of Educational Data Mining and the Journal
of Learning Analytics, could offer outlets for publication and
modes of studies using the new kinds of datasets.
Specialized professional associations might be formed to
create opportunities for education researchers to learn
from one another via conferences and other activities.
Education researchers could also join existing communities of researchers in intelligent tutoring systems, artificial
intelligence in education, educational data mining, learning
analytics and knowledge, and the International Society of
Learning Sciences.
Perhaps the most substantial element of a new training
infrastructure would be the incorporation of courses, specializations, and degree programs in the graduate schools
where education researchers are prepared and in collaborative efforts to create joint programs with departments
INFRASTRUCTURE NEEDED TO SUPPORT RESEARCH
21
of computer science, statistics, psychology, and sociology
where big data science is being developed and where education research topics are coming to be addressed. This, of
course, would require the preparation of faculty to handle
such efforts.
Prototypes and Protocols
Providing training to education researchers to take on the currently nonstandard and unwieldy datasets emanating from adaptive learning technologies is only one approach to developing an
infrastructure to support education research on adaptive systems.
Another approach involves the development of prototypes and
protocols that could lead to greater standardization of the data produced by adaptive educational technologies. Education researchers
have a role in developing prototype systems, particularly around
data gathering and reporting. In addition, organizations such as the
U.S. Department of Education with its Learning Resource Metadata
Initiative and Learning Registry and the Schools Interoperability
Framework Association with its SIF protocol offer models of the
kind of standard setting that may alleviate some of the difficulties of
accessing data from diverse systems from any number of providers.
Investments in the development of publicly shareable prototypes
and protocols could accelerate the use of data from adaptive technologies in education research.
Tools
The tools used for the analysis of data from adaptive learning
technologies have not been developed with education researchers
in mind. Most of the tools are generic and have a steep learning
curve for scholars outside the specialized areas of inquiry, often
in computer science, for which they have been developed. This
means that education researchers new to adaptive technology will
need to invest considerable time and resources to make good use of
the available tools. Investments in the development of tools would
reduce the burden on researchers and encourage use of data from
adaptive technologies for education research. These investments
could take the form of improved documentation, more intuitive and
understandable interfaces, and enhanced technical support.
22
ADAPTIVE EDUCATIONAL TECHNOLOGIES
Issues in Creating a Data Governance Model
Education researchers are accustomed to dealing with the complexities of securing access to data about students and their learning.
Such efforts address issues of informed consent and the protection of human subjects. They meet the requirements of human subjects committees at the institutions where the research is based as
well as requirements of the schools and districts where the data are
gathered.
The data gathered via adaptive educational technologies present
new complexities for all concerned. Adaptive systems capture data
on students and their learning in ways that may not be transparent to either students or their parents. Because adaptive systems
are often operated by software vendors, publishers, or other third
parties, and because the data are often located in systems physically outside the schools and districts where they are collected, the
various rights to the data may not be clear. This, coupled with the
standards common for other education research (e.g., informed consent), currently presents barriers that make it difficult for researchers
to use data from adaptive systems for education research. However,
there are efforts under way to overcome such barriers through the
development of principles, policies, and practices to leverage the
value of individual data while protecting the privacy rights of individuals. Such efforts include those of the U.S. Department of Education (2012), the OECD (2012), and the World Economic Forum (2012).
Nevertheless, challenges to the evolving policies regarding student
data suggest that issues involving student data are far from settled
(Electronic Privacy Information Center, 2013).
Addressing the complexities of data rights will require the development of a management governance process to specify the various rights to data. Elements of the management governance must
include:
1.
2.
3.
A clear understanding of the kinds of data gathered by
adaptive educational technologies;
Specification of the various rights that may be associated
with data (e.g., the right to delete or modify data, and the
right to use data to enhance the educational experience,
for system improvement, and to address general research
questions);
The parties who have an interest in the data on students
and their learning (e.g., students, parents, teachers, schools,
districts, states, system providers, colleges, employers, governance organizations);
INFRASTRUCTURE NEEDED TO SUPPORT RESEARCH
4.
5.
6.
23
The relationships among the various parties;
The conditions under which specific rights may be exercised; and
The precautions required to protect the interests of each
party involved.
Developing one or more models for a data rights governance
process will save considerable time and expense for all concerned
as they develop data-sharing arrangements.
Additional Reading
For more on the topic of data mining and adaptive learning technologies, see Enhancing Teaching and Learning Through Educational
Data Mining and Learning Analytics: An Issue Brief, published by the
U.S. Department of Education. Copies are available at: http://www.ed.gov/
edblogs/technology/files/2012/03/edm-la-brief.pdf.
6
Next Steps
T
he rapid growth of the utilization of adaptive learning technologies has implications for all concerned: students, parents,
educators, educational agencies, system developers and providers, and education researchers. Workshop participants suggested
some possible next steps to protect the interests of the education
research community and the opportunities for and integrity of the
education research process.
Step 1. Standards for Research Data
The education research community could develop standards for
data gathered through adaptive educational technologies to support
education research. Ideally, these standards would be developed
by a consortium of research associations. These standards could be
used to encourage developers to make provisions for gathering data
as part of the design and development process. Additionally, the
research community could offer a review procedure leading to the
designation of an adaptive technology system as meeting research
standards.
Step 2. Credentials for Education Researchers
The education research community could develop standards for
a program of study leading to proficiency in using data from adaptive educational technologies. Such standards might be developed
25
26
ADAPTIVE EDUCATIONAL TECHNOLOGIES
by a group of graduate programs in conjunction with one or more
research associations. The completion of the program of study or
individual components of study could result in a certificate or other
credential.
Step 3. Guidelines for Human Subjects Committees
The education research community could develop guidelines
for human subjects committees to facilitate the review of research
proposals that involve data from adaptive educational technologies.
These guidelines might address the major concerns posed by the
more complicated data-gathering processes, more elaborate data
structures, and more distributed ownership patterns associated
with adaptive systems. The guidelines could be issued by a consortium of research associations, government agencies, and graduate programs in education research. At the closing session of the
meeting, Bob Hauser noted that the federal government had issued
notice of proposed rulemaking in the area of human subjects in July
2011 and that a key response document had been prepared under
the leadership of Felice Levine of American Educational Research
Association representing the work of a collectivity of social science
groups and organizations. The National Academies’ Committee on
Revisions to the Common Rule for the Protection of Human Subjects in Research in the Behavioral and Social Sciences is currently
working in this area with a report expected later in 2013 (see http://
www8.nationalacademies.org/cp/projectview.aspx?key=49500 for
a description of the project).
The growth of adaptive educational technologies presents new
opportunities for education research that can advance our understanding of student learning and performance. The full participation
of the education research community is necessary to create the conditions that will guarantee that the promise of adaptive educational
technologies is fully realized for research as well as practice.
7
References
Azevedo, R., Guthrie, J. T., & Seibert, D. (2004). The role of self-regulated learning in
fostering students’ conceptual understanding of complex systems with hypermedia. Journal of Educational Computing Research, 30, 87-111.
Baker, R. (2007) Modeling and Understanding Students’ Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.
Baker, R., & Yacef, K. (2009). The state of educational data mining in 2009: A review
and future visions. Journal of Educational Data Mining, 1(1), 3-17.
Barab, S., Gresalfi, M., and Ingram-Goble, A. (2010). Transformational play: Using
games to position person, content, and context. Educational Researcher, 39 (7),
525-536.
Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction, 6, 87-129.
Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User-Adapted Interaction, 11, 87-110.
Castro, F., Vellido, A., Nebot, A., & Mugica, F. (2007). Applying data mining techniques to e-learning problems. Studies in Computational Intelligence, 62, 183-221.
Chi, M., VanLehn, K., Litman, D., & Jordan, P. (2010). Inducing effective pedagogical strategies using learning context features. In P. De Bra, A. Kobsa, & D. Chin
(Eds.), User modeling, adaptation, and personalization, Lecture notes in computer science, Vol. 6075 (pp. 147-158). Berlin and Heidelberg: Springer.
Computer Research Association. (2005). Cyberinfrastructure for education and learning
for the future: A vision and research agenda. Washington, DC: Author.
Crespo, R., Pardo, A., Perez, J., & Kloos, C. (2005). An algorithm for peer review
matching using student profiles based on fuzzy classification and genetic algorithms. In M. Ali & F. Esposito (Eds.), Proceedings of the International Conference on
Innovations in Applied Artificial Intelligence (pp. 685-694). Berlin and Heidelberg:
Springer.
27
28
ADAPTIVE EDUCATIONAL TECHNOLOGIES
Electronic Privacy Information Center. (2013). EPIC vs. U.S. Department of Education: Challenging the Department of Education’s Family Educational Rights and
Privacy Act (FERPA) 2011 Regulations. Washington, DC: Author. Available at
http://epic.org/apa/ferpa/default.html.
Feng, M., & Heffernan, N. T. (2007). Informing teachers live about student learning:
Reporting in ASSISTment system. Technology, Instruction, Cognition, and Learning
Journal, 3(1-2), 63.
Gardner, H. (2009). Personalized education: A quantum leap in learning will allow
everyone to go to the head of the class. Foreign Policy, 172, 86.
Glenn, D. (2008). Huge databases offer a research goldmine and privacy worries.
Chronicle of Higher Education, 54(35), A10.
Heraud, J., France, L., & Mille, A. (2004). Pixed: An ITS that guides students with
the help of learners’ interaction log. In Proceedings of the International Conference
on Intelligent Tutoring Systems (pp. 57-64), Maceio, Alagoas, Brazil. Berlin and
Heidelberg: Germany: Springer.
Hurley, T., & Weibelzahl, S. (2007). Using MotSaRT to support on-line teachers in
student motivation. In R. Duval, R. Klamma, & M. Wolpers (Eds.), Proceedings
of the European Conference on Technology Enhanced Learning (pp. 101-111), Lecture
notes in computer science, Vol. 4753. Berlin and Heidelberg: Springer.
Hwang, G. (1999). A knowledge-based system as an intelligent learning advisor on
computer networks. Journal of Systems, Man, and Cybernetics, 2, 153-158.
Jong, B., Chan, T., & Wu, Y. (2007). Learning log explorer in e-learning diagnosis. IEEE
Transactions on Education, 50(3), 216-228.
Kelly, D., & Tangney, B. (2005). First aid for you: Getting to know your learning
style using machine learning. Advanced Learning Technologies, 1-3, ICALT
2005. Fifth IEEE International Conference on Advanced Learning Technologies.
Washington, DC.
Lajoie, S. P., & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (2nd
ed.) (pp. 803-821). Mahwah, NJ: Erlbaum.
Lovett, M., Meyer, O., & Thille, C. (2008). The open learning initiative: Measuring the
effectiveness of the OLI statistics course in accelerating student learning. Journal
of Interactive Media in Education, 14, 1-16.
Maeroff, G. (2003). A classroom of one: How online learning is changing schools and colleges. New York: Palgrave Macmillan.
McManus, T. F. (2000). Individualizing instruction in a web-based hypermedia learning environment: Nonlinearity, advance organizers, and self-regulated learners.
Journal of Interactive Learning Research, 11(3), 219-251.
Mislevy, R., Behren, J., Bennett, R., Demark, S., Fresso, D., Levy, R., Robinson, D.,
Rutstein, D., Shute, V., Stanley, K., & Winters, F. (2010). On the roles of external
knowledge representations in assessment design. Journal of Technology, Learning
and Assessment, 8(2).
Muehlenbrock, M. (2005). Automatic action analysis in an interactive learning environment. Usage Analysis in Learning Systems. Twelfth International Conference
on Artificial Intelligence in Education. Amsterdam, The Netherlands (pp. 73-80).
National Research Council. (2011). Learning Science Through Computer Games and Simulations. Washington, DC: The National Academies Press.
National Science Foundation Task Force on Cyberlearning. (2008). Fostering learning
in a networked world: The cyberlearning opportunity and challenge. Washington, DC:
National Science Foundation.
Nissenbaum, H. (2010). Privacy in context: Technology, policy, and the integrity of
social life. Palo Alto, CA: Stanford University Press.
REFERENCES
29
OECD. (2012). The Protection of Children Online: Recommendation of the OECD
Council, Report on Risks Faced by Children Online and Policies to Protect Them.
Paris: OECD. Available at http://www.oecd.org/sti/ieconomy/childrenonline_
with_cover.pdf.
Office of Science and Technology Policy, Executive Office of the President (2012). Fact
Sheet: Unlocking the Power of Education Data for All Americans. Washington,
DC: The White House.
Pavlik, P., Cen, H., & Koedinger, K. (2009). Learning factors transfer analysis: Using
learning curve analysis to automatically generate domain models. In T. Barnes,
M. Desmarais, C. Romero, & S. Ventura (Eds.), Proceedings of the 2nd Inter national
Conference on Educational Data Mining (pp. 121-130). International Educational
Data Mining Society.
Pierce, N., Conlan, O., & Wade, V. (2008). Adaptive educational games: Providing
non-invasive personalized learning experiences. In Second IEEE International
Conference on Digital Games and Intelligent Toy Enhanced Learning (pp. 28-35).
Washington, DC: IEEE Computer Society.
Pitman, J. and McLaughlin, B. (2000). Making cyberspace safe for children. Educational
Leadership, 67(6), 67-71.
Reese, D. D. (2012). CyGaMEs: A full service instructional design model harnessing
game-based technologies for learning and assessment. In L. Moller & J. B. Huett
(Eds.), The next generation of distance education (pp. 157-170). New York: Springer.
Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’
help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21, 267-280.
Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to
2005. Expert Systems with Applications, 33, 135-146.
Romero, C., Ventura, S., & De Bra, P. (2004). Knowledge discovery with genetic
programming for providing feedback to courseware. User Modeling and UserAdapted Interaction, 14(5), 425-464.
Romero, C., Ventura, S., Pechenisky, M., & Baker, R. (2011). Handbook of educational
data mining. Boca Raton, FL: CRC Press.
Romero, C., Ventura, S., Zafra, A., & De Bra, P. (2009). Applying web usage mining for
personalizing hyperlinks in web-based adaptive education systems. Computers
and Education, 53(3), 828-840.
Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning
in video games. Cambridge, MA: MIT Press.
Simko, M. & Bielikova, M. (2009). Automatic concept relationships discovery for an
adaptive e-course. In Proceeding of the 2nd International Conference on Educational
Data Mining (pp. 171–178).
Skinner, B. F. (1986). Programmed instruction revisited. Phi Delta Kappan, 68, 103-110.
Smith, S. G., & Sherwood, B. A. (1976). Educational uses of the PLATO computer
system. Science, 192(4237), 344-352.
Stephens, C., & Sukumar, R. (2006). An introduction to data mining. In R. Grover &
M. Vriens (Eds.), Handbook of marketing research (pp. 455-485). Thousand Oaks,
CA: Sage.
Superby, J., Vandamme, J., & Meskens, N. (2006). Determination of factors influencing
the achievement of first-year university students using data mining methods. In
Proceedings of the Workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems (pp. 37-44). Berlin and Heidelberg: Springer.
Suppes, P., & Fortune, R. F. (1985). Computer assisted instruction: Possibilities and
problems. NASSP Bulletin, 69, 30-34.
30
ADAPTIVE EDUCATIONAL TECHNOLOGIES
Tang, C., Lau, R., Li, Q., Yin, H., Li, T., & Kilis, D. (2000). Personalized courseware
construction based on web data mining. In Proceedings of the First International
Conference on Web Information Systems Engineering (pp. 204-211). Washington, DC:
IEEE Computer Society.
Tang, T., & McCalla, G. (2005). Smart recommendation for an evolving e-learning
system. International Journal on E-Learning, 4(1), 105-129.
U.S. Congress, Office of Technology Assessment. (1988). Power on: New tools for teaching and learning. Washington, DC: U.S. Government Printing Office.
U.S. Department of Education. (2012). Safeguarding Student Privacy. Washington,
DC: Author. Available at: http://www2.ed.gov/policy/gen/guid/fpco/ferpa/
safeguarding-student-privacy.pdf.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring
systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.
Vialardi, C., Bravo, J., & Ortigosa, A. (2008). Improving AEH courses through log
analysis. Journal of Universal Computer Science, 14(17), 2777-2798.
Woolf, B. (2009). Building intelligent interactive tutors. Burlington, MA: Morgan
Kaufmann.
World Economic Forum. (2012). Rethinking Personal Data: Strengthening Trust. Geneva,
Switzerland: World Economic Forum. Available at http://www.weforum.org/
reports/rethinking-personal-data-strengthening-trust.
Zakrzewska, Z. (2008). Cluster analysis for user’s modeling in intelligent e-learning
systems. In Proceedings of the 21st International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (pp. 209-214). Berlin and
Heidelberg: Springer.
APPENDIX A
Workshop Agenda
Planning Meeting
May 12, 2011
Keck Center—Room 101
500 Fifth Street, NW
Washington, DC 20001
MEETING AGENDA
Thursday, May 12
8:00-8:30 am
Breakfast
8:30-8:40 am
Welcome and Meeting Overview
Susan Fuhrman, Teachers College, Columbia
University; President, National Academy of
Education
8:40-9:00 am
Participant Introductions
9:00-10:00 am
Product Demonstrations—
Envisioning the Scope of AETs
Chair: Susan Fuhrman
15-minute demos—focused equally on the product
and on the data captured
• MasteringPhysics(RasilWarnakulasooriya)
• WISE(MarciaLinn)
• ASSISTments(NeilHeffernan)
31
32
10:00-10:30 am
10:30-11:00 am
ADAPTIVE EDUCATIONAL TECHNOLOGIES
Discussion—Analysis of Data from AETs
• 10-minutepresentationontheliterature
(Ken Koedinger)
• 20-minutegroupdiscussion
Student Learning from AETs
• 10-minutepresentationondataarchiving
(John Stamper, PSLC Datashop)
• 20-minutegroupdiscussion
11:00 am-12:00 pm Roundtable—Developing Models Using
Data from AETs
10-minute presentations, followed by group
discussion
• StudentAffect(BobDolan)
• SocialNetworks(ShaneDawson)
• TeacherImplementation(BrianRowan)
• InterventionEffectiveness(GuidoGatti)
12:00-12:30 pm
Working Lunch: Discussion of Background
Paper
Chair: James Gee, Arizona State University
12:30-3:30 pm
Addressing the 3Q’s and Identifying Topics
for the Summit
Chair: Brian Rowan, University of Michigan
Discussion of Question #1:
• Whatresearchopportunitiesarepossible
using these data?
Discussion of Question #2:
• Whatkindsofanalyseshaveresearchers
conducted in the past using such data?
And, what has been learned from such
analyses?
Discussion of Question #3:
• Whatmoreisneededtodevelopresearchin
this area?
— What are the costs and benefits of using
such data for research?
33
APPENDIX A
— What kind of organizational supports
would be needed from developers if
data were used for research and program
improvement?
— What other accommodations might be
needed for researchers (e.g., to ensure
confidentiality of data, allow data to be
processed statistically, etc.)?
3:30-4:00 pm
Wrap-up, concluding comments, and next
steps
4:00 pm
Meeting adjourned
34
ADAPTIVE EDUCATIONAL TECHNOLOGIES
WORKSHOP PARTICIPANT LIST
May 12, 2011, Planning Meeting
Co-Chairs:
Susan Fuhrman, Teachers College, Columbia University
James Gee, Arizona State University
Brian Rowan, University of Michigan
Participants:
Judie Ahn, National Academy of Education
Roger Azevedo, McGill University
Sasha Barab, Indiana University
John Behrens, CISCO
Larry Berger, Wireless Generation
Christopher Brown, Pearson Foundation Research Program
Allan Collins, Northwestern University
Katie Conway, Teachers College, Columbia University
Shane Dawson, University of British Columbia
Andrea diSessa, University of California, Berkeley
Bob Dolan, Assessment & Information, Pearson
Guido Gatti, Gatti Evaluation, Inc.
Richard Halverson, University of Wisconsin-Madison
Michael Hansen, Urban Institute and CALDER
Aaron Harnly, Wireless Generation
Neil Heffernan, Worcester Polytechnic Institute
Paul Horwitz, Concord Consortium
Caitlin Kelleher, Washington University in St. Louis
Ken Koedinger, Carnegie Mellon University
Carol Lee, Northwestern University
Marcia Linn, University of California, Berkeley
Robert Mislevy, University of Maryland
Fred Mueller, Pearson Learning Technologies Group
Gary Natriello, Teachers College, Columbia University
Zoran Popovic, University of Washington
Seth Reichlin, Pearson
Erin Reilly, University of Southern California
Steve Ritter, Carnegie Learning
Dan Schwartz, Stanford University
David Shaffer, University of Wisconsin-Madison
John Stamper, PSLC DataShop
Elizabeth Tipton, Teachers College, Columbia University
APPENDIX A
Kurt VanLehn, Arizona State University
Rasil Warnakulasooriya, Pearson Learning Technologies Group
Gregory White, National Academy of Education
35
APPENDIX B
Summit Agenda
Agenda for December 1-2, 2011 Summit
Keck Center
500 Fifth Street, NW
Washington, DC 20001
All locations are Keck 100 unless otherwise specified
Day 1: Where We’ve Been
8:30–9:00 am
Continental Breakfast
9:00–9:30 am
Welcome
9:30–11:00 am
Demonstrations (Keck 100 and Breakout
Rooms)
11:00 am–12:15 pm
Learning Panel
This panel is about our ability to learn about
learning through AET data analysis. The panel
will focus on cognition, and social and emotional learning, as well as contextual factors.
It will include theory building opportunities
and the development of new learning models,
as well as the possibilities to conduct pioneering studies in learning and development.
Moderator: Susan Fuhrman, Teachers
College, Columbia University
Panelists:
Jere Confrey, North Carolina State
University
James Gee, Arizona State University
Roy Pea, Stanford University
12:15–12:30 pm
Keynote Address: Senator Michael Bennet
12:30–1:15 pm
Lunch (Keck Cafeteria, 3rd Floor)
37
38
ADAPTIVE EDUCATIONAL TECHNOLOGIES
1:15–2:30 pm
Instruction Panel
This panel will focus on how instructional
practices have changed because of new technologies, as well as how they contribute to
the ability to assess the effects of instructional approaches.
Moderator: Brian Rowan, University of
Michigan
Panelists:
Sasha Barab, Arizona State University
Arthur Graesser, University of Memphis
David Pritchard, MIT
2:30–2:45 pm
Break
2:45–4:00 pm
Assessment Panel
This panel will focus on the immediate feedback on student progress allowed by these
technologies and the possibilities for tailoring instruction as a result.
Moderator: James Gee
Panelists:
Robert Mislevy, University of Maryland,
College Park
David Shaffer, University of
Wisconsin–Madison
Valerie Shute, Florida State University
4:00–5:15 pm
Concerns Panel
This panel will focus on how to best address
privacy and proprietary concerns, ensure
quality control, and ensure theoretically
sound analyses.
Moderator: Brian Rowan
Panelists:
George Alter, Inter-University
Consortium for Political and Social
Research, University of Michigan
Gary Natriello, Teachers College,
Columbia University
Lauren Resnick, University of Pittsburgh
John Stamper, Carnegie Mellon
University and Pittsburgh Science of
Learning Center DataShop
39
APPENDIX B
5:15 pm
Reception (Keck Atrium, 3rd Floor)
Day 2: Where We’re Going
8:30–9:00 am
Continental Breakfast
9:00–10:15 am
Institutional Responses and Innovation
Panel
This panel focuses on how AETs influence
institutions and on providing data-based feedback to schools and other learning settings.
Moderator: Susan Fuhrman
Panelists:
Richard Halverson, University of
Wisconsin–Madison
Ken Koedinger, Carnegie Mellon
University
Marcia Linn, University of California,
Berkeley
10:15–11:30 am
Developing Infrastructure
This panel includes the roles for public and
private enterprise in building AET data analysis as a field. It will also focus on roles for a
variety of stakeholders, including researchers,
instructors, developers, and end users.
Moderator: James Gee
Panelists:
John Behrens, CISCO
Ed Dieterle, Bill & Melinda Gates
Foundation
Carl Wieman, Office of Science and
Technology Policy, Executive Office
of the President
11:30 am–12:00 pm
Closing
40
ADAPTIVE EDUCATIONAL TECHNOLOGIES
SUMMIT PARTICIPANT LIST
December 1-2, 2011 ,Summit
Chairs:
Susan Fuhrman
James Gee
Brian Rowan
Teachers College, Columbia University
Arizona State University
University of Michigan
Participants:
George Alter*
Eva Baker
Marni Baker
Ryan Baker
Marianne Bakia
Sasha Barab*
John Behrens*
Randy Bennett
Marie Bienkowski
Christopher Brown
Jack Buckley
Jamika Burge
Steve Cantrell
Isabel Cardenas-Navia
Karen Cator
John Cherniavsky
Jody Clarke-Midura
Stephen Coller
Allan Collins
Jere Confrey*
Lyn Corno
William Cox
Richard Culatta
Phil Daro
Shane Dawson
Arlene de Strulle
Chris Dede
Ed Dieterle*
Bob Dolan**
[* panelist]
[** demonstration]
University of Michigan
UCLA
Columbia University
Worcester Polytechnic Institute
SRI International
Arizona State University
CISCO
ETS
SRI International
Pearson
National Center for Education Statistics
DARPA (i_SW)
Bill & Melinda Gates Foundation
Office of Naval Research
U.S. Department of Education
NSF Division of Research on Learning
Harvard Graduate School of Education
Bill & Melinda Gates Foundation
Northwestern University
North Carolina State University
Teachers College, Columbia University
DSA Capital
U.S. Department of Education
University of California, Berkeley
University of British Columbia
National Science Foundation
Harvard University
Bill & Melinda Gates Foundation
Pearson
APPENDIX B
Nancy Doorey
Janice Earle
John Easton
Stuart Elliott
Robert Floden
Daniel Goroff
Art Graesser*
Richard Halverson*
Jane Hannaway
Aaron Harnly**
Robert Hauser
Ryan Heath
Neil Heffernan
Laurence Holt**
Paul Horwitz**
Kim Jacobson
Thomas James
Caitlin Kelleher
Anthony Kelly
Diane Jass Ketelhut
Don Knezek
Kenneth Koedinger*
Janet Kolodner
Keith Krueger
Andrew Latham
Eric Lindland
Marcia Linn*
Christopher Lohse
Ellen Meier
Edward Metz
Natalie Milman
Jessica Mislevy
Robert Mislevy*
Gary Natriello*
Brian Nelson
Kenny Nguyen
Barbara Olds
Nicole Panorkou
Roy Pea*
Kathy Perkins**
Jefferson Pestronk
41
ETS
National Science Foundation
IES, U.S. Department of Education
National Research Council
Michigan State University
Sloan Foundation
University of Memphis
University of Wisconsin-Madison
CALDER/AIR
Wireless Generation
National Research Council
Columbia University
Worcester Polytechnic Institute
Wireless Generation
The Concord Consortium
Junyo
Teachers College, Columbia University
Washington University in St. Louis
George Mason University
University of Maryland, College Park
International Society for Technology in
Education (ISTE)
Carnegie Mellon University
National Science Foundation
Consortium for School Networking
ETS
Frameworks Institute
University of California, Berkeley
Council of Chief State School Officers
Teachers College, Columbia University
IES, U.S. Department of Education
George Washington University, GSEHD
SRI International
ETS
Teachers College, Columbia University
Arizona State University
Friday Institute for Educational
Innovation
National Science Foundation
North Carolina State University
Stanford University
University of Colorado at Boulder
U.S. Department of Education
42
Penelope Peterson
Dave Pritchard*,**
Lauren Resnick*
Steven Ritter**
Patrick Rooney
Mark Schneiderman
Steve Schoettler
Marilyn Seastrom
David Shaffer*
Russell Shilling
Valerie Shute*
Chuck Simon**
Emily Dalton Smith
Mike Smith
Sarah Sparks
John Stamper*
Constance Steinkuehler
Squire
Ken Stephens
James Stigler
Martin Storksdieck
Jana Sukkarieh
Elizabeth Tipton
Greg Tobin
Robert Torres
Elizabeth VanderPutten
Hugh Walkup
Denny Way
Sandra Welch
Carl Wieman*
Lauren Young
Sabine Zander
ADAPTIVE EDUCATIONAL TECHNOLOGIES
Northwestern University
MIT
University of Pittsburgh
Carnegie Learning
U.S. Department of Education
Software & Information Industry
Association
Zynga
NCES, U.S. Department of Education
University of Wisconsin–Madison
DARPA
Florida State University
Pearson Digital Learning
Bill & Melinda Gates Foundation
Carnegie Foundation for Advancement
of Teaching
Education Week
Carnegie Mellon University
White House, OSTP
Pearson plc
UCLA
National Research Council
ETS
Teachers College, Columbia University
Pearson
Bill & Melinda Gates Foundation
National Science Foundation
U.S. Department of Education, Office of
Educational Technology
Pearson plc
National Science Foundation
White House, OSTP
Spencer Foundation
Teachers College, Columbia University
Staff:
Judie Ahn
Katie Conway
Regan Ford
Philip Perrin
Jennifer Tinch
Gregory White
National Academy of Education
Teachers College, Columbia University
National Academy of Education
National Academy of Education
National Academy of Education
National Academy of Education
Adaptive Educational Technologies
TOOLS FOR LEARNING and for
LEARNING ABOUT LEARNING
The National Academy of Education advances high quality education research
and its use in policy formation and practice. Founded in 1965, the Academy consists
of U.S. members and foreign associates who are elected on the basis of outstanding
scholarship related to education. Since its establishment, the Academy has undertaken
research studies that address pressing issues in education, which are typically conducted
by members and other scholars with relevant expertise. In addition, the Academy sponsors professional development fellowship programs that contribute to the preparation of
the next generation of scholars.