A Study on the Development of Evaluation Criteria for Digital Textbooks
in Korea
Hyung Mi Joo, Chong Uk Ahn
Division for Textbook Authorization
Korea Institute for Curriculum and Evaluation Korea
joohm@kice.re.kr, ahncu@kice.re.kr
Abstract: In Korea, digital textbooks will be developed from the 3rd grade to the 11th grade
primarily for the subject matters of English, Science, and Social Studies in 2013, and then be
applied to schools in 2014. The purpose of this study was to develop the evaluation criteria for
digital textbooks in Korea. The study employed the Delphi method and textbook analysis. The
thirty experts answered questionnaires in three rounds. It also analyzed the four types of the
establishing digital textbooks which were categorized according to textbook developers. This
study concluded that the evaluation criteria for digital textbook consist of 4 evaluation
domains and 17 questions. This study suggests that the evaluation criteria for digital textbooks
need to be established based upon the integration of technology evaluation with content
evaluation. Further research should be needed to identify the problems and challenges which
may potentially occur during the evaluation processes of digital textbooks.
Introduction
The development of computer technology has innovated concepts and issues of material designs,
pedagogical theories, and modes of instruction in education. More recently, the innovative technology such as a
tablet PC and a cloud computing system spread around educational fields, so that school environments need to
be changed accordingly. As computer technology innovations have applied substantially in various educational
settings, the Korean government has also come to emphasize the application of computer technology to
education. The Ministry of Education in Korea announced ‘The SMART Education Plan’ in 2011, which is an
overall educational innovation plan on the improvement of school textbooks and the construction of educational
infrastructure facilities for cloud computing(MEST, 2011). Here, ‘SMART’ is an acronym for Self-directed,
Motivated, Adaptive, Resource-enriched, and Technology-embedded. One of its strategic plans is the
development and implementation of digital textbooks. The Korean government defines a digital textbook as a
textbook for the future generation, which keeps an image of a printed textbook; on the other hand, contains
various multimedia materials and functions, and provides or connects to Learning Management System(LMS)
functions. Students and teachers will be able to utilize the digital textbook anytime and anywhere, using diverse
mobile devices such as smart phone, smart pad, and so on. The government expects the development and
implementation of a digital textbook to be the first step of the educational reform in Korea, which is suitable for
fast changing learning environments.
Digital textbooks will be applied to from the 3rd grade in elementary school to the 2nd grade in high
school. Digital textbooks will be developed primarily for the subject matters of English, Science, and Social
Studies in 2013, and be applied to schools in 2014(MEST, 2012). Meanwhile, digital textbooks need to be
authorized or approved by the government before applying to schools. Korea Institute for Curriculum and
Evaluation(KICE), which is a government research organization and is responsible for textbook authorization on
behalf of the Ministry of Education, needs to develop the evaluation criteria to authorize digital textbooks.
The Study
The purpose of the study was to develop the evaluation criteria for digital textbooks. In order to
achieve the purpose, this study first analyzed the characteristics of the existing digital textbooks. Second, it
established the evaluation criteria of contents in digital textbooks which were based on the main characteristics.
Third, it provided some suggestions and policy implications in terms of the authorization of digital textbooks.
This study utilized some research methods of literature review, textbook analysis, a Delphi method, and expert
panel meetings.
Through the literature review, this study investigated the differences of digital functions between
diverse domestic and foreign digital textbooks in terms of the types of learning materials, interfaces, and design
principles. And it found some main characteristics of digital textbooks. The study analyzed the establishing
digital textbooks. The four types of digital textbooks were categorized according to textbook developers. The
first type of the digital textbooks was developed by a government research organization, Korea Education &
Research Information Service(KERIS). It has been used for about four years in some experimental schools in
order to examine the effectiveness of digital textbooks. The second one was developed by domestic publishing
companies. The third one was developed by the government organization, the Korea Foundation for the
Advancement of Science and Creativity(KOFAC). This study analyzed the Physics digital textbook for middle
school students, which is an application containing the contents of virtual experiments and simulations. The
fourth one was developed in foreign countries such as the United States of America and Japan. This study
analyzed the science digital textbooks developed by publishers such as Pearson and McGraw-Hill in the United
States. And in case of Japan, it analyzed the Arts digital textbooks developed by the publisher of Kairyudo
which is one of the big publish companies in Japan.
In addition, this study employed a Delphi method to develop the evaluation criteria for digital textbooks.
The thirty experts answered the questionnaires in three rounds. The experts were professors, researchers,
government officials, and teachers who have experience with developing and utilizing digital textbooks,
e-learning materials, educational software programs, and the like. The experts were given the 1st, 2nd, and 3rd
drafts of the evaluation criteria for digital textbooks. They provided their opinions about the criteria.
Findings
The final version of the evaluation criteria for digital textbooks was completed based on the textbook
analysis and the results of the Delphi method. The evaluation criteria for digital textbook consist of 4 evaluation
domains and 17 evaluation questions. ‘Domain I: Compliance with national curriculum’ contains four questions
to examine whether a digital textbook meets with the ‘objects,’ ‘contents,’ ‘instructional methods,’ and
‘evaluation’ in the national curriculum. ‘Domain II: Accuracy and Appropriateness of Contents’ deals with
seven questions to evaluate whether 1) the level and scopes of contents are appropriate for students in use, 2) the
contents are accurate, 3) the contents are objective, neutral, and fair, 4) the contents contain diverse types and
sufficient amounts, 5) the sources of contents are accurately presented, 6) the contents motivate students’
interests, and 7) the orthography and expressions of the sentences are accurate. ‘Domain III: Effectiveness of
Teaching and Learning Support Functions’ has four questions to evaluate whether 1) a digital textbook
effectively supports students’ self-directed learning, 2) the teaching and learning support functions are
effectively served, 3) a digital textbook effectively promotes interactive learning, and 4) a digital textbook
effectively and easily connects to external instructional materials and programs(applications). Lastly, ‘Domain
IV: Usability of Interface has two questions to check 1) the interface such as button, menu, icon, and the like is
convenient for students to use and 2) the design such as color and typography has the aesthetic appreciation and
is effective in learning. The evaluation domains and questions for digital textbooks are as follows:
Evaluation Domains
I. Compliance with
National Curriculum
Evaluation Criteria
Evaluation Questions
1. Does the digital textbook meet with ‘Objectives’ of the national curriculum?
2. Does the digital textbook meet with ‘Contents’ of the national curriculum?
3. Does the digital textbook meet with ‘Teaching & Learning Methods’ of the
national curriculum?
4. Does the digital textbook meet with ‘Evaluation’ of the national curriculum?
5. Are the scope and level of the contents appropriate for students?
6. Are the contents accurate?
7. Are the contents objectively, neutrally, and fairly described?
II. Accuracy and
Appropriateness of
Contents
8. Does the digital textbook provide a variety of and sufficient amount of
contents?
9. Are the citations and the sources of the contents accurate?
10. Does the contents motivate students?
11. Are the orthography and the expressions of the sentences accurate?
12. Are the digital textbook well-organized for self-directed learning?
III. Effectiveness of
Teaching and
Learning Support
Functions
13. Are the teaching and learning support functions effectively realized to
enhance students’ learning?
14. Does the digital textbook provide functions for student-student,
student-teaching, and student-contents interactions?
15. Are the digital textbook easily and conveniently connect to external
instructional materials and programs(applications)?
IV. Usability of
Interface
16. Is the interface such as button, menu, icon, and the like convenient for
students to use?
17. Does the design such as color and typography have the aesthetic appreciation
and is it effective in learning?
Table 1: Evaluation criteria for digital textbooks
Conclusions
It is the ultimate aim of the digital textbook policy of the Korean government that students would be
effectively learned through digital textbooks. Therefore, the government needs to provide digital textbooks with
high quality and diversity to students. The evaluation criteria this study established are expected to be used as
the primary guidelines for the development of digital textbooks with high quality.
This study suggests that the evaluation criteria for digital textbooks should be established based upon
the integration of technology evaluation with content evaluation. It also recommends that further research is
needed to identify the problems and challenges which may potentially occur during the evaluation processes of
digital textbooks.
References
Ministry of Education and Science Technology. (2011). Strategy plans for SMART education. Seoul: Ministry
of Education and Science Technology.
Ministry of Education and Science Technology. (2012). 2012 Strategy plans for the development and
implementation of digital textbooks . Seoul: Ministry of Education and Science Technology.