Page 1 of 7
Erik Mitchell
Comprehensive Exam Question 4
4. Metadata Literacy
You briefly discuss pedagogy and learning theory with regard to metadata literacy. One
area missing from this discussion, however, is discussion of instructional goals and objectives. If
the purpose of this research project is to determine to what extent individuals are comfortable
engaging with metadata, then an inevitable future direction for this research is to increase
individuals' comfort with metadata.
Please discuss how librarians and/or other instructors could/should construct instructional
goals and objectives to increase individuals' metadata literacy, and assess individuals learning,
with reference to the literature on instructional design and evaluation.
Time started: 8:15
Time completed: 11:47
Overview
This question asks me to discuss how librarians and educators could construct instructional goals and
objectives in order to increase students’ levels of metadata literacy. In order to answer this question,
this response will; a) examine the foundations of metadata literacy in relation to other forms of literacy;
b) discuss the role of metadata in learning environments; c)review relevant approaches to instructional
design and evaluation; and d) propose a method for creating and evaluating instructional goals and
outcomes for metadata literacy. Throughout this response, two themes will recur. First, the idea that
this approach for defining instructional goals and objectives works for any form of literacy and second
that a key aspect of effective information literacy instruction is recognizing the connection to domain
knowledge or a larger curriculum relevant for that form of literacy.
1. What is metadata literacy
Metadata literacy (ML) has been defined in the ML literature review as the ability to conceptualize,
create, and work with metadata. This definition is based on the information literacy (IL) framework that
was created in the IL literature review and reflects the three ideas that literacy includes skill and
conceptual elements, that it occurs in specific environments and contexts, and that it views the student
as an active participant in learning. Many summary literature reviews on IL (Bawden, Sundin, OwusuAnsah) discuss how IL includes a wide variety of specific literacies such as reading literacy, social literacy,
digital literacy, and work literacy. Given the wide body of literature on this topic, it is relevant to ask
why metadata literacy warrants its own area of discussion. It is appropriate in part because the field of
IL accepts the definition of both narrowly focused forms of literacy and broadly defined forms of
literacies (aka. Meta-literacy). Metadata literacy uses this approach by examining the role of metadata
across multiple literacy concepts. Metadata literacy emerged as a relevant concept in thinking about
literacy in relation to the use of information communication technology (ICT) in part because the use of
Page 2 of 7
information communication technology (ICT) in learning is increasing at a quick pace. As demonstrated
in the metadata literacy review, metadata is an integral part of ICT and digital documents (Sen et. al,
Lotherington) which means that it is also being increasingly used in learning environments. Despite this
increased use, research has shown that organization and metadata tasks are not a naturally held skill by
most students (Nicol et al). For this reason it is important to think about metadata skills and
conceptual understandings in regards to literacy.
The field of librarianship is the only place where the concept of ML is defined. Articles by Schwartz,
Intner, and a recent ALCTS discussion (ALA 2009?) focus on a professional definition of IL. The
consensus of these articles is that metadata has become a ‘core competency’ for librarians who need to
understand it in order to effectively create and use new information tools. There are articles which
touch on metadata issues in relation to learning however including Hert et. al., Ju, and Jacob. Further,
there are literacy models such as the National Research Council fluency with information technology
(FIT) which include metadata tasks and concepts. The idea of ML is based on these ideas and other
literacy models reviewed in the IL literature review including Bruce, Horton, Hughes & Shapiro, and Talja
et al. The two key elements from the literature reviewed relevant to the idea of ML are a consideration
of the role of metadata in ICT and the role of metadata in social information spaces. As such, ML can be
thought of both as a ‘stand alone’ form of literacy in regards to the librarian view of ML or as an
integrated or meta-literacy that impacts learning in specific environments. The following discussion of
how to teach and evaluate ML considers both of these perspectives.
2. What role does ML play in learning?
ML has been shown to play a role in learning. Hert et. al found that metadata is used more by experts
than non experts and that metadata served an important scaffolding role in enabling learning during
use. This idea of scaffolding and the importance of tasks such as categorization as part of the learning
process are also seen in works by Ju and Jacob. Further, Hert et al. documented how domain knowledge
influences the ability of the student to recognize and interact with ICT. This literature suggests that
there is a feedback loop which occurs in ICT with regards to metadata. First, metadata serves a
scaffolding role – allowing students to recognize and use information with which they are not readily
familiar. Second, metadata serves to enable more advanced use of the system as participants
understand the context of the data in ICT.
Metadata has been shown to be an integral part of ICT as well. Sen et al. discuss how tagging is a
common feature in social/community sites while Churches discusses the popularity of such sites as
learning platforms. Masielo(?)(sic) discusses how enthusiasm with regards to new approaches to
learning management systems are leading teachers and students from traditional systems to newer,
social based systems. An example of this is the increasing use of facebook (Smith, Mitchell) as a learning
management and collaborative platform.
3. Instructional design and evaluation
Page 3 of 7
Although there is a lot of enthusiasm surrounding the use of ICT (and correspondingly metadata tools) in
teaching, there is a documented disconnect between the fields of teaching and technology. Lajoie &
Azevedo discuss the gap between research and practice with regards to ICT while Tuominen discusses
the lack of research between information literacy and information technology. Likewise, Masielo(?)
observes that any use of ICT needs to have a strong pedagogical focus developed in conjunction with
faculty. In thinking about incorporating metadata literacy into a curriculum it is important to consider
what learning objectives are being filled. The discussion of instructional design and evaluation
approaches for ML below assumes that the curriculum benefits from the use of metadata as either a
learning tool or outcome.
The literacy framework discussed in the IL and ML literature reviews shows the relationship between ML
skills and concepts and teaching pedagogy and learning theory. Each axis of the table includes an
element which focuses on the context (ML perspective) or environment (teaching perspective) of the ML
interaction. Considering the context (e.g. is ML instruction primary or secondary to the learning
objective) and environment (e.g. what ICT tool is being used?) of the instructional element is important
because metadata literacy is most applicable in digital and social environments. It does not make sense,
for example, to focus on metadata when working with print-based narrative texts. However, if asking
students to work in a metadata-rich ICT environment or if teaching a subject matter that naturally
integrates with the ideas of categorization, tagging, or other metadata tasks it makes sense to
incorporate into instructional design.
3.1. What are some current approaches to instructional design?
Areas of instructional design that are relevant for ML include the ideas of constructivism, problem based
learning, and active learning. These approaches to teaching work well with ML concepts and typically
include environments in which metadata plays a strong role. These are certainly some major areas, but
aren’t there possibly more? For instance group work, where tagging, shared reviews, etc can play a bit
role.
For example, constructivist/problem based learning (PBL) environments (Brooks & Brooks) are typically
student driven, contextualized by ill defined problems, employ scaffolding techniques where
appropriate to support learning, and focus on having the instructor serve a mentor rather than authority
role. Some of the necessary elements of these environments include the need for an authentic
problem to motivate students, ongoing constructive feedback, motivational elements to encourage
students to take ownership of the learning process, and if group work is used, appropriate group
definition and feedback. Constructivism often includes ML elements in that; a) scaffolding can include
the use of metadata and categorization elements; b)students are asked to engage in metadata tasks
such as tagging, categorizing, harvesting, re-using during their research and reporting; and c)metadata
tasks tend to be a core element of social and group based ICT tools. For example, popular ICT and LMS
tools include blogs, wikis, and facebook groups. Each of these platforms includes metadata tasks as part
of the interface.
Page 4 of 7
Active learning is another form of instructional design which emphases student participation in the
learning process. It parallels the approaches of constructivism but also places an emphasis on student
driven exercise and participation as opposed to lecture or discussion. The relationship of ML to active
learning tasks can be seen in the use of abstracting tasks to engage students (Pinto et. al), the creation
of digital libraries as part of the learning experience (Mitchell – unpublished), and the creation of a
student-populated information literacy wiki (Smith, Mitchell, Numbers) as the course output. Each of
these three examples required students to engage with metadata as part of their learning process and
required the instructors to discuss the role of metadata at some level with the students.
3.2. What are some forms of evaluation?
Evaluation of student work in ICT can be difficult in that objective metrics of ICT use do not necessarily
correspond with learning. Nickles (sic) discusses using process based evaluation (e.g. did the student
access a resource, did they complete objectives) and how that falls short of evaluating actual learning.
(? – can’t remember name right now) reported difficulties in using logging to determine if students had
accessed a resource, asserting that it tracked only a very low level of participation. Smith, Mitchell, and
Numbers found that attempting to use wiki page edit history allowed the instructors to see exactly what
each group member had contributed but also found that students tended to parcel out the wiki page
creation to a specific group member so that these edit histories were not accurate. Each of the above
methods is a form of summative evaluation which attempts to record student effort but fails to evaluate
a level of learning. There are several other forms of summative evaluation which would prove effective
in evaluating literacy such as tests, reports, and presentations in which students were required to
discuss their engagement with a specific literacy task. These forms of evaluation are valid approaches to
evaluating learning but also tend to identify only a limited set of elements with regards to learning. In
contrast to summative evaluation, formative evaluation tends to be used more heavily in constructivist
learning environments and can be more successful in conducing a holistic evaluation of learning levels.
Formative evaluation includes ongoing feedback from both student and instructor sources, progress
reports, and group discussion. Two specific forms of formative feedback; a)student driven feedback and
b)instructor rating of progress are discussed in the following paragraph.
One of the main benefits of student-driven feedback is that it encourages the student to take ownership
of evaluation and encourages self and group reflection. An example of this approach is self-efficacy
evaluation (Kurbanolugu(sic)). Self-efficacy evaluation is a student driven evaluation of their perception
of their abilities in a given area. It combines a cognitive estimate (can I do this) and an affective
estimate (how confident am I). It can be an effective measure of competency in a given area and can be
used as a tool to allow students to understand their strengths and weaknesses. An example of
formative instructor progress rating includes the evaluation of student work using Bloom’s taxonomy.
This approach is particularly valid in metadata literacy because, although ML may include objective
evaluation of correctness (e.g. did the student correctly tag an item, did they appropriately describe an
item), these forms of evaluation tend to be single dimensional and do not allow an evaluation of an
overall level of literacy. Bloom’s taxonomy as revised by Krathwohl includes six levels of knowledge
Page 5 of 7
ability; remember, understand, apply, analyze, evaluate and create. It also includes four types of
knowledge; factual, procedural, conceptual, and metacognitive. In the education arena, Bloom’s
taxonomy is operationalized for evaluation by framing questions related to a specific competency in
relation to the framework. For example, with regards to metadata literacy in evaluating the factual
knowledge of tagging, the instructor could ask during evaluation “Was the student able to recognize
tags, was the student able to use tags to discover more information, was the student able to create new
tags to describe information?” These questions allow the instructor to pinpoint student literacy levels in
a way that provides a more holistic view of their literacy but also which gives direction on where
improvement is needed. A chart of the version of bloom’s taxonomy updated by Krathwohl is replicated
in table 1.
Table 1 Blooms taxonomy (updated)
Remember
Understand
Apply
Analyze
Evaluate
Create
Factual
Procedural
Conceptual
Metacognitive
In this table, each type of knowledge (factual, procedural, conceptual, and metacognitive) allows the
instructor to evaluate student achievement along a different axis. For example, if the goal is to enable
students to think metacognitively about their research, they would ask questions about whether or not
the student identified ways of monitoring their research, employed techniques to identify gaps in
research, evaluated their own progress, or created new techniques to help them complete their
research. One of the main benefits of this approach is that it adds structure to evaluation while
preserving a holistic concept of student progress. Using this table to think about a learning objective is a
good way for the instructor to define specific goals and objectives for a class as well as identify specific
tasks and assignments which will help fulfill those objectives.
4. Methods for designing instructional goals and objectives
As stated in the opening paragraph, ML can both be thought of as a standalone element of an IL
curriculum or as an integrated part of another curriculum. There is a discussion in the literature which
questions how much librarians and non-librarian instructors need to focus on teaching ICT skills and
concepts. Mabrito & Medley for example assert that student skills are simply different by virtue of their
familiarity with information technology. In contrast, Rowlands et. al. observe that students tend to have
very specific skills but do not have the tools to help them generalize them. Further, information
Page 6 of 7
technology requires specialized skills that may not be commonly held by all students. The implication of
this discussion is that instructors must be prepared to accommodate a diversity of student abilities and
attitudes with regards to literacy (particular information technology literacy) instruction.
There has been a trend in academic librarianship in recent years to reposition the role that libraries
serve from information experts to facilitators of teaching and collaboration. For example, the idea of
embedded librarianship focuses on using librarians in specific contexts to aid specific educational goals.
In these cases librarians may serve research support needs, technology support needs, of content
specialist needs. Another example of this mission shift in libraries is the idea of participatory
librarianship (Lankes et. al.) in which libraries serve as a facilitator for collaboration and conversations.
Lankes observes that as the mission of libraries transition away from being warehouses of information
that they need to redefine their mission in supporting and capturing institutional efforts and
collaborations. These two concepts underscore approaches to curriculum integration which has been
more popular in the k-12 environment than it has been in higher education to date.
In both of these cases (embedded librarianship and participatory libraries), it often falls to librarians to
decide when and how to incorporate literacy of any sort into an instructional environment. The
information literacy framework that was developed in the IL literature review provides a way to think
about the applicability of a given literacy in a specific teaching setting. The framework is replicated in
table 2 with the ML element of tagging used as an example
Table 2 IL framework for teaching tagging
Pedagogical role
Information /
Learning role
Environmental
role
Skill
Tagging gives the
student an active
task that can be
used in
conjunction with
other students
Students learn
how to interact
with tagging as a
new form of
information
technology
Tagging occurs in
an electronic
environment
(blog, citation
tracking, etc)
Concept
Tagging serves as
a way form of
collaboration
Tagging helps the
student engage
with and
contextualize
content
Tagging occurs in a
‘type’ of system
that can be used
to help students
understand the
role of different
technology
environments
Context of
Is tagging a goal by
Understanding
What is the goal of
Page 7 of 7
skill/concept
itself or is it part of tagging allows
another objective? students to use it
in other systems
tagging within this
environment?
This framework allows the librarian/instructor to ground a specific literacy (in the above table it is
tagging) in a teaching environment (supported by a learning theory, pedagogical approach, and learning
environment). As seen in the values entered in the table, by breaking the process of teaching tagging
down into its respective parts, the instructor can ensure that the literacy being taught fits with the
learning goals and environment in which the students are engaged. Using this framework allows the
instructor identify when metadata literacy skills or concepts are called for either due to the use of a
particular environment or the use of a particular learning objective. In some cases, metadata serves a
supporting or scaffolding role but not a primary learning objective. In these cases it is necessary to keep
the instruction focused on the primary objective and only introduce metadata literacy as needed.
By using this approach in conjunction with the evaluation approaches discussed in section 3, a librarian
or instructor can decide whether or not a literacy element fits with a given instructional goal. Once they
have decided how the literacy element works within a curriculum they can use Bloom’s revised
taxonomy to create specific objectives around which they can center their curriculum.
Conclusion
This response has examined how to design instruction and evaluation for literacy instruction with a
specific eye on metadata literacy. It discussed how librarians are called upon to offer literacy instruction
both as a standalone course and as a supplement to another curriculum. It outlined an approach for
identifying whether or not instruction of a literacy element is called for by using the information literacy
framework designed in the IL literature review. It discussed two key challenges with literacy evaluation;
a) student assessment of skill and; b) holistic evaluation of level of literacy. It suggested two forms of
evaluation that address these issues; a) self-efficacy evaluation and b) definition of an evaluation rubric
using Bloom’s updated taxonomy and discussed how to use Bloom’s taxonomy to define course
objectives and evaluative criteria. While there are any number of environments in which it is
appropriate to teach literacy, this response suggests metadata literacy is applicable in specific instances
where either the instructional environment or the learning objectives include metadata literacy
elements.
Good answer. Good coverage. There are other instructional methods and contexts (as Erik
indicates), so it’s impossible to describe all; but he does a good job picking some and giving
examples, and describing in general an approach. Only follow-up question might be to ask about
other instructional methods (like group work), and other evaluation metrics.