Workshop on
Quality/Selectivity of the DLESE
Collections
• Framing the Question
• History of the Discussion
Kim Kastens, June 30, 2003
Framing the Question
• DLESE has Broad Collection and a
Reviewed Collection. We are (mostly)
talking about the Broad Collection.
Framing the Question
• Resources enter the DLESE Broad Collection via two
routes:
– Individually, via the DLESE Cataloging Tool (the “Community
Collection”)
– As part of an aggregated or themed collection, a collection
accessioned into DLESE in its entirety.
• We are concerned with quality and relevance of
resources entering via both routes.
Framing the Question
• Anyone can submit a resource to DLESE via
the cataloging tool, which is an open set of
web forms.
• This has given rise to concerns that “junk”
could get into the DLESE Collections.
Framing the Question
• This workshop needs to make
recommendations on two issues:
– What should be the criteria by which resources are
approved for inclusion in the DLESE Broad
Collection?
– What should be the procedures by which these
criteria are implemented?
Framing the Question: Criteria
• Resources submitted for the DLESE Broad
collection currently must meet two criteria:
– The resource is relevant to Earth System
Education
– The resource works (i.e. it has no conspicuous
bugs).
Framing the Question: Criteria
• Other possible criteria that have been suggested for
the DLESE Broad Collection:
• No cost or low cost for educational users
• Resource is in English
• No commercial message
• No intrusive advertising
• No blatant religious message
• No blatant political message
• No blatant errors of fact
• Educational effectiveness
• Well documented
Framing the Question: Procedures
• With respect to procedures, we have two issues:
– By what process shall we identify problematic resources?
– What shall we do when we find a problematic resource?
Framing the Question: Procedures
• By what process shall we identify problematic
resources?
– Ask the resource contributor (current system)?
– Screening by the community?
– Screening by paid staff?
Framing the Question: Procedures
• What shall we do when we find a problematic
resource?
– Exclude it from the collection?
– Include it in the Broad Collection but annotate
it?
This is what an annotation might look like in the
Discovery System:
Framing the Question: Procedures
Screening by
paid staff
Proble matic
resources
exc luded from
li brary
Proble matic
resources
anno tated in
Discovery
System
Screening by
comm unit y
1
2
3
4
History of the Discussion
• Coolfont: August 1999
– Collections Policy drafted
– There shall be a Reviewed Collection and an
“Unreviewed” Collection
– Collection Committee established
History of the Discussion
• Coolfont: August 1999 (cont’d)
– Rationale for Reviewed Collection:
• Users’ Perspective:
“…. recognized, efficient source for quality
teaching and learning materials.”
• Creators’ Perspective:
“…. a
recognized stamp of professional
approval at the level of publication in a peer-reviewed journal.”
– Rationale for the “Unreviewed” Collection:
• “Users are seeking materials on a huge range of topics. The DL
provides added value by being inclusive while providing powerful
search and classification capability.”
History of the Discussion
• Coolfont: August 1999 (cont’d)
– Criteria for Reviewed Collection:
•
•
•
•
•
•
•
Accuracy, as evaluated by scientists
Importance/significance
Pedagogical effectiveness.
Well documented.
Ease of use for students and faculty
Inspirational or motivational for students
Robustness/sustainability
History of the Discussion
• Coolfont: August 1999 (cont’d)
– No Criteria established for “Unreviewed” Collection
– After debate, it was decided that there would be a
human-mediated step between submission of
resource and ingestion into library.
History of the Discussion
• Spring 2000: Academic Career Recognition
Task Force Web Survey
– Seven selection criteria for the Reviewed
Collection met approval of prospective
DLESE users, resource creators, and
department Chairs.
History of the Discussion
• Mid-late 2000: Collecting began
– DPC: testbed collection for exercising metatdata
framework
– Montana State: Dave Mogk & students
– Foothill College: Chris DiLeonardo & students
History of the Discussion
• October 2000: Collections Meeting at Boulder:
– DLESE Community Cataloger tool introduced
to non-DPC collecting groups (AGI,
Montana State, others?)
History of the Discussion
• November 2000: Steering Committee Meeting
at Lamont:
– Contentious discussion about “filters” at the gateway to the Broad
Collection
– Agreement on only two of the discussed “filters”:
(1) relevant to Earth System Education
(2) “It works”, e.g. no conspicuous bugs
- Contentious discussion of how to apply “filters”; clarity seemed to
emerge when John Snow described a “holding tank” system used in his
history group.
History of the Discussion
• Nov-Dec 2000: Steering Committee Meeting
at Lamont (cont’d):
- Meeting Minutes:
• “The
general concept of a 30-day public comment period on new resources
was agreed to. This will allow a time for the community to review
resources….”
• “In the short term, partners collecting resources …. will review them to
make sure they are appropriate
• “….the Collections Committee, collection proposal team and the DPC will
work together to investigate mechanisms for encouraging review….”
History of the Discussion
• February 2001 Collections Meeting:
– Joint meeting of Collections Committee, “Collections Partners”, and
Community Review System Editorial Review Board
– DLESE Community Cataloging Tool open to the world
– Collections Committee drafted Deaccession Policy
History of the Discussion
• February 2001 Collections Meeting (cont’d):
– Collections Committee discussed “filters” at gateway to Broad
Collection. Imperfect consensus:
• Relevance Filter
– Is the resource relevant to Earth System Science education?
• Integrity Filter
– Are there no blatant errors of fact in the resource?
– Are there no blatant political, religious, or commercial messages in
the resource?
– Does it function reasonably; i.e., seem to be basically bug-free?
History of the Discussion
• April 2001 Steering Committee meeting at Biosphere 2:
– Collections Committee/DPC Collections group presented
fleshed out version of the “holding tank” or “provisional status”
plan.
– Many questions and issues. Who are reviewers? How
mobilized and overseen? No $ to oversee the “army of
filterers.”
– No resolution.
History of the Discussion
• July-Aug 2001 Steering Committee meeting at Flagstaff:
– 850 resources in library. Metadata QA streamlined.
–
“Mike Mayhew indicated a concern …about the broad collection. ….Where is
the quality control in developing the collection? Do we dilute the value of library
with variable quality?”
– Holding tank idea revisited, in simpler form without “designated reviewers”
– Action item: “Boyd …. will develop a draft proposal/set of guidines to
implement a holding tank in which resources are discoverable in the system and
identified as accessioned within a 30-day period with some mechanism to
accept comments. The proposal for implementation will not include a designated
reviewer”
History of the Discussion
• February 2002 Steering Committee meeting at Boulder:
– Draft Collections Accession Policy presented
– Revised throughout spring
– DLESE oversight would be review of review process, rather
than review of individual resources
History of the Discussion
• July 2002 Steering Committee & Annual meeting at
Cornell:
–
–
–
–
Deaccession Policy approved
Interim Collection Accession Policy approved
First annotation service demo’d within DLESE
Faulker reported that NSDL content philosophy was:
“Educational value …to be manifest in capabilities for
annotation and selective filtering, rather than an accession
threshhold”
– Possibility raised that annotation option might be solution to
ongoing dilemma about quality of DLESE Broad Collection.
History of the Discussion
• Fall 2002:
– Sumner et al focus group study of Educators’
perceptions of Quality.
– Best Practices for Resources summited to the DLESE
Reviewed Collection begins to take shape.
History of the Discussion
• Spring 2003:
– Ad hoc Collections group met in Boulder, worked on
how to implement Interim Collections Accession
Policy and on Pathways to Reviewed Collection
document
– June 13: 12 collections met documentation
requirements to be accessioned as collections.