ASM Representative: Alix Darden, alix.darden@citadel.edu
FIRST Undergraduate Assessment
Database: The Role of ASM
http://first.ecoinformatics.org
We Need Your Help
Your Research Question
Metadata - data about data
What we need ASM members to do for us:
• If we build it, will you use it?
• Be willing to test-drive the user interface and provide feedback.
• Help seed the database with assessments and data from your course(s)
• Give Alix your name, email and institution
Did teaching students to create, interpret and critique models result in better student
understanding of evolution?
To promote STEM education research, you tag assessment items with metadata, such
as Discipline-specific concepts. In biology, for example, we plan to use the National
Biological Information Infrastructure (NBII) Biocomplexity Thesaurus (Figure 5).
Additional metadata tags will include standard psychometrics such as difficulty and
discrimination, Bloom’s Taxonomy of Educational Objectives, Professional society tags
and whether an assessment item is copyrighted (as in a textbook question or a
published concept inventory).
What is the FIRST Database?
Faculty Institutes for Reforming Science Teaching (FIRST) has engaged faculty from
over 50 institutions in professional development focused on active, inquiry-based
teaching designed to improve student learning.
Metadata collected by the FIRST database will also encompass information about
courses and institutions (Figure 6). These data will facilitate longitudinal studies in
addition to comparisons among courses across institutions.
We are constructing a database to support research on undergraduate STEM
education. The database will support storing, searching and analyzing assessment
data from undergraduate STEM courses. This database will facilitate both data-driven
instructional decision making (Figure 1) and research in science education.
Content-driven instructional design
Discipline-based
knowledge (textbooks)
Design
course
Teach
course
Assess student
outcomes
Data-driven instructional design
Discipline-based
knowledge (textbooks)
Design
course
Teach
course
Analysis of student
data drives course
modification
Figure 1. Two models of instructional design.
Formative
feedback
Assess student
outcomes
Figure 3. Data sources from a faculty member, including: (a) output from course
management software, (b) excel spreadsheets of student exam, homework and inclass assignments, (c) clicker data, (d) student open-ended responses and (e) exams.
Your Data Sources
Analyze
student
learning
outcomes
You upload to the FIRST Database all assessments and related data from your
introductory biology course (Figure 3), including assessment questions and grading
rubrics. You tag each assessment item with both a concept category (see Figure 5)
and an instructional strategy (i.e., modeling, pair-share, JiTT, etc). You have the option
to tag assessment in several additional ways (see the discussion of Metadata, next
column), or you may bypass this task at this time.
Database Functionality
We are using a large variety of assessment data collected at Michigan State University
and from colleges and universities across the nation to design the FIRST Assessment
Database. Faculty from all STEM disciplines will input and retrieve data from the
database to explore questions about effective teaching and learning in undergraduate
education (Figure 2).
Microbiology
Bacteriology
Clinical microbiology
Mycology
Virology
Food processing
Infectious diseases
Microbial activity
Microbial contamination
Microbiological analysis
Microbiological culture
Microbiologists
Microorganisms
Parasitiology
Pharmacology
Activity
Antibacterial activity
Antifungal activity
Antiprotozoal activity
Antiviral activity
Microbiocidal activity
Antimicrobial agents
Figure 5: Example of concepts retrieved from the NBII Biocomplexity Thesaurus
(http://thesaurus.nbii.gov).
Course-level Metadata
Institution type/size
Course format (lab, lecture, discussion, etc)
Course size
Targeted students (majors, non-majors, lower or
upper level)
Course Syllabus
Assessment-level Metadata
Type of assessment (e.g., in-class, open book, exam)
Proportion of final grade
Bloom’s level of understanding
Concept category
This project will facilitate cross-institutional studies using assessment data from large
numbers of students and classes. The database is the bridge between teaching and
research that enables faculty to become both expert users of and contributors to the
scholarship of scientific teaching.
Figure 6: Examples of course- and assessment-level metadata captured by the FIRST
Assessment Database.
Security, Intellectual Property, IRB
Figure 4. Potential data output from the FIRST database.
Data Analysis
To analyze whether teaching students to make models improved their learning of
evolution, you query your current course and two past courses for data on evolution
and modeling. The FIRST database returns a single spreadsheet file with the data
from each course (Figure 4). You import the data into your favorite statistical analysis
software.
As a correlated question, your are interested in comparing your students’ modeling
abilities with those of students from other institutions. Your again search the FIRST
database for courses at other colleges or universities using model-based education to
teach evolution. You limit your query to include only large introductory biology courses
at institutions similar in size to yours. You then limit the returned data to include only
assessments related to modeling in evolution. The FIRST database again returns a
spreadsheet that you can import into statistical software for analysis.
Based on your analysis, model-based learning led to significant gains in student
learning of evolution in your course and courses at other universities. These results
prompt you to redesign additional portions of your course to include modeling.
Figure 2. Overview of the FIRST Assessment Database.
Institutional review boards (IRB) are unique to each college and university. This reality
demands a flexible database that is both secure and protective of student data. The
FIRST Assessment Database will support two levels of data access - restricted
(available only to the faculty member who uploaded the data) and public (available to
all registered database users).
Many assessment items are copyrighted - as in textbook questions or published
concept inventories. Faculty members using the database must be aware of and take
responsibility for obtaining appropriate permissions for copyrighted or otherwise
protected material. Without such permissions, an assessment item cannot be part of
the public database. For more information on copyright issues and intellectual property
rights, please see our website.
Student responses to the FIRST Assessment Database are de-identified by a hashing
function that ensures student anonymity. To promote longitudinal studies, however, the
database will link students across courses at an institution.
A faculty member may chose or be directed by their institution’s IRB to remove
identifying information about themselves and the institution.
For more information, please contact Diane Ebert-May (ebertmay@msu.edu) or Mark
Urban-Lurain (urban@msu.edu).
This material is based upon work supported by the National Science Foundation (NSF) under award 0618501. Any opinions, findings and
conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.