Evaluation & Assessment
in the Undergraduate Curriculum
Assessment as Collaboration
Kris Stewart
NPACI Education Center on
Computational Science and Engineering
San Diego State University
http://www.edcenter.sdsu.edu/
Professor, Computer Science
http://www.stewart.cs.sdsu.edu/
ASCP Workshop 04June04 LA076 SDSU
What is Computational Science?
Science Discipline
Computational Science
Teamwork and Collaboration
Physics, Chemistry, Biology, etc.
Computer Science
Hardware/Software
ASCP Workshop 04June04 LA076 SDSU
Applied Mathematics
Numerical Analysis, Modeling,
Simulation
What is Computational Science?
Recognized as separate discipline
E.g. Society for Industrial and Applied
Mathematics
http://www.siam.org/cse/
and Osman Yassar,
IEEE Computing in Science & Engineering
http://www.cps.brockport.edu
Should CSE be a separate major?
(Computational Science [no, too general] or
Computational Physics [yes, natural extension])
Should it be a separate department?
(How to support interdisciplinarity?)
ASCP Workshop 04June04 LA076 SDSU
Kris’ Faculty Background
(Kris Stewart, Director, San Diego State University, California State University System)
• Numerical Analyst* led to
• Supercomputing and Undergraduate Education
(SUE**) led to
• Supercomputing Teacher Enhancement Program
(STEP***) led to
• Education Center on Computational Science &
Engineering (EC/CSE)
*MS/CS SDSU 1979, JPL 1981, PhD UNM 1987, SDSU 1986
** UCES (DoEnergy 1994)
*** Smithsonian Research Collection (1996)
ASCP Workshop 04June04 LA076 SDSU
UCES Paradigm
(thanks Tom Marchioro and the “crew”, 1994?)
Previous exposure to “assessment”
ASCP Workshop 04June04 LA076 SDSU
1998/99 Assessment by LEAD
Assessment as a Collaboration
Background
• Workshop in Wisconsin April 1997 to learn
about assessment and make it real
• NPACI starts 1 October97
• EC/CSE requested assessment for 1998
Preparation for SDSU Campus Visit by Baine & Julie
• Discussions at SC98- SuperComputing Fall98 Orlando
• Email to SDSU faculty gather attitudes
ASCP Workshop 04June04 LA076 SDSU
Grand Challenges for HPC
(Stewart & Zaslavsky, SC98)
1.
Faculty system of rewards does not encourage teaching
innovations
2. Lack of awareness of HPC technologies already used in
research or teaching for different fields
3. Faculty & students unaware of benefits and accomplishments
of HPC
4. HPC technologies considered too complex/inaccessible for
undergraduate instruction
5. Sequential HPC-related curricula is absent
6. Curricula using very large data sets not widely available
7. Adjust to different learning styles when material is complex
8. Variety of platforms/software leads to fragmented curricula
9. School administration/support staff not ready for HPC
10. Specs of computers and networks below user expectations
We had been thinking about this (based on April 97 LEAD Workshop in WI)
ASCP Workshop 04June04 LA076 SDSU
Building the Community of Faculty
• These challenges are people-centric, not
technology-centric and of interest to the
community
• Systemic Change requires understanding the
system and working within it
• Empower teachers (find the time), recognition
(from chair/dean), support (student assistants)
ASCP Workshop 04June04 LA076 SDSU
Interviews on SDSU Campus
LEAD applying info from email surveys (Spr 99)
Faculty skepticism:
• Convincing evidence that
computer-based tools
enhance teaching process?
• Knowledge of modern
computational methods
and availability?
• Incentive from department
and insufficient tech
support?
LEAD Interviews with V.P.,
Deans, Chairs
Faculty Fellows program
identified as a target
OUTCOMES:
• our local infrastructure at
SDSU took us more
seriously
• survey instruments refined
• SWB value recognition
ASCP Workshop 04June04 LA076 SDSU
Assessment not just requirement
Rather, found to be
• vital tool to assist in clarifying student and faculty
needs
• improve prioritization skills
• validation of focus on human factors to integrate
HPC (modeling & visualization) into undergrad
curriculum
ASCP Workshop 04June04 LA076 SDSU
California Education Infrastructure
• K12 Education (standards based, performance
based)
• Community Colleges (Freshman/Sophomore)
– Vocational
– University preparation
• California State University System (23 campuses)
• University of California (9 campuses, Merced in
2004)
• Independents
(Stanford U., CalTech, U. Southern California)
ASCP Workshop 04June04 LA076 SDSU
California Education Infrastructure
(testbed for change)
IMPAC http://www.cal-impac.org/
Intersegmental Major Preparation Articulated
Curriculum Project
Community Colleges and Four Year Universities
Charter Schools (Preuss School) preuss.ucsd.edu
Dr. Rozeanne Steckler and Dr. Mike Bailey
“Fostering Scientific Curiosity in All Children”
San Diego Supercomputer Center
ASCP Workshop 04June04 LA076 SDSU
Involving University Faculty
Infrastructure for Change
• NPACI/SDSU Faculty Fellows
Local Support from College Deans and
Department Chairs (participation buy-in and
faculty recognition)
• SDSU Academic Advisors (across disciplines)
• Professional Meeting (SC2001, SIAM, ACM,
SIGCSE, your suggestions?)
ASCP Workshop 04June04 LA076 SDSU
Grand Challenges for HPC
(Stewart & Zaslavsky, SC98)
1.
Faculty system of rewards does not encourage teaching
innovations
2. Lack of awareness of HPC technologies already used in
research or teaching for different fields
3. Faculty & students unaware of benefits and accomplishments
of HPC
4. HPC technologies considered too complex/inaccessible for
undergraduate instruction
5. Sequential HPC-related curricula is absent
6. Curricula using very large data sets not widely available
7. Adjust to different learning styles when material is complex
8. Variety of platforms/software leads to fragmented curricula
9. School administration/support staff not ready for HPC
10. Specs of computers and networks below user expectations
ASCP Workshop 04June04 LA076 SDSU
Building the Community of Faculty
• These challenges are people-centric, not
technology-centric and of interest to the EPSCOR
community
• Systemic Change requires understanding the
system and working within it
• Empower teachers (find the time), recognition
(from chair/dean), support (student assistants)
ASCP Workshop 04June04 LA076 SDSU
Faculty Fellows Fall 01 Synergy
among themselves and with their chairs and deans
People, Time, Support,
Recognition …
ASCP Workshop 04June04 LA076 SDSU
NSF/EHR
National Science
Foundation/Education and
Human Resources
Directorate
www.ehr.nsf.gov/EHR/
RED/EVAL/handbook/
handbook.htm
LEAD
Assessment and
Evaluation
1998 Formative for the
EC/CSE
http://www.cae.wisc.edu/
~lead/pages/produts/eotpaci.pdf
ASCP Workshop 04June04 LA076 SDSU
EC/CSE involve CSU
Faculty Workshops
• Introduce Computational
Science
• Host Biology Workbench
(BioQuest)
Support at NPACI AHM02
Extend Faculty Fellows
ASCP Workshop 04June04 LA076 SDSU
Contact Us
Kris Stewart, Director
http://www.edcenter.sdsu.edu
stewart@sdsu.edu
Jeff Sale
jsale@edcenter.sdsu.edu
Kirsten Barber
barber@edcenter.sdsu.edu
ASCP Workshop 04June04 LA076 SDSU
The Ed Center Project and its
Evaluation
• Goal: incorporate computational science techniques in
undergraduate curricula
• Differences from STEP: target audience, institutional
arrangements
• Methods and Strategies:
• Faculty Fellows
• Workshops, presentations, in-house projects, trying out new
approaches in own teaching, tools development
• Computational Science Olympics (curriculum from the bottomup!!)
• Surveys, continuous assessment of faculty involvement and
learning outcomes
• Evaluated by the LEAD Center in 1998-99
ASCP Workshop 04June04 LA076 SDSU
Introducing the EC/CSE
The mission of the Ed Center on Computational Science
and Engineering? www.edcenter.sdsu.edu
Who are the people involved? www.edcenter.sdsu.edu/staff
Some of our projects: www.edcenter.sdsu.edu/projects/
Some of our activities: www.edcenter.sdsu.edu/news/
Some resources:
www.edcenter.sdsu.edu/repository
ASCP Workshop 04June04 LA076 SDSU
Undergraduate Faculty: A Tough
Target Group
• Obstacles: lack of time, tenure and review
considerations, lack of awareness about available
technologies
• Undergraduate faculty:
– ¾ have used WWW often or sometimes (1997), but not in the
classroom (only 18% - 1998)
– The gap between those NEVER using computers in the classroom,
and those using them OFTEN, is the largest for untenured faculty,
increasing towards tenure review
– Only 12% of surveyed faculty saw themselves as having a use for
HPC applications in courses (higher for Sciences and Engineering)
– 11% of faculty have students working with computer models
OFTEN
ASCP Workshop 04June04 LA076 SDSU
Using computers in the classroom versus number of years as a faculty member
(1997 Faculty Survey)
ASCP Workshop 04June04 LA076 SDSU
Students Using Computers in the Classroom (1997 Faculty Survey)
1. 1 OFTEN
2. 2 SOMETIMES
3. 3 RARELY
4. 4 NEVER
5. 9 DK/REF (Missing values)
Primarily graduate faculty
Primarily undergraduate faculty
College of Arts & Letters
College of Sciences
ASCP Workshop 04June04 LA076 SDSU
Strategies for Building Faculty
Community
• Reliance on most enthusiastic and technically advanced instructors who are
already using computing and modeling in classes
• The Faculty Fellows program:
• Stakeholders:
–
–
College Deans - Specific support
Faculty - Compensation, and acknowledgement, of the value of the faculty members contribution
• Benefits
–
–
–
–
College
Department (Faculty Fellows as discipline-specific spokespersons for EC/CSE)
Faculty (as individuals)
Ed Center on Computational Science and Engineering
– Building a special infrastructure for curriculum transformation: human, institutional,
technical – is a requirement for successful introduction of advanced techniques (since they
are more demanding on faculty time and efforts)
The problems and strategies are not that much different from STEP!!
ASCP Workshop 04June04 LA076 SDSU
Faculty Fellows during 1998-2000
• Faculty Fellows representing four departments from four
colleges: Geological Sciences, Geography, Computer
Engineering, Business Information Systems
• Bi-weekly meetings at the Ed Center
• Faculty Fellows as “ambassadors” of computational
science
• Partnership with LEAD for evaluation during 1998-99
• Follow-on Activities (Susan Millar, LEAD)
– CATS (Classroom Assessment Techniques)
– FLAG (Field-tested Learning Assessment Guide)
– SALG (Student Assessment of Learning Gains)
ASCP Workshop 04June04 LA076 SDSU
More Strategies
• Trying the new approaches in our own teaching:
– Teaching the Supercomputer class with group-based
problem-solving environments
– Real-time distance teaching with Web-based
collaborative software (featured as Microsoft Case
Study in Higher Ed.)
– Development of Sociology Workbench, a free on-line
survey data analysis system that can be used for
evaluation of student outcomes and other surveys
ASCP Workshop 04June04 LA076 SDSU
Channels for Influencing Pre-Service
Teacher Preparation
• Use of advanced computing modules in general ed courses
(e.g. Geol 303 “Natural Hazards”)
• Cooperation with College of Education faculty and
students, esp. in Education Technology: focus on
experimentation with new technologies in classroom
setting
• Computational Science Olympics: supporting the “bottomup” development of computational science curricula
• Providing on-line assessment technologies:
– Sociology Workbench: http://edcenter.sdsu.edu
ASCP Workshop 04June04 LA076 SDSU
SWB Convenient Tool to Learn from
Student Survey Data
• Online tool for “standard public data sets” or your
own data set http://edcenter.sdsu.edu
• Small Sample, therefore only useful as feedback
for the instructor
• Can be used with “forms” interface directly into
SWB format, as in June ‘99 CSU Faculty
Workshop
ASCP Workshop 04June04 LA076 SDSU
SWB as Analysis Tool
View Student Comments (text)
•
ASCP Workshop 04June04 LA076 SDSU
SWB as Analysis Tool
Isolate on Specific Survey Response
•
ASCP Workshop 04June04 LA076 SDSU
SWB as Analysis Tool
Explain the Response on Learning with “doing more”
•
ASCP Workshop 04June04 LA076 SDSU
SWB as Analysis Tool
Explain learning with “active participation”
ASCP Workshop 04June04 LA076 SDSU
Lessons Learned
•
•
•
Institutional support required for program to
be sustainable
Individual reform-ready faculty is focus for
support
Infrastructure:
– Build a Synergistic Environment (across disciplines)
for Faculty
– Continuous monitoring through interviews, surveys,
discussions
ASCP Workshop 04June04 LA076 SDSU
What’s Next
• The approaches we described proved useful for
two target audiences. We believe that the strategies
and lessons learned can be extrapolated in a
targeted effort to incorporate computational
science technologies in pre-service teacher
preparation
• This may be a funding opportunity?
• We will be happy to contribute our knowledge and
share experiences…
ASCP Workshop 04June04 LA076 SDSU
References ASCP Workshop
•User-Friendly Handbook for Project Evaluation: Science, Mathematics,
Engineering and Technology Education, NSF 93-152
www.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm
•Sid Karin: The Importance of Science Literacy in a Computing World (see
enVision Science Magazine, V.15 No. 2
•K. Stewart & I. Zaslavsky, “Ten Grand Challenges”, Supercomputing ’98
Orlando FL SC98 Conference Paper
•J. Foertsch & B. Alexander, “Integrating HPC into the Undergraduate
Curriculum”, report by LEAD Center June 1999
http://www.cae.wisc.edu/~lead/pages/products/eot-paci.pdf
ASCP Workshop 04June04 LA076 SDSU
Evaluation & Assessment
• U. Wisconsin LEAD for 1998/99 Ed Center
evaluation
• www.cae.wisc.edu/~lead/pages/products/eot-paci.pdf
• Follow-on Activities (Susan Millar, LEAD)
CATS (Classroom Assessment Techniques)
FLAG (Field-tested Learning Assessment Guide)
SALG (Student Assessment Learning Guide)
ASCP Workshop 04June04 LA076 SDSU
Evaluation and Assessment of
Classroom Practice
Where to start?
• User-Friendly Handbook for Project Evaluation: Science,
Mathematics, Engineering and Technology Education,
NSF 93-152
www.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm
• Student Surveys - Need a compatible tool for instructor to
examine results with
• Sociology WorkBench (SWB) developed by team of
undergraduate computer science majors employed by the
EC/CSE
ASCP Workshop 04June04 LA076 SDSU
Information Overload
• How can teachers help out?
• How can technology providers help out?
• Ans: personalize the linkage.
ASCP Workshop 04June04 LA076 SDSU
Information Overlab Lab
• Using the WWW extensively in the classroom,
implies obligation to discuss Information
Overload
• http://www.stewart.cs.sdsu.edu/infolab/index.html
Timeline of Technology
http://www.stewart.cs.sdsu.edu/infolab/timeline_tech.html
ASCP Workshop 04June04 LA076 SDSU
Computer Network Security
• Campus-wide “buy-in” recognizing
the use of high-speed networks
the responsibility for “secure network use”
• SDSU Academic Senate adopts interim policy
• UCSD Acceptable Use Policy useful start
ASCP Workshop 04June04 LA076 SDSU
A Little History - May 10, 1869
• What was the major event?
• Http://memory.loc.gov
• Thank you to the US Library of Congress for the
on-line exhibit preserving our “Memory”
ASCP Workshop 04June04 LA076 SDSU
ASCP Workshop 04June04 LA076 SDSU
Promentory Point, UTAH
•
ASCP Workshop 04June04 LA076 SDSU
How does this relate to us today?
ASCP Workshop 04June04 LA076 SDSU
Technologies to Motivate Learning
• Mr. Sid uses Wavelets for compression of Railroad
Maps
• Sense of History must be tied to personal, studentbased, “stories”
• University Faculty need encouragement this will
“count” for professional recognition by their peers
and administrators (Deans…)
ASCP Workshop 04June04 LA076 SDSU
ONLINE Resources for Undergrads
• The Education, Outreach and Training (EOT) of the PACIs
http://www.eot.org
• feature several projects
•
Ed Center on CSE
•
Shodor
Undergraduate Computational Physics www.orst.edu
•
•
•
Minority Institutions, a Directory
Student Biology Workbench (Bioinformatics)
Underrepresentation in Computer Science (CDC)
ASCP Workshop 04June04 LA076 SDSU
Learning Technologies from the
PACIs
•
•
•
•
•
•
•
•
Data Intensive Computing Environment
Digital Video/eTEACH
Interaction Environments: Bays to Brains
Neuroscience: Telescience/Tomography
NISE College Level One Institute on
Learning Technology
Remote Collaborative Learning
www.eot.org
ASCP Workshop 04June04 LA076 SDSU
This is Only the Beginning...
YOU ARE HERE
TIME
ASCP Workshop 04June04 LA076 SDSU
What’s Next
• Your comments and thoughts?
• SIAM Computational Science & Engineering
Conference – Sept. 2000 – Washington D.C.
• SC2000 – November 2000 – Dallas TX
ASCP Workshop 04June04 LA076 SDSU
ASCP Workshop 04June04 LA076 SDSU
NPACI Sources of Information
• NPACI Partnership Report
• enVision quarterly science
magazine, especially June2000
“Why the Future May Very Well Need Us”
• www.npaci.edu/enVision/director.html
• “Online” biweekly electronic
publication, www.npaci.edu/online/
• www.npaci.edu
ASCP Workshop 04June04 LA076 SDSU
References - EPSCoR - July 2000
• This Presentation
• http://www.stewart.cs.sdsu.edu/PPT/epscor_20july00.pp
t
• Ed Center on Computational Science & Engineering
• http://www.edcenter.sdsu.edu/
• Timeline of Technology Development
• http://www.stewart.cs.sdsu.edu/infolab/timeline_tech.ht
ml
• Education, Outreach and Training
• http://www.eot.org
• Information Overload
• http://www.stewart.cs.sdsu.edu/infolab/index.html
ASCP Workshop 04June04 LA076 SDSU