SAIR 2005 Conference
Charleston, SC
October 24, 2005
Joseph Hoey
Jon Gordon
Office of Assessment, Georgia Tech

Development of OATS to document
Institutional Effectiveness
Description of OATS features
OATS structure and elements
Issues encountered, solutions generated
Path forward
Questions and discussion

Web-based documentation of assessment for demonstrating progress towards Institutional
Effectiveness is an option, not a
requirement – we have found it very useful at Georgia Tech.
Georgia Tech’s approach to documenting systematic assessment processes is our
Online Assessment Tracking System (OATS).
Similar systems have been developed by other institutions and commercial vendors.

Annual Assessment Update concept was generated by GT unit coordinators in 1998 to document Georgia
Tech’s responsiveness to SACS recommendations re: assessment of student learning.
Requests received to move to online environment
Online process provides structure, formalizes best practices in assessment, serves as a cross-campus communication vehicle, and thus facilitates demonstration of compliance.
Having an online documentation system proved to be a key feature in Georgia Tech’s compliance review with the Principles of Accreditation in 2004.

We document and track assessment at the degree program level
For each degree program, we document:

Faculty expectations for student learning




How these expectations are defined in practice
How these expectations are assessed
What results are gained (summary level)
Based on results gained, what actions are being taken


What did you look at?


What did you find?
What did you do about it?

Paper-Based Method

Many different formats




Hard copy only
Difficult to track progress over time
Flexibility (but no consistency across Institute)
Difficult to provide feedback internally and to facilitate institutional sharing of good practices
Online Method

Consistent format

Database storage




Ability to track progress over time
Flexibility maintained
Process facilitates accreditation e-review
Easier to provide feedback; facilitates institutional sharing

Annual Assessment Update now done online via OATS.
Becomes a longitudinal record of assessment and changes to each instructional program.
Ideally, can be used to satisfy multiple internal and external information and accreditation needs.

Includes user id/password login and Georgia Tech authentication
Web accessible from any location; software plug-in required for editing capability
Defined format structure—Objectives/Outcomes,
Methods, Results, and Actions/Impact
Allows posting of formatted text (tables, charts, etc.)
Default annual rollover feature (except for assessment results and actions taken) saves time
Assessment Updates due: November 1 each year

Archiving of all past submissions by degree program,
Print-preview feature that produces a laddered report of objectives, methods, results, and actions taken,
Notes feature that permits the storage of submission notes and commentary that won’t be “seen” officially; could be notes to colleagues
Feedback feature is built in that permits units to obtain formative feedback on their update prior to final submission.
College and Institute-level views, whereby deans are able at a click to view the online assessment update submissions of all units within their colleges.

Main Menu
Program Menu
Unit Summary Section
Objectives/Outcomes
Methods
Results
Actions
Print Preview

Includes links to instructions and changes in
OATS
Link to administrative page for those with administrative access
Selections for:

 editing current updates, viewing current updates (print preview format only), and
 viewing updates from previous years (print preview format for 2003; .pdf for earlier years)

Main Menu: Current Year and History

Searchable by College
Includes enumeration of objectives/outcomes, methods, results, actions/impacts and date last modified
Indicates status/progress towards completion
Options to combine programs, view notes, edit individual update, and delete update (“Are you sure?” screen appears)
Notes feature can be used as information storage area by user (“here’s why we did this last year…”) or for feedback from Office of Assessment

College Level: Ivan Allen College

Summary section of each OATS update includes portions for program purpose, responsibility for implementation, and operational (programmatic) objectives
Programs use this section to describe the program purpose, set the context, describe operational objectives for the program, and to provide a brief summary of overall direction.

We ask programs to limit OATS updates to 3-8 most important student learning objectives/outcomes faculty expect from program graduates.

Engineering undergraduate programs: prioritize and develop a rotational timeline for assessing ABET EC 2000 program outcomes – don’t try to assess all of them each year.

For each objective/outcome, we ask programs to operationally define the activities and competencies to be assessed.
What will students actually demonstrate?
Describe what is to be achieved.
State in terms of expected behaviors.
Measurable.
Aggregate (group) level – not individual.

School Level: History, Technology & Society

We ask programs to specify the method by which student competencies will be assessed (presentations, embedded exam questions, projects, etc.)
Optional: Indicate location in curriculum.
Optional: Specify performance standards to be achieved.

We remind programs that the same method may be used to cover more than one learning objective/outcome.
 e.g., rubrics or checklists developed to assess senior design projects might be used to assess technical skills, teamwork, and communication skills.
Consider triangulation (multiple methods) to increase reliability of measurement.
Consider the reliability, validity, and quality of instrumentation. Will we be willing to trust the results?

School Level: History, Technology & Society

Succinctly summarize assessment results found
 e.g., “95% achieved passing scores on relevant portion of Fundamentals of
Engineering Exam.”
Provide brief analysis and interpretation of results.

School Level: History, Technology & Society

Include a discussion of actions taken or pending in context of the instructional program based on consideration of results obtained.
Refer to instrumentation used if URL is available.
Give timetable for completion of actions.
 e.g., “The school has redesigned our undergraduate structures sequence and hired an additional faculty to teach the additional course beginning next semester.”

OATS includes an option to create a formatted report for each assessment update
Summary section of update appears as distinct from learning outcomes section
Update report is “laddered” such that methods, results, and actions are indented
Print preview may be saved as an html document and email to others for review

Degree Program Level: BS in HTS

Who is really the customer?
Multiple operating systems and levels of security
Issues with plug-in software, supported applications
Training needs
Operational roles vary across units
Varying levels of administrative enthusiasm and involvement
Tree vs. multidimensional structure

Multiple evaluations/multiple cultures problem:


Many external reviews and accountability demands, all with unique features: SACS,
Board of Regents program review; specialized accreditation (especially ABET)
Culture of disciplines demands custom solutions (especially ABET)

Process to move towards integrated, customizable, culturally acceptable solution

Interviews with key players and users to assess needs during Summer 2005




Task force assigned to develop conceptual recommendations by November 2005
Technical working group will write specifications by March 2006
Programming and testing completed by August
2006
Next OATS version operational for November 2006
OATS

Desirable features identified so far:







Customized “shell” views selected at login for each relevant specialized accreditation body (ABET, CAC, AACSB, NAAB,
NASAD, etc.)
Ability to link to representative courses in curricula
Ability to view OATS entries by objectives/outcomes, by assessment methods used, and by courses in major
Ability to import IR data into unit summary
Ability to import assessment data from GT online assessment data query tool directly into updates
Ability to link across disciplines for project or program evaluations
Ability to do all of the above by yesterday with no faculty effort

Contacts:


Joseph Hoey, joseph.hoey@oars.gatech.edu
or telephone
404.894.0510
Jon Gordon, jon.gordon@oars.gatech.edu
or telephone 404.385. 1419