ASSESSMENT PROGRESS PREPORT (2012 ‐ 2013 ACADEMIC YEARS) Food Science Program: University of Illinois Name of coordinator: Shelly J. Schmidt Title: Professor of Food Chemistry E‐mail: sjs@illinois.edu Faculty and staff who participated in the development or approval of this Assessment Progress Report (please list all): Soo Lee, Dawn Bohn, Mark Lippi, Terri Cummings, Mike Miller, Nicki Engeseth, Faye Dong, and Sharon Nickols‐
Richardson I. PROGRAM OUTCOMES OR COURSE LEARNING OUTCOMES THAT WERE ASSESSED IN THE 2012 ‐ 2013 ACADEMIC YEAR As discussed in our 2011‐2012 Assessment Progress Report, the Food Science Faculty, through a series of four workshop‐style meetings, redeveloped the Illinois Food Science Program Student Learning Outcomes (Appendix A). During those meetings, the 2012 to 2013 Assessment Plan was also developed. The 2012 to 2013 Assessment Plan focused on assessing Program Learning Outcome number 3: Graduates are proficient communicators. This outcome was assessed by evaluating the written and oral communication skills of undergraduate Food Science students at the beginning (freshman in the FSHN 101 Introduction to Food Science and Human Nutrition), intermediate (juniors in FSHN 302 Sensory Science), and end (seniors in FSHN 466 Product Development) of their B.S. degree program. Program Outcome Measured: Graduates are proficient communicators (Outcome number 3 in Appendix A) Methods of Assessment: In FSHN 101, written communication skills were assessed by the course instructor and teaching assistants using the Taking It Home Journal Assignment – an assignment where students keep a journal documenting any instances in which they think about, talk about, or use material from FSHN 101 in a nonacademic situation (e.g., when they are eating out, talking with friends, playing sports, etc.,). Journals were handed in 4 times during the semester with a requirement of at least three entries per submission. Journal entries were assessed by assigning them a ranking of excellent, adequate, or needs improvement. Journal assessment criteria were: content (including following directions), idea development (includes support for thoughts and opinions), organization and clarity, and mechanics (grammar and spelling). In FSHN 302, written communication skills were assessed by the instructor and the teaching assistants through the weekly lab reports and oral communication skills were assessed by the instructor, teaching assistants, and peers through a group project presentation at the end of the semester. Lab reports 1
Summary of Key Findings: were assessed by assigning 90% of the scores to required content and 10% to mechanics of writing. The group project presentations grading was based on 60% content and 40% presentation skills. In FSHN 466, oral and written communication skills were assessed by 4 FSHN staff members (2 Faculty and 2 Teaching Associates; two of which were associated with the course, two were not). Students were assessed as they presented their final group product development projects using the Written and Oral Communication Skills rubrics provided in Appendix B and C, respectively. Project presentations consisted of a short presentation based on questions asked by assessors (see Appendix D for questions asked), a detailed poster presentation, and an opportunity for product tasting. FSHN 101 Overall, most journal entries were assessed as excellent or adequate (an average of 96% of the students received an 80% or better), with only a few entries assessed as “needs improvement.” However, in general, most students wrote their journal entries using a rather informal writing style. In hindsight it would have probably been more appropriate to use a more formal writing assignment for assessment. It is interesting to note that one of the biggest reasons for rating a journal entry as “needs improvement” was that the students did not follow the assignment directions. FSHN 302 The students in FSHN 302 showed improvement in their writing skills as evidenced by the increase in the average lab report scores from the start of the semester of 18.25 out of total of 20 points to the end of the semester of 18.93 out of 20. In general, students did a good job in writing the lab reports, with 100% of students scoring 80% or above by the end of the semester. The majority of comments for improvement in the earlier lab reports were about the missing required content, which students learned to include in their subsequent reports; resulting in the improvement in the average. For the group project presentations, the average scores received from the instructor, the TAs and peers was 95%, demonstrating that overall the students exhibited good oral presentation skills. Since there was only one presentation given at the end of the semester, measure of improvement could not be made. 2
FSHN 466 Regarding their written communication skills, overall the students scored highest on creativity (3.17/4.00) and lowest on documentation (1.79/4.00). The students were deficient in the use of scientific, peer‐reviewed and technical literature during their product development projects. As far as oral communications skills, students scored highest on delivery (2.92/4.00) and lowest on supporting materials (2.17/4.00). In some of the groups, there was a wide variation among the group members in how much each contributed orally to the project presentation – some talked a lot more than others and some seemed a lot more comfortable speaking than others. Additional written and oral rubric scores for FSHN 466 are provided in Appendix E and F, respectively. II. ACTIONS BEING TAKEN AS A RESULT OF THE ABOVE ASSESSMENT FINDINGS On average, students examined in our assessment demonstrated good oral and written communication skills. However, there were extremes in both directions at all three levels ‐ some students far exceeded the average and some were far below the average. For example, in FSHN 466, one team scored a 3.71/4.00 and a 4.00/4.00 on their written and oral communications rubrics, respectively; while another team scored a 1 .86/4.00 and a 2.36/4.00. One action to be taken is to make an intentional effort to determine early in their college experience those students who would benefit from extra assistance in regards to their written and/or oral communication skills, so that additional assistance could be provided. One University resource that would benefit all students is the Writer’s Workshop. The Writer’s Workshop, part of the Center for Writing Studies, provides free writing assistance for University of Illinois students, faculty, and staff from all disciplines and at all stages of the writing process. Writers are able to discuss their writing with consultants who are experienced writers and teachers of writing. In order to expose students to the benefits of the Writer’s Workshop, as part of an assignment in both FSHN 101 Introduction to Food Science and Human Nutrition and FSHN 199 Food Science: Issues and Careers, students will be strongly encouraged to visit the Writer’s Workshop to obtain assistance with their writing assignments. Note, that it is against the Writer’s Workshop policy to make student attendance at the Writer’s Workshop mandatory. In order to provide more feedback to the students on their oral presentation skills, the group project presentation scheme in FSHN 302 is being changed in Fall 2013. There will now be 3 presentations throughout the semester; thus, the FSH 302 teaching team will be able to have a greater impact on improving the oral presentation skills of the students. A trend observed in FSHN 466, was that students were deficient in the use of scientific, peer‐
reviewed and technical literature in their product development project. One of the specific measureable outcomes under the overarching “Graduates are proficient communicators” 3
outcome is that Students who complete our undergraduate Food Science program will be able to critically evaluate scientific and technical literature and mass media information (Appendix A). Additional attention at all levels needs to be paid to helping students understand the value and shortcomings of scientific, peer reviewed and technical literature related to food science and human nutrition, as well as mass media communications. In order to enhance students’ understanding of and appreciation for scientific and technical literature, a session will be added to FSHN 466 during which the students tour the Illinois library and are taught by one of the Illinois librarians. Emphasis will be placed on both the importance of scientific and technical literature when performing research and how to preform a literature search using some commonly used search engines. The students will then carry out a literature search assignment. Lastly, directions will be added to the FSHN 466 project instructions requiring that a minimum number of scientific, peer reviewed and technical articles be included as references in their product development project. One concern we are facing that needs a creative solution is to provide meaningful opportunities for students to practice their written and oral communication skills despite increasing enrollments in all of our courses. We will continue to work on possible solutions to the good, yet challenging problem of increasing enrollments in our courses. Our assessment plan for the coming academic year (2013‐2014) is two fold. First, in order to examine the effectiveness of the actions taken based on this year’s assessment (discussed above), we will continue to assess Program Outcome number 3: Graduates are proficient communicators (details provided in Appendix A). Courses used for the assessment will be FSHN 199 Food Science: Issues and Careers, FSHN 398 Food Science: Professional Issues, and FSHN 466 Product Development. Secondly, we will begin to assess Program Outcome number 1: Graduates are competent in core food science areas and can integrate and apply their knowledge to solve real‐world problems and make decisions (details provided in Appendix A). We are currently developing a knowledge based assessment instrument that students will take at the beginning of their senior year as part of their capstone product development course (FSHN 466). The instrument will assess student competency in core food science areas, including food chemistry, food analysis, food engineering, food microbiology and processing, sensory, and basic nutrition. In addition, at the conclusion of FSHN 466, students will be interviewed to assess their ability to integrate and apply their knowledge to solve real‐world problems and make decisions. 4
Appendix A University of Illinois at Urbana‐Champaign, FSHN Program Level Student Learning Outcomes (Developed Spring 2012) The Bachelor of Science degree program in Food Science at the University of Illinois at Urbana‐
Champaign prepares students to enter careers in the global food industry and related industries, in government, or gain admission to graduate or professional school. The program level student learning outcomes listed below reflect the knowledge, skills, and behaviors that our students acquire as they progress through our undergraduate Food Science program. 1. Graduates are competent in core food science areas and can integrate and apply their knowledge to solve real‐world problems and make decisions Food chemistry and analysis Food safety and microbiology Food processing and engineering Applied food science Nutrition and health aspects of food materials Technical, quantitative, and analytical skills Measurable outcomes: Students who complete our undergraduate Food Science program will be able to: 1. apply the principles of chemistry, microbiology, and processing to produce foods of acceptable stability (chemical, microbial, and physical), quality, and safety 2. evaluate food system reactions during food processing, storage, and utilization 3. analyze the components, properties, and structure of food systems using appropriate methods, including sensory analysis 4. explain the principles behind microbial processes in food systems, including adaptation and environmental factors, fermentation, spoilage and pathogenicity, and relate these to processes for production of safe, stable food systems 5. assess the impact of various food processing operations and conditions on product stability, quality, and safety 6. describe the nutritional aspects of food materials and explain their impact on human health 7. display a working knowledge of contemporary issues/trends and their impact on the food industry and the consumer Example assessment tools: Homework assignments, exams, lab reports, experiential learning 5
activities, field trips, quizzes, journals, individual and group projects, laboratory practical, reviewing published literature 2. Graduates possess strong critical thinking and problem solving skills Question assumptions Practice reflective thinking Think broadly and outside the box Employ a systems perspective Measurable outcomes: Students who complete our undergraduate Food Science program will be able to: 1. apply the scientific method and statistical principles to design and conduct experiments and analyze and interpret data to devise creative, sustainable solutions 2. demonstrate a systems perspective (taking into account all of the behaviors of a system as a whole in the context of its environment) in critical thinking 3. explain the causes of uncertainty in scientific and technical situations 4. critically examine technical literature and apply it to solving real world problems 5. identify and implement innovative technologies and problem solving strategies Example assessment tools: Exams, projects, microtheme questions, case studies; field trips. system design 3. Graduates are proficient communicators Display effective written, symbolic, and oral communications skills Practice good listening skills Utilize scientific literacy skills Measurable outcomes: Students who complete our undergraduate Food Science program will be able to: 1. write clear technical documents (e.g., lab reports, abstracts, project reports, term papers, literature reviews, opinion papers, etc.,) 2. design and deliver effective oral presentations 3. ask thoughtful, appropriate, and suitable questions 4. critically evaluate scientific and technical literature and mass media information 6
5. clearly and appropriately communicate within their professional community, as well as with the general public, about food and agricultural related research and topics Example assessment tools: Written ‐ Short answer and essay questions, reports, term papers, executive summaries, posters, resume construction, literature reviews, portfolios; Oral ‐ Presentations, laboratory practical exams, interviews, sound bites, networking conversations 4. Graduates possess impactful professional and leadership skills Implement effective leadership skills (e.g., self‐motivated, manages conflict, adapts quickly to change, displays integrity and ethical values, manages information and activities, accepts and responses constructively to criticism, provides constructive criticism, good time management skills, etc.,) Work effectively independently and in multidisciplinary teams Self‐directed learner Respectful of diversity Know when to unitask versus multitask Measurable outcomes: Students who complete our undergraduate Food Science program will be able to: 1. demonstrate the ability to work independently and in teams 2. synthesize and prioritize information and activities 3. work effectively with others 4. differentiate between activities that require uninterrupted focus (unitask) and those that can be done in tandem (multitask) 5. continue to educate one’s self as needed (a self‐directed learner) 6. articulate the professional and ethical responsibilities associated with their position Aspirational goals: Graduates of our program 1. value diversity 2. are confident in their abilities 3. volunteer their services to their professional and societal communities 4. take professional and ethical responsibility for their actions Example assessment tools: Individual and group projects, discussion groups, experiential learning activities, field trips, laboratory partners, position paper, scavenger hunt, ice breaker activities, peer evaluation, resumé construction, networking assignment, pre‐flight questions, attendance and class participation; self‐evaluation quizzes; extracurricular activities 7
Appendix B. Written Communication Rubric Beginning = 1
Criteria
A. Content
Importance of topic, relevance,
accuracy of facts,
overall treatment of topic
B. Sense of Audience
Tone, style and word
selection are targeted to
audience
C.
Organization/Clarity
Logical ordering of ideas,
transitions between paragraphs,
coherent, concise
D. Completeness
Level of detail, depth,
development of ideas,
appropriate length
E.
Grammar/Mechanics
Correct usage, spelling,
proper sentence/paragraph
structure, follows accepted
format conventions
F. Documentation
Proper use of citations,
support for major ideas,
use of visual aids
G. Creativity
Original topic, unique or
original presentation of
ideas
Developing = 2
Benchmark
Proficient = 3
Milestones
Mastery = 4
Capstone
Report topic lacks focus or
relevance; text contains multiple
fact errors or omissions.
Topic would benefit from more
focus; text contains some fact
errors or omissions.
Topic is adequately focused;
information is generally relevant
and accurate.
Report topic is tightly focused;
text contains relevant information
with no fact errors.
Unfocused or absent sense of
audience
Inconsistent sense of audience; Uses
language inappropriate to target
audience
Conveys an accurate sense of
audience with appropriate use
of disciplinary language
Strong sense of audience
demonstrated through form and
language
Ideas are not presented in
proper order; text lacks transitions
between major ideas;
text is wordy or unclear.
Some ideas not presented
in proper order; paragraph
transitions needed in places;
some text is wordy or unclear.
Most ideas are in logical order
with adequate transitions
between most paragraphs; text
is generally clear and readable.
Ideas are presented in logical
order with effective transitions
between paragraphs; text is
clear and concise.
Text does not provide adequate
depth; important details or ideas
are omitted; unclear or undeveloped;
report is too short or too long.
Additional depth needed in
places; important details or ideas
sometimes omitted or not fully
developed; report may be short or
long.
Text provides adequate depth;
few needed details or ideas are
omitted; major ideas adequately
developed; report is proper length.
Text provides good depth and
detail; ideas well developed; facts
have adequate background
information; report is within
specified length.
Text contains several spelling,
grammar or punctuation
errors; sentences are incomplete;
report does not contain
required sections.
Text may contain some spelling,
grammar, punctuation or
sentence errors; report contains
most required sections.
Text has no serious spelling,
grammar or punctuation errors;
sentences are mostly complete;
report contains required
sections.
Text contains no spelling,
grammar or punctuation errors;
sentences are readable
and complete; report contains
required sections.
Little or no support provided for
major ideas; citations and/or
visual aids are missing, old or
inadequate.
Some major ideas need additional
support; visual aids
and/or some citations contain
errors or need work.
Most major ideas are supported
by citations and adequate
visual aids; most citations are
current and used properly.
Major ideas are supported by
citations and effective visual
aids; citations are current and
used properly.
Report provides no new or
original ideas; approach lacks
Report contains some original
ideas or attempts to use
an original or innovative
approach.
Report may address a unique
or original topic; creative effort
is evident and could be successful
with minimal editing.
Report addresses a unique
or original topic; innovative
approach is used in introduction
or other sections.
8
Appendix C. Oral Communication Rubric Criteria
A. Content
Importance of topic, relevance,
accuracy of facts, overall treatment of
topic
B. Organization/Clarity
Appropriate introduction, body,
conclusions; logical ordering
of ideas; transitions between
major points
C. Completeness
Level of detail, depth, appropriate
length, adequate background
information
D.
Grammar/Vocabulary
Correct grammar and usage
that is appropriate for
audience(s)
E. Delivery
Adequate volume, appropriate
pace, diction, personal appearance,
enthusiasm/energy, posture,
effective use of visual aids
F. Interaction
Adequate eye contact with
audience, ability to listen and/or
answer questions
G. Supporting
materials
Tone, style and word
selection are targeted to
audience
Beginning = 1
Developing = 2
Benchmark
Proficient = 3
Milestones
Mastery = 4
Capstone
Topic lacks focus or
relevance; text contains multiple
fact errors or omissions.
Topic would benefit from more
focus; text contains some fact
errors or omissions.
Topic is adequately focused;
major facts are generally relevant
and accurate.
Topic is tightly focused; presentation
contains accurate information with no fact
errors.
Ideas are not presented in
proper order; transitions are
lacking between major ideas;
several parts of presentation are
wordy or unclear.
Some ideas not presented in
proper order; transitions are
needed between some ideas;
some parts of presentation may
be wordy or unclear.
Most ideas are in logical
order with adequate transitions
between most major ideas;
presentation is generally clear
and understandable.
Ideas are presented in logical
order with effective transitions
between major ideas; presentation
is clear and concise.
Presentation does not provide
adequate depth; key details are
omitted or undeveloped; presentation
is too short or long.
Additional depth needed in
places; important information
or details sometimes omitted or
not fully developed; presentation
may be short or long.
Presentation provides adequate
depth; few needed details are
omitted; major ideas adequately
developed; presentation is
within specified length.
Presentation provides good
depth and detail; ideas well developed;
facts have adequate
background; presentation is
within specified length.
Presentation contains several
major grammar or usage errors;
sentences are long, incomplete
or contain excessive jargon.
Presentation may contain some
grammar or sentence errors;
sentences may contain jargon
or are too long or hard to follow.
Presentation has no serious
grammar errors; sentences are
mostly jargon-free, complete
and understandable.
Presentation contains no
grammar errors; sentences are
free of jargon, complete and
easy to understand.
Low volume or energy; pace
too slow or fast; poor diction;
distracting gestures or posture;
unprofessional appearance;
visual aids poorly used.
More volume/energy needed
at times; pace too slow or fast;
some distracting gestures or
posture; adequate appearance;
visual aids could be improved.
Adequate volume and energy;
generally good pace and diction;
few or no distracting gestures;
professional appearance;
visual aids used adequately.
Good volume and energy;
proper pace and diction; avoidance
of distracting gestures;
professional appearance;
visual aids used effectively.
Little or no eye contact with
audience; poor listening skills;
uneasiness or inability to answer
audience questions.
Additional eye contact needed
at times; better listening skills needed;
some difficulty answering
audience questions.
Fairly good eye contact with audience;
displays ability to listen;
provides adequate answers to
audience questions.
Good eye contact with audience;
excellent listening skills;
answers audience questions
with authority and accuracy.
Insufficient supporting materials
(explanations, examples, illustrations,
statistics, analogies, quotations from
relevant authorities) make reference to
information or analysis that minimally
supports the presentation or establishes
the presenter's credibility/authority on
the topic.
Supporting materials (explanations,
examples, illustrations, statistics,
analogies, quotations from relevant
authorities) make appropriate reference
to information or analysis that partially
supports the presentation or establishes
the presenter's credibility/authority on
the topic.
Supporting materials (explanations,
examples, illustrations, statistics,
analogies, quotations from relevant
authorities) make appropriate reference
to information or analysis that generally
supports the presentation or establishes
the presenter's credibility/authority on
the topic.
A variety of types of supporting materials
(explanations, examples, illustrations,
statistics, analogies, quotations from relevant
authorities) make appropriate reference to
information or analysis that significantly
supports the presentation or establishes the
presenter's credibility/authority on the topic.
9
Appendix D 2012 ‐ 2013 Program Level Outcome Assessment for FSHN 466 The goal is to assess our communications Program Level Outcome (Graduates are proficient communicators, see complete excerpted text below) using the attached written and oral communication rubrics. Please use the following base and follow up questions as a starting point for assessing your assigned student groups. Mark Lippi will assign each of us to review 3 out of the 12 groups. Since we are assessing both written and oral communications skills make sure to take some time and review the student’s poster as well as ask them the questions below. Once you have filled out the rubrics, please give them to Shelly Schmidt by Dec 21, 2012. Feel free to make note of how the students are doing regarding other Program Level Outcomes, but our main task this time is to assess their oral and written communication skills. I will set up a time for all of us to meet and discuss the results of our assessment for all the courses at the beginning of next semester. Q1. Can you provide me with an overview of your project? (Main opening question)
Follow up questions: Q2. If you had a group of experts at your disposal that could help you with your project, what questions do you have for them? Q3. Tell me about your use of the scientific and technical literature as you worked on your project. Q4. Explain the benefits of your product to a lay audience. 10
Appendix E. Written Communication Rubric Scores – FSHN 466 Fall 2012 Criteria Team No. Content Sense of Audience Organization/Clarity
Completeness
Grammar/Mechanics
Documentation
Creativity
Average and Range 1 2 3 4 5 6 7 8 9 10 11 12 Average and Range 3 3 2.5 2.5 3 4 2.5 3 2 3 2 2 3 3 1.5 2.5 3 4 2.5 3 3 3 1 3 4 3 2 2.5 3 4 2 2 3 2.5 1 3 3 2 2 2.5 2 4 2 2.5 2 2.5 2 3 4 2 3.5 3 4 4 3 2 3 3 3 3 2 2 1 2 2 2 1 2 2 2 1 2.5 4 3 3 2 4 4 3 2.5 4 3 3 2.5 3.29 2.57 2.21 2.43 3.00 3.71 2.29 2.43 2.71 2.71 1.86 2.71 2.71 2.71 2.67 2.46 3.13 1.79 3.17 1.86‐3.71
1.79‐3.17
11
Appendix F. Oral Communication Rubric Scores – FSHN 466 Fall 2012 Criteria Team No. 1 2 3 4 5 6 7 8 9 10 11 12 Average and Range Content Organization/Clarity Completeness Grammar/Vocabulary Delivery Interaction Supporting Average Materials and Range 3 3 3 2 3 2 2 2.57 3 3 2.5 3 3.5 3 2 2.86 3 2.5 2.5 3 3 3.5 1.5 2.71 2 2 2.5 3 2.5 2.5 1.5 2.29 3 3 2 2 2 2 2 2.29 4 4 4 4 4 4 4 4.00 2 2 2 3 3 3 1.5 2.36 2.5 3 2.5 3 3 3 2 2.71 3 3 2 2 3 3 2 2.57 3 3 2.5 3 3 2.5 3 2.86 2.5 2 2.5 2.5 2.5 2.5 2 2.36 3 3 3 2.5 2.5 3 2.5 2.79 2.83 2.79 2.58 2.75 2.92 2.83 2.17 2.29‐4.00
2.17‐2.92 12