ASSESSMENT REPORT OF THE GRADUATE PROGRAM IN
COMMUNICATION
Graduate Director: Dr. Jill Rudd Graduate Committee: Dr. Leo W. Jeffres, Dr. Cheryl Bracken Department Chair: Dr. Richard M. Perloff
The purpose of this report is to describe findings that emerged from the assessment of
the Master’s in Applied Communication Theory and Methodology and to discuss
implications for curricular growth and change that can be implemented next year. The
report draws on a departmental assessment procedure developed two years ago that
employs multiple methods to evaluate program goals. Assessment, at its best, generates
critical insights that can be used to improve the quality of an academic program, and we
offer a variety of research-based conclusions in the final section of this report.
A. Description of Program
The Master’s of Applied Communication Theory and Methodology provides
advanced instruction in the fundamental knowledge of the discipline and in the major
quantitative and qualitative research skills used by communication scholars and
practitioners in public and private contexts. The program is designed to provide knowledge
and skills for students pursuing different paths, helping them apply communication research
strategies to career-related problems and in doctoral programs in the field. One of the
oldest social science-based Master’s at Cleveland State, the Communication program
introduces students to the heuristic powers of theory, the essential role methodology plays
in research, and the excitement of applying research to real-world communication
problems.
A Master’s degree student must complete 32 or 38 credit hours, depending on the
choice of exit options. All students are required to take Communication 501, Seminar in
Communication, and Communication 5 12, Communication Research Methods (both core
courses), as well as eight credits of communication theory, an additional four credits of
research methods, four credits in a cognate course, and electives. Upon satisfactory
completion of coursework, the student is allowed to select an exit option appropriate to his
or her career goals. The options are: (1) writing a thesis; (2) completing an applied project
that brings research to bear on a communication problem; (3) taking additional coursework
and successfully passing a comprehensive examination; or (4) successfully completing a
collaborative research project, in which students turn in a final individual paper based on
research conducted by the group.
One of the hallmarks of the Master’s program is its emphasis on introducing graduate
students to the intellectual traditions of the field. Over the years, our students have
presented papers at regional and national academic conferences, stimulated by faculty role
models. They attend conferences like the International Communication Association and the
Midwest Association for Public Opinion Research. They meet scholars from all over the
country. The intellectual ferment engages them cognitively and stimulates them to think
about ways to integrate research into their own lives. Our convention-attending students
have, as a consequence, obtained their Ph.D.s or adopted a research approach in their jobs
in Northeast Ohio companies.
This year graduate students extended these activities by creating a Communication
1
/
Graduate Student Organization. The organization was developed to promote graduate
student research and professional development. The group applied for $1,500 from Student
Government Association and received it! The money was spent on dues and purchases at
the Central States Communication Association meeting in Cleveland in April. Our students
presented research papers, attended sessions, met scholars, and sponsored a well-attended
CSU reception. Students learned a lot and did an excellent job promoting the CSU graduate
program.
B. Goals
The overarching goals of the Master’s program are:
1. To impart to students the ability to identify communication problems and apply
appropriate conceptual frameworks for investigation and problem solving.
2. To impart to students the theories and conceptual frameworks of communication
based on the academic literature produced by the discipline.
3. To teach students the basic procedures for conducting basic and applied
communication research.
4. To develop students’ abilities to conduct and report professional-quality
communication research.
Faculty are involved in the realization of these goals through teaching courses, advising
students, and serving on the Graduate Committee. The committee consists of three faculty
representatives from the three major areas of the program (mass communication,
interpersonal communication and organizational communication), a stbdent representative,
and the Graduate Studies Director. This committee makes recommendations to the general
faculty for changes and modifications in the program. The chair of the department is also
consulted on policy matters and reviews committee actions.
C. Outcomes
Students who successfully compete the program are expected to be able to demonstrate
competence in: (1) clearly formulating communication problems; (2) explicating
communication theories; (3) understanding methodological procedures; (4) conducting
basic/applied research; and (5) professionally reporting research frndings. Evaluation of the
program in accordance with these goals and outcomes is performed by a departmental
assessment committee. Evaluation criteria have been established, and each assessment
committee member independently reviews student’s work (see Appendix A). As noted in
Appendix A, there are specific criteria to determine the extent to which students can
formulate communication problems, demonstrate knowledge of communication theories,
understand research methodology and statistics, conduct basic and applied research, and
professionally report results.
2
D. Assessment Process
The assessment procedure was developed in 2002 and submitted to the OEce of the
Vice Provost for Planning, Assessment, and Information Resource Management last fall. It
provides a mechanism to evaluate the degree to which outcomes are achieved and gives
weight to both student and faculty stakeholders. Assessment has four components.
First, there are regular, formative evaluative reviews during the academic year that are
handled by the Graduate Committee. The committee addresses faculty and student
concerns, and makes recommendations on an ongoing basis.
Second, the Graduate Assessment Committee reviews all students’ exit option work to
evaluate students’ competence levels, as established in the program goals described above.
The committee consists of faculty who can competently address the five different outcomes
described above.
Third, focus groups are conducted with graduate students who have successfully
completed the coursework in an effort to obtain their perspectives on the program. A
recorder takes notes on student reactions and compiles them into a report that is forwarded
to the Graduate Committee. This becomes part of an assessment report that is given to the
graduate faculty in the department.
Fourth, a survey of graduate students is conducted to take into account students’
experiences in the department. It focuses on perceived quality of professors, advising,
views of required courses, and open-ended issues.
E. Assessment Results:Exit Options
In this section we review results of available and prototypical exit option projects. The
assessment is based on an examination of a thesis, project, and three papers completed for
Collaborative Research, Communication 589. Assessments were provided by members of
the Graduate Assessment Committee.
Thesis. Committee members favorably evaluated a thesis on consumer attitudes toward
sexiness completed by Jennifer Eden. This was an experiment that investigated audience
responses to the topical issue of varying levels of sex appeals in magazines. A reviewer
gave the thesis an exemplary rating in its formulation of communication problems and
application of conceptual frameworks. A second reviewer, focusing on methodological
procedures, concluded that the student had provided the necessary details about the
stimulus materials, displayed knowledge of scale construction, and showed she had an
above average understanding of the experimental method.
Project. This student produced a media literacy module for a high school language arts
class. Robert Kelber developed a workbook that provided a media literacy definition,
history, and benefits of the concept, and a nine-week course in media literacy for a
language arts classroom. The reviewer evaluating conceptual frameworks noted that it was
3
a direct application of communication theory to educational goals and called it exemplary.
The reviewer focusing on methodology observed that the project was not entirely
empirical, but said that the method seemed to fit the description of the study.
Collaborativeresearch. The collaborative research team examined direct-toconsumer
advertising through a paper-and-pencil survey. Student papers examined consumers’
reactions to drug ads, violation of consumers’ expectations, and patient satisfaction.
Reviewing the papers, the methodology reviewer noted that the students demonstrated
understanding of scale construction, hypothesis testing, and statistical tests, such as t-tests
and logistic regression.
Concept-oriented and methodology-focused reviewers noted that there was variability in
the exit projects, but regarded them collectively as “exemplary.*
F. Assessment Results: Focus Group Summary
Faculty evaluations lend one perspective on the graduate curriculum. Another vantage
point is provided by students. A focus group was conducted to solicit students’ reactions.
and a quantitative survey offered more precise results. The focus group consisted of three
students who had completed the program (one of whom had an assistantship). It was
facilitated by a faculty member. Results are presented below, with facilitator questions
followed by answers.
Did the program prepare you to identifi communicationproblems and apply appropriate
conceptualframeworks to problem solving? How do you expect to use your skilk; and
knowledge in the real world?
There was general agreement that the program prepared students to identify and
conceptualize communication problems. Students cited Communication 501, the
introductory course, and conducting small studies in mass communication and
persuasion seminars. They stated these assignments reinforced what they learned in
other courses. The students reported that the program helped them in their jobs and
assisted them in better understanding the world as it is presented through the media.
There was consensus that methods courses, especially statistics, helped them in their
jobs.
How would you rate your education in terms of introducing you to various theories and
conceptualframeworks of communication? Which theories and frameworks did youfind to
be most useful?
There was consensus that the program does a good job teaching theory. Two
students stated that they felt there needed to be more emphasis on applying theory to
real-world situations. Another stated that using theories in small studies helped them
better understand theories. The students cited several mass communication theories
as especially helpful: agenda-setting, priming, and gatekeeping.
4
How prepared do you feel you are to conduct basic or applied communication research?
What would you need tofeel more confident of your abilities?
The students agreed that the collaborative exit-option helped them feel more
confident about conducting research. All agreed that conducting studies in their
classes along the way would have made more confident. They said that the few
classes in which they actually conducted studies helped them understand the process
of conducting research. They reported less confidence about using statistics, but felt
that repeated use would boost their confidence.
How easy or diflcult is it to report the results of research? Compared to when you arrived
in the program, is it easier or more dincult?
The consensus was that it is much easier to understand how to conduct research and
interpret results after completing the program. Students reported that they could
read and understand research reports, and had a moderate comfort level in
interpreting their own results. They mentioned again that the program needed to
emphasize the applied aspect more in courses. They also mentioned that they would
feel more comfortable writing research reports if they had been asked to do so in
their coursework.
How would you describe your experience in the collaborative research c h s ?
Students reported enjoying the experience. When asked why they choose
Collaborative, students offered two answers: 1) they wanted to finish the program
quickly [thesis was not perceived as being quick]; and 2) they wanted to avoid the
politics they felt were involved in selecting and working with a thesis committee.
They suggested that there needs to be better information about’howto choose exit
options (in the department’s one- credit course or COM 501), and more attention
paid to helping prepare students to do a thesis.
What would you change about the program?
The students suggested that there needed to be more information available to
students who were not research or teaching assistants. Students who were not
around the department as often as the assistants said they felt “out-of-the-loop.”
They also stated that it would have been helpful to have more information about
how the program works. Students suggested assigning graduate students to an
adviser early on, holding, frequent informational meetings, and mentoring. They
said they would like to have academic tracks within the graduate program. They felt
strongly that COM 531 should be taken the semester prior to Collaborative. They
also were disappointed with the lack of courses offered and the cycle of courses.
5
8
What did you like most about the graduate program?
Students talked enthusiastically about the program in general, specifically the
quality of the courses and several helpful professors. They enjoyed working with
faculty and other students.
G. Assessment Findings: Survey Results
Results of a questionnaire of graduate students conducted in the spring indicated that
students had positive experiences in the program and rated the quality of classes and
advising very positively (see Appendix B). ‘‘This is a wonderful environment,” one student
reported. Students were critical of the lack of access to computers and copiers and
suggested that the department increase access to computers and printers, and begin advising
early in the program.
H. Action
Assessment summary. The assessment of the exit option demonstrated that students
taking three different paths to completing their degrees are acquiring the conceptual and
methodological skills faculty expect Master’s students to acquire. Focus group and survey
results offered self-report-based evidence that students have positive pedagogical
experiences in the graduate program. They reported that the program is good at teaching
theory, facilitates an understanding of the research process, and enhances the ability to
conceptualize communication problems. At the same time, the evaluation identified
shortcomings, including a need to focus more on writing up research, particularly applied
studies, offering methods courses more frequently, providing more access to computers,
and integrating non-RAs and TAs into the program.
Action steps taken. The assessment produced tangible changes in the program. These
include:
1. In response to a comment that methods courses should be offered earlier and more
frequently, faculty made a curricular change this year. Rather than offering
Communication 501, the introductory course, before Com 512, a methods course,
which meant some students had to wait until the second year to take methods,
faculty opted to let students take these courses simultaneously. Efforts will be made
to offer Communication 531, the multivariate course, regularly.
2. In response to student concerns about updating equipment, the department is trying
to add computers to graduate student offices.
I. Future Plans
The assessment generated several new objectives for 2004-2005. First, given the
enthusiasm our students have for the program and its success in connecting them with
professional academic associations, we want to get the word out about our program. This
leads to the following goal:
6
I
*
Objective 1: Recruit more undergraduate students from the CSU program into the
Master’s program and seek to attract more promising undergraduate students from
nearby universities.
We made headway on this objective later in the spring by asking graduate students to
give brief presentations to undergraduate classes on benefits of the program. This resulted
in a flurry of applications from our top undergraduates. In addition our faculty continues to
network at professional conferences to promote our program.
At the same time, based on the ongoing Graduate Committee evaluations, we noted that
there was variability in our students’ test scores, with some scoring high on standardized
tests and others scoring lower than desired. This led to the following goal:
Objective 2: Make a greater effort to recruit high quality students for our program,
including those who obtain high GFE or MAT test scores.
Action Steps:
1. Make assistantships offers early in the spring semester for the following
academic year to ensure a commitment from promising students.
2. Promote the program more actively at professional conferences so that
colleagues will refer their better students to our graduate program.
Objective 3: Help non-RAs and TAs feel more integrated into the program.
Action Steps:
1. Encourage RAs and TAs to network with these students.
2. Consider ways of getting faculty to communicate more frequently in classes and
at the website with these students about the program.
J. Conclusions
In sum, the Master’s in Applied Communication Theory and Methodology, now in its
second quarter-century , continues to stimulate research-oriented graduate students. The
faculty is particularly proud that students took the initiative and formed their own
organization this year. This should increase their involvement in professional and scholarly
research. In this, the first formal assessment of our graduate program using the new
university model, we found it useful to have in place the templates developed two years
ago. We also realized that data needs to be collected more regularly all year. On a positive
note, the assessment suggested several areas for improvement, and the graduate director
and committee will work on these next year. Finally, the inauguration of the
Communication School has implications for the graduate program. The focus of the School
has been primarily on undergraduate curricula, but the graduate program might also be
7
/
I
revitalized. Graduate students’ suggestions about providing more opportunities for applied
research, as noted in the focus group, suggest that we might begin thinking about ways to
integrate research experiences more into classes. New classes that bridge theory and
career-related practice might also be considered. That, alas, is an issue for next year’s
assessment.
8
Appendix A Graduate Program Assessment Criteria Students who successfully complete this program will be able to demonstrate
competence in the following areas: a) Formulating Communication Problems; b)
Communication Theory; c) Methodological Procedure; d) Conducting BasidApplied
Research; e) Professionally Reporting Research Results.
Criteria:
Formulating Communication Problems
Identijjing Communication Problems & Applying Conceptual Frameworks
Exemplary--Demonstrates in thesis/project/collaborativeproject that communication
processes are relevant to either: a) theoretical development, or b) solving problem
of a client in an interpersonal, organization or mass communication context.
Satisfactory--Identifiescommunication processes as relevant to theoretical
development or client’s problem but inappropriately frames research questions for
project.
Unacceptable/Marginal--Failsto state relevant communication processes or selects
inappropriate processes relevant to the stated theoretical development or client’s
problem solving.
Communication Theory
Exemplary--Provides an accurate and correct explanation of identified
communication theories in mass, interpersonal or organization and demonstrates a
comprehensive knowledge of the current academic research literature testing or
employing the theories.
Satisfactory--Provides an accurate and correct explanation of the relevant theories
in mass, interpersonal or organization communication identified but demonstrates a
knowledge of the current academic research literature testing or employing the
theories that is incomplete or exemplary rather thatn comprehensive.
Unacceptable/Marginal--Shows a misunderstanding of the relevant theories in mass,
interpersonal or organization communication identified and demonstrates only a
fragmentary knowledge of the current academic literature testing or employing the
theories.
9
Methodological Procedures
Understanding Social Science Methodologies:
Exemplary--Shows a comprehensive understanding of relevant social science
methodologies (used in projects or examined on comprehensive exams, e.g.,
surveys, experiments, content analyses, unobtrusive methods, participant
observation, panel studies, qualitative methods), when and how they are applied to
test research questions and hypotheses.
Satisfactory--Shows an accurate but fragmentary or incomplete understanding of
relevant social science methodologies, when and how they are applied to test
research questions and hypotheses.
Unacceptable/Marginal-Shows lack of understanding of relevant social science
methodologies, when and how they are applied to test research questions and
hypotheses.
Understanding Statistical Analysis Procedures:
Exemplary--Shows an ability to conduct analyses using such multivariate statistics
and provide appropriate interpretations of results.
as
Satisfactory--Shows an ability to conduct analyses using limited multivariate
statistics and provide accurate but incomplete interpretations of results.
Unacceptable/MarginaI--Showsmisuse of multivariate statistics in analyses and
provides inaccurate interpretations of results
Conducting BasidApplied Research
Exemplary--Develops design for research project that follows steps for social
science research, appropriately integrating conceptualization, a review of the
literature, operationalization of concepts, data collection, analysis and conclusions
appropriate to the project. Successfully carries out all steps of the project from
introduction to conclusions.
Satisfactory--Develops appropriate design for research project that follows steps for
social science research but fails to sufficiently develop one or more steps in design
or execution.
Unacceptable/MarginaI--Designfor research project shows serious
misunderstanding of research process and several steps are flawed in design or
execution.
10
Professionally Reporting Research Results
Exemplary--Provides an accurate, readable and complete report of a research
project following appropriate social science steps, following APA style as stipulated
by Department guidelines.
Satisfactory--Provides an accurate report but with some steps describe incompletely
and errors in style.
Unacceptable/Marginal-Provides a report of a research project that fails to follow
traditional social science practices, fails to follow APA style and is difficulty to
read.
Feedback to Students:
Each year, all graduate students who have successfully completed the coursework and are
eligible for selecting an exit option shall be invited to a focus group meeting with selected
faculty members present. A recorder shall take notes of student reactions to preserve
confidentiality and compile them into a report to be forwarded to the Graduate Committee of
the Department of Communication. A summary of these comments shall be added to the
committee’s report being prepared for the Curriculum Committee and Graduate Faculty in the
Department of Communication.
11 Appendix B Summary of Graduate Student Exit Survey Term:
1. SPRING
Year: 2004
Overall, my experience in the Communication Graduate program was:
Satisfying 45% : 45% : 10% :
2. (n=5)
:
:
Unsatisfying
:
In general, I thought the Graduate Communication courses I took were:
Excellent 66% : 33% :
:
:
:
Poor
:
3.
Specifically, I thought the required courses in the Communication Graduate program
were:
Excellent 10% : 90% :
4. :
Poor
:
:
:
:
:
Poor
:
Overall, the quality of departmental advising was:
Excellent 60% :
6. :
Overall, I thought the quality of departmental instruction was:
Excellent 100% :
5. :
: 40% :
:
:
:
Poor
The department’s facilities were:
Excellent
: 40% : 40% : 10% :
:
:
Poor
7.
What aspect(s) of the Communication Graduate program, if any, do you feel most
positive about?
The faculty and support staff.
Everyone is so helpful and supportive.
This is a wonderful environment.
8.
What aspect(s) of the Communication Graduate program, if any, do you feel most
negative about?
Course cycling-methods 50 1, 512 and 53 1. Access to computers, computer labs, printers, and copiers. It is important to have working and available tools at hand to be productive and efficient. 12 9. What specific steps would you suggest taking to improve the experience of graduate
students in the future?
Making computers more accessible
Upgrading the computers in grad assistant offices
More access to printedcopiers
Updated Comm. Resource Room would be great
5 ) Mentoring/advising from start of the program
1)
2)
3)
4)
13