Setting the Stage for Program Assessment:
A Case Study of the School of Communication
(Part I—9:40-10:10)
Putting on the Play:
A Case Study of the School of Communication
(Part II—10:20-10:50)
Brian H. Spitzberg
San Diego State University

Or,
“How I learned how to stop kvetching and just do assessment in a way that’s relatively painless and yet, helpful.”


 Not if, but when
 If we must do assessment, we should…
 optimize value-added / effort + investment
 keep it as fair as possible
 keep it as reliable & valid as possible
 All assessment is subjective in one way or another


 All assessment must occur at both:
 molar/holistic & molecular/microscopic levels
 both self and other perspectives
 Assessment should reflect:
 both inductive & deductive development
 change over time (multiple assessment times)
 motivation, knowledge, and skills
 Don’t reinvent the wheel

I.
Survey to Produce Content (mission, goals, SLOs)
A. Deductive approaches
B. Inductive approaches
II. Translation into Goals & SLOs
A. Reduction
B. Formalization draft
III. Preliminary Validity
A. Representational/Face
B. Expert review
IV. Formal Articulation:
A. Translation into rubrics
B. Anchoring the scaling
V. Integration into Turnitin/Grademark  WEAVE

I.
Survey to Produce Content (mission, goals, SLOs)
A. Deductive approaches:
 Disciplinary consensus documents
 Accreditation documents
 A priori program goals
 Theoretical models (e.g. Bloom)
B. Inductive approaches:
 Survey faculty
 Survey stakeholders (alumni, profession, etc.)
 Survey curricula and syllabi
C. [For us, this produced ≈ 250 SLO’s]

Adapted Bloom’s
Learning Domains
[MOTIVATION]:
Affective attitude
Responding to phenomena
Interpreting phenomena
Perceiving phenomena
Internalizing values
Organizing
Evaluating/Creating
Synthesizing/ Reassembling
Analyzing/ Disassembling
[KNOWLEDGE]:
Cognitive/Mental
[SKILLS]:
Applying/Generalizing
Comprehending/Understanding
Psychomotor & Behavioral
Procedural record revision
Feedback adaptation
Origination
Threshold routine activation
Procedural records review

[KNOWLEDGE]:
Cognitive/Mental
Evaluating/ Creating
Synthesizing/
Reassembling
Analyzing/
Disassembling
Applying/
Generalizing
Comprehending/
Understanding
Designing, constructing, planning, checking, producing, inventing, animating, critiquing, representing, writing, imagining, validating, brainstorming, analogizing, etc.
COMMUNICATION
EXEMPLARS:
Asserting
Campaigning
Collaborating
Comforting
Commenting
Integrating, hypothesizing, modeling, linking, organizing, judging, detecting, monitoring, reviewing, moderating, collaborating, networking, etc.
Contributing
Debating
Empathizing
Comparing, outlining, deconstructing, attributing, finding, structuring, reverse engineering, separating, extracting, etc.
Expressing
Interviewing
Leading
Listening
Implementing, specifying, planning, using, executing, running, loading, playing, operating, instantiating, exemplifying, testing, experimenting, etc.
Mediating
Negotiating
Networking
Persuading
Recognizing, listing, describing, identifying, retrieving, naming, locating, finding, highlighting, remembering, recalling, specifying, delineating, etc.
Questioning
Adapted broadly from: http://edorigami.wikispaces.com/Bloom%27s+Digital+Taxonomy
Replying
Reviewing
Speaking etc.

II. Translation into goals & SLOs
A. Reduction:
 Scan for redundancy, excessive abstraction, etc.
 Edit for consistency in grammar, action verbs, tense, etc.
 Print goals/SLOs on cards, and conduct “sort”
 Multifinality: multiple solutions can be explored
 Decide on a workable preliminary ‘model’
B. Formalization draft: Write a preliminary…
 “micro” model (exemplar SLOs)
 “macro” model (conceptual integration)

III. Preliminary validity & utility
A. Representational/‘Face’:
 Check for faculty buy-in
 Run by some colleagues who are also working on assessment
B. Expert:
 Check professional associations
 Check for consultants
 Check with those at successful programs

I
THE COMPETENT ‘CITIZEN’
COMMUNICATOR
II
DISCIPLINARY
AWARENESS (HISTORY,
BOUNDARIES, ETC.)
III
CENTRAL CONCEPTS AND
ASSUMPTIONS OF
COMMUNICATION
IV
KNOWLEDGE OF
APPLIED CONTEXTS,
FUNCTIONS
V
KNOWLEDGE OF
THEORY, MODELS,
CONCEPTS
VI
KNOWLEDGE/APPLICATION
RESEARCH METHODS &
APPLICATIONS
VII
COMPETENT ARGUMENT
CONSTRUCTION,
ANALYSIS, PRESENTATION
VIII
WRITING (APA, LIBRARY
RESEARCH)

I
THE COMPETENT ‘CITIZEN’
COMMUNICATOR
II
DISCIPLINARY AWARENESS
(HISTORY, BOUNDARIES,
ETC.)
III
CENTRAL CONCEPTS AND
ASSUMPTIONS OF
COMMUNICATION
IV
KNOWLEDGE OF THEORY,
MODELS, CONCEPTS
V
KNOWLEDGE OF APPLIED
CONTEXTS, FUNCTIONS
VI
KNOWLEDGE/APPLICATION
RESEARCH METHODS &
APPLICATIONS
COMPETENT ARGUMENT
VII
CONSTRUCTION, ANALYSIS,
PRESENTATION
VIII
WRITING (APA, LIBRARY
RESEARCH)
Ability to communicate ethically in a participative society
Perform basic communication competence/skills
Demonstrate awareness of the role of communication in context
Demonstrate knowledge of history/structure of the discipline
Demonstrate understanding of primary contexts of communication
Demonstrate awareness of major communication media & systems
Identify and critique core communication principles & assumptions,
Demonstrate understanding of importance of communication
Demonstrate ability to analyze complexity of communication processes
Identify, differentiate, compare, contrast theories
Create & apply theories with conceptual rigor and contextual relevance
Evaluate theories using established paradigmatic criteria for critique
Identification of concepts & principles in applied contexts
Demonstration of relevant knowledge in applied contexts
Demonstration of relevant skills in applied contexts
Identify, differentiate, compare, contrast methodologies
Apply methodologies to conduct original paradigmatic research
Evaluate methodologies within and across relevant paradigm standards
Develop comprehensive, relevant, coherent arguments
Present persuasive arguments orally and in writing
Evaluate arguments by established tests of reasoning and evidence
Demonstrate scholarly authorial voice in writing
Demonstrate mastery of APA style guide
Demonstrate mastery of scholarly research tools/engines

I
THE COMPETENT ‘CITIZEN’
COMMUNICATOR
II
DISCIPLINARY AWARENESS
(HISTORY, BOUNDARIES,
ETC.)
III
CENTRAL CONCEPTS AND
ASSUMPTIONS OF
COMMUNICATION
IV
KNOWLEDGE OF THEORY,
MODELS, CONCEPTS
V
KNOWLEDGE OF APPLIED
CONTEXTS, FUNCTIONS
VI
KNOWLEDGE/APPLICATION
RESEARCH METHODS &
APPLICATIONS
COMPETENT ARGUMENT
VII
CONSTRUCTION, ANALYSIS,
PRESENTATION
VIII
WRITING (APA, LIBRARY
RESEARCH)
The School of Communication seeks to graduate students who can communicate competently in the major walks of life they will encounter.
The School of Communication seeks to graduate students with an awareness of the history, nature, scope, and evolution of the communication discipline.
The School of Communication seeks to graduate students who have a working knowledge of the core concepts, definitions, and assumptions of the communication discipline.
The School of Communication seeks to graduate students who can identify, differentiate, and relate core paradigms, theories, and models in the communication discipline.
The School of Communication seeks to graduate students who can diagnose the relevance and implications of paradigms, theories, and models of communication in hypothetical and actual contexts.
The School of Communication seeks to graduate students who can identify, differentiate, relate and apply the major methodological paradigms in the generation of original research.
The School of Communication seeks to graduate students who can extemporaneously and proactively generate and competently present sound arguments in communicative performance contexts.
The School of Communication seeks to graduate students who can competently locate, filter, retrieve, evaluate, and cite scholarly research in the composition of well-written textual documents employing APA format.

IV. Formal Articulation:
A. Translation into rubrics:
 Review existing rubrics
 Then compose your own
B. Anchoring the scaling:
 Review existing rubrics
 Personally, I recommend a 5-point
 Then compose your own
 Consider both elaborated & abbreviated forms

WRITING/APA:
FORM:
Demonstrates proficiency in grammar, syntax, semantics, academic voice, and application of APA style guidelines.
CLAIMS/
PROPOSITIONS--
CONTENT:
Demonstrates ability to articulate researchable claims specifying the interrelationship among variables.
1: 0-20 (F)
Form displays: multiple types & instances within type of writing or grammatical errors in expository text,
&/or displays inconsistency in rule application; frequent re-editing or rephrasing to achieve more professional voice is suggested.
1: 0-20 (F)
The key claims are not clearly articulated or delineated.
Propositions fail by level of scaling, relationship, or syllogistic entailment.
“Object lessons” or “list of horrors” are repeated.
2: 21-40% (D)
Form displays: moderately low level of professional voice, composition, and grammatical form with moderate number of errors, inconsistency of rule application, or required editing
(intermediate to scales 1 & 3).
2: 21-40% (D)
Form displays: moderately low level of relationship specification or implications of claims or propositions, or makes several errors in wording
(intermediate to scales 1 & 3).
3: 41-60% (C)
Form displays: few types & instances within type of writing or grammatical errors in expository text,
&/or inconsistency in rule application; occasional reediting or rephrasing to achieve more professional voice is suggested.
3: 41-60% (C)
Only minor or one or two claims or propositions need editing for sake of clarity.
4: 61-80% (B)
Form displays: moderately high level of professional voice, composition, and grammatical form with moderately few errors, inconsistency of rule application, or required editing
(intermediate to scales 3 & 5).
5: 81-100% (A)
Writing displays consistent use of professional voice, composition, and grammatical form.
4: 61-80% (B)
Form displays: moderately high level of relationship specification or implications of claims or propositions, or makes few errors in wording
(intermediate to scales 3 & 5).
5: 81-100% (A)
Propositions are both logically sound, and sophisticated in their thematic connection &/or articulation of complex relationships.

PRESENTATION
PERFORMANCE:
Demonstrates the ability to orally express ideas, thoughts, claims, propositions, arguments, and evidence in a competent verbal and nonverbal communication
RESEARCH/DATA:
CONTENT:
Demonstrates ability to locate and appropriately cite and list recent, relevant, and reasonable scholarly research, consisting mostly of peerreviewed journal sources.
1: 0-20 (F)
Demonstrates neither awareness of nor ability to integrate competent verbal and nonverbal communication into an oral presentation of express ideas, thoughts, claims, propositions, arguments, and evidence.
1: 0-20 (F)
No more than one or two directly or peripherally related external sources were brought to bear on the validity of the selected source claims, &/or those sources applied are distantly relevant to source claims; &/or sources lack recency, relevance, or scholarly imprimatur.
2: 21-40% (D)
Demonstrates little awareness of and ability to use competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.
2: 21-40% (D)
Form displays: moderately low number and quality of location, citation, and listing of recent, relevant, and reasonable scholarly sources
(intermediate to scales 1 & 3).
3: 41-60% (C)
Demonstrates an awareness and ability to use competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence with a minimal confusion and lack of clarity.
3: 41-60% (C)
At least 1 to 2 studies relevant to each selected proposition, argument, component, or source claim are provided, &/or some sources applied are distantly relevant to source claims;
&/or sources lack recency, relevance, or scholarly imprimatur.
4: 61-80% (B)
Demonstrates an intermediate awareness of and ability to employ competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.
4: 61-80% (B)
Form displays: moderately high level of development of sound, articulated, and evidential warrants for claims, with high status and appropriate sources
(intermediate to scales 3 & 5).
5: 81-100% (A)
Demonstrates a mastery of competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.
5: 81-100% (A)
Each major claim is evidenced by sources high in scholarly credibility (i.e., relevance, recency, peer review, etc.)

V. Integration into Turnitin/Grademark
A. Re-check faculty buy-in and editing input
B. Every faculty selects:
 at least 1 assignment
 in each course to which
 at least 1 of the rubrics will be applied
C. A diagnostic grid results of SLOs by courses
 Which SLO’s are underrepresented?
 Which courses are under-assessed?
D. Export rubric to all faculty
E. End-of-Semester downloads to Excel
F. A program “Assessment Chair” will receive, strip, aggregate, and summarize across curricula

 Assignment inbox
 GradeMark report
 Rubrics
 Export Excel
 Auto-Average each common rubric across curricula
 Transfer data to WEAVE

IMPACCT Home Page IMPACCT AdminPage

“Take your total SAT scores, add your average net income for the last three years, divide by the number of times you have sex per week, then multiply by your age. That gives us a numerical indicator of your over-all worth as a human being”

 Description : An online, self-administered communication & critical thinking skills assessment.
 Advantages :
 Heritage
 Breadth
 360
 peer assessment
 Longitudinal assessment
 Relative convenience (online, self-administered)
 Potential for national/regional comparative norms
 Psychometric soundness
 Portfolio/feedback resource for students
IMPACCT Home Page IMPACCT AdminPage

 Composition : ≈ 40 constructs, organized as:
 Motivation (e.g., public speaking confidence)
 Knowledge/Critical thinking (e.g., problem-solving)
 Conversation/interaction (e.g., attentiveness)
 Interpersonal/Relational (e.g., negotiation)
 CMC (e.g., medium selection x message content)
 Group/Leadership (e.g., agenda management)
 Public (e.g., attention-getting introduction)
 Outcomes (e.g., appropriateness, effectiveness)
IMPACCT Home Page IMPACCT AdminPage

IMPACCT
What is impacct?
Students
Taking
COMM
Depts.
Test
Overview
Informed
Consent
Student log-in
Campaign
Creation
Preview
Assessment
Control
Panel
IMPACCT Home Page IMPACCT AdminPage

Assessment site architecture:
IMPACCT
What is impacct?
#
Students
Taking
COMM
Depts.
Test
Overview
Informed
Consent
Student log-in
Campaign
Creation
Preview
Assessment
Control
Panel
IMPACCT Home Page IMPACCT AdminPage

Assessment site architecture:
IMPACCT
What is impacct?
Students
Taking
#
COMM
Depts.
Test
Overview
Informed
Consent
Student log-in
Campaign
Creation
Preview
Assessment
Control
Panel
IMPACCT Home Page IMPACCT AdminPage

Assessment site architecture:
IMPACCT
What is impacct?
Students
Taking
COMM
Depts.
#
Test
Overview
Informed
Consent
Student log-in
Campaign
Creation
Preview
Assessment
Control
Panel #
IMPACCT Home Page IMPACCT AdminPage

 Approximately 2 years of data collection
 All constructs reliable
 No apparent sex or ethnicity bias
 Modest ‘Wobegon’ effect (means 4.5-5.5 on 7-point)
 Significant improvement over time (38 of 40 constructs)
 R 2 = .31 to .73 in predicting communication quality
 Peer ratings slightly related to other and to self-ratings
 Impacct 2.0 expanded to multiple campaigns

 Develop an “assessment institute” at SDSU
 Fund RAs
 Provide resource for theses
 Innovation cauldron
 Research grants/awards for annual proposals
 Generate annual “state of the communication nation” report
 Provide consulting opportunities for institutional reporting needs
 Prospective innovations:
 Video capture for peer assessment stimuli
 Listening test
 Emotional intelligence/sensitivity test
 Media literacy test
 Brief therapeutic screens

IMPACCT:
Spitzberg, B. H. (2011). The Interactive Media Package for Assessment of Communication and
Critical Thinking (IMPACCT © ): Testing a programmatic online communication competence assessment system. Communication Education, 60, 145-173.
Spitzberg, B. H., Lee, C. M., & Lindemann, K. (2011, February). Of dimensions and dementia:
Desiderata in determining the composition of communicative competence and skills. Top
Four paper presented at the Western States Communication Association, Monterey, CA.
Competence:
Spitzberg, B. H. (2003). Methods of skill assessment. In J. O. Greene & B. R. Burleson (Eds.),
Handbook of communication and social interaction skills (pp. 93-134). Mahwah, NJ:
Erlbaum.
Spitzberg, B. H. (2009). Axioms for a theory of intercultural communication competence [invited article, Japanese Association of Communication and English Teachers]. Annual Review of
English Learning and Teaching, No.14, 69-81.
Spitzberg, B. H., & Cupach, W. R. (2011). Interpersonal skills. In M. L. Knapp & J. A. Daly (Eds.),
Handbook of interpersonal communication (4 th ed., pp. 481-524). Newbury Park, CA: Sage.

sdsu
.
“I think you’ll find my test results are a pretty good indication good indication of your abilities as a teacher.”