Interpreting IDEA reports
Diagnostic Form
Short Form
www.lvc.edu/idea
Page 1 of report
Page 1 of report
Guide says: “Report Responds to
These Questions”
1. Overall, how effectively was this class taught?
2. How does this compare with the ratings of other
teachers?
3. Were you more successful in facilitating progress
on some class objectives than on others?
4. How can instruction be made more effective?*
5. Do some salient characteristics of this class and
its students have implications for instruction?
* Diagnostic Form Report only.
Our Modest Ambitions Today
• Understand summative scores
– Raw averages
– Converted scores
– Adjusted scores
– How used at LVC
• Formative use of p.2 – relevant objectives
– POD papers on objectives
• Formative use of p.3 for Diagnostic only
– POD papers on strategies
Summative scores ….
How “Summary Evaluation” Computed
• 41. Overall, I rate this instructor an excellent
teacher. (25%)
• 42. Overall, I rate this course as excellent.
(25%)
• Objectives 1-12 (21.-32) that are ranked as
Important or Essential (50%)
“Converted averages”
• Transform group’s mean to 50, s.d. to 10.
• Examples:
– Score 57 means .7 group s.d.’s above average
– That’s at 75.8 %-tile of Normal Distribution
– … but these groups of IDEA scores not Normal
6
5
4
Normal
3
IDEA
2
1
0
0
20
40
60
80
Adjusted Scores:
Effects that are removed
P. 2 of Diagnostic Form report:
Adjusted Scores:
Effects that are removed
P. 2 of Diagnostic Form report:
Only these on Short Form
When to Use Adjusted Scores for Administrative Decisions:
Where is there evidence of effective teaching? (old)
Are
adjusted
scores lower or
higher than
unadjusted
scores?
Lower
Higher
Yes
No
Use
adjusted
scores
*Expectations defined
by your unit.
Is unadjusted
Score on
Progress toward
Objectives high?
(≥4.2)
Use
unadjusted
(raw) scores
When to Use Adjusted Scores for Administrative Decisions:
Where is there evidence of effective teaching? (new)
Are
adjusted
scores lower or
higher than
unadjusted
scores?
Lower
Higher
Yes
No
Use
adjusted
scores
*Expectations defined
by your unit.
Do
unadjusted (raw)
scores meet or
exceed
expectations?*
Use
unadjusted
(raw) scores
College Use for Evaluating Faculty
• Current:
– Summative: Pay attention to extremes
– Assess formative use of IDEA by instructor
• IDEA recommendation
– Summative: Establish standards
• Faculty set standards
• Department level or College level
• Assess progress in meeting them
Page 2: Objectives
P. 3 of Diagnostic Form Report:
Directions….
Follow the links…
www.theideacenter.org/podidea
Teaching Methods Relevant to
Target Objectives
Teaching Methods Relevant to
Target Objectives
Try it
1. Identify objective…
2. Find related teaching method…
3. Follow link: www.theideacenter.org/podidea
Last page: Raw response frequencies
… especially additional questions
We can retrieve the additional questions so you
know which are which.
Effects of Evals being On-Line?
• Study of on-line vs paper by IDEA published in
2010:
– http://www.theideacenter.org/sites/default/files/Tech
nical Report 16.pdf
• Online avg response rate 57%
– LVC response rates 70-75%
• No significant change in means or s.d.’s of
responses
– If loss of middle group, s.d. would go up.
– On 12 objectives, s.d.’s slightly lower for on-line!
Key questions for LVC (from IDEA)
• How much emphasis will you give student ratings in
the evaluation of teaching?
• How many classes need to be rated? [LVC: all]
• Performance categories: How many should you use?
• Do you plan to authenticate objectives?
• Will you weight the importance of items or groups of
items in the decision-making process? If so, how?
• Will you use averages on a five-point scale or
converted averages?
• Will you use adjusted or unadjusted scores?
References
• www.lvc.edu/idea – Campus IDEA page
• POD notes:
– The 20 methods: www.theideacenter.org/podidea
– The 12 objectives:
www.theideacenter.org/podidea/PODNotesLearning.html
• IDEA: www.theideacenter.org
• Technical:
http://www.theideacenter.org/sites/default/files/
techreport-12.pdf - Data about adjusting, etc.