The Testing Environment
and Threats to Validity Considered
“. . . the play's the
thing wherein I'll
catch the conscience
of the King.” –
Hamlet, Prince of Denmark
Or
Catching your own
conscience; reflecting on
your testing approaches
to improve validity. You
don’t want extraneous
factors to impact scores.
Topics on Administering & Analyzing Tests
When using them for important summative assessments
 Test administration
 Item analysis
 Cheating
Fairness – Real and Perceived
 Important that students perceive value in the assessments

Motivated to do especially well on tests related to


recorded grades, report cards
passing or failing decisions
 Three attributes of perceived fairness

Teach what you test.


Give adequate notice.


Broadly meant; not just what you said in class.
Pop quizzes are okay as long as you let students know they are a
possibility.
Be fair in grading.


Consistent and reasonable.
Fairness does not mean easy. Many students appreciate high
standards and a challenging test.
Suggestions, prior to test day, for . . .
Preparing Test Materials
1.
Give clear directions – what, if any, supplemental materials are
permitted (scratch paper, dictionaries, calculators).
2.
3.
Use a simple format – use “white space” and logical page breaks.
Allow lead time for printing up the exams – don’t race into office
and expect your test to be done right away.
4.
5.
Spot check – pages complete, in correct order, not inverted.
Take one test and answer it – non-fatal mistakes you can correct
with oral directions; fatal means re-printing (don’t postpone).
6.
7.
Worry about security – keep secure; may wish to number copies.
Comparable forms for multi-section courses – students in later
classes will ask students in earlier sections about the test.
Suggestions, prior to test day, for . . .
Preparing Yourself and the Students
Assuming you have provided excellent content
instruction, recommended actions include:
Give adequate notice to the students. This includes the type(s) of
items that will be found on the test.
 Schedule the test so that you have time built-in so that when the test
is completed you have sufficient time for scoring of the test.
 Consider likely circumstances.

Are there any special equipment needs?
 Is there anything scheduled that could be disruptive (fire drill, pep rally)?
 Should you pre-guide students as to what to do if there is unanticipated
commotion in the room or building? (What to do with test documents.)


Make advance preparation for accommodations.
Suggestions, on the day of the test, for . . .
Immediate Preparation of Students
1. Think about classroom seating arrangement – would rearranging be
offensive; how would you rearrange (e.g. every other seat).
2. Determine how materials (e.g., test, answer sheet) will be passed out
(yourself; passed front to back) - check that everyone has all materials.
3. Establish control – students often talk when materials are passed about,
decide when to declare conversation time over.
4. Direct students to place names on answer sheet – and on test itself if
this is important to you for security reasons. This may suggest to student’s that you
look at any marks they make on the test itself.
5. Review test directions – some students skip the written directions while
others agonize too long over them (should be no surprises).
6. Provide directions as to what the students do when they finish their
test (e.g., turn in, wait quietly, study).
7. Give quick words of encouragement – humor may work or may backfire.
Suggestions, once the test starts, for . . .
Managing the Testing Environment
1.
Monitor the testing situation – don’t use this time as personal time
to work on other classes; circulate around the room, help students
stay on task; see if it appears someone is not correctly following the
directions (make sure you tell students you will be doing this). When
you do sit, sit behind students out of their vision.
2.
Answer questions with fairness in mind – don’t give hints to
individual students, clarification only. If it appears what you might be
telling them is important, consider telling the entire class.
3.
Apprise students of remaining time – if timing is important or if
there is no clock visible.
Suggestions, once the test finishes, for . . .
Scoring and Returning Test Results
1.Many teachers dislike this part; it is time consuming. The utility of having
provided yourself with a previously prepared scoring key and time to score
is key for success in this area. I suggest you score exams yourself.
2.Return results promptly; most often the sooner the better. Getting results
back quickly is good for the mental health of students, and yourself!
3.Don’t announce individual scores (although you can publicly
acknowledge a student’s outstanding performance); don’t post grades
(legal issues). You may make general narrative statements (either positive
or negative) . . . I prefer positive (for example, the class did well).
4.Reiterate the scoring system you used. Your scoring approach should
not be a surprise to the students but, by restating it, you demonstrate you
tested them exactly as you said you would.
5.Decide how you want to deal with student questions. Will you do over all
items, some items? What kinds of questions will you accept, how long will
the questions go on? Will you change grades based on the discussions?
Suggestions, once the test review finishes, for . . .
Retaining Test Results
1.
Recover materials as desired – do this in such a way
that students do not see the scores of others. Do you want all
parts back? Can students keep old tests for studying? If
collected, what will you do with them?
2.
Record scores – do this after you have gone over test
with students. Use grade book (hardcopy or electronic).
3.
Make notes - on changes you would like to make in the
assessment system when you teach this in the future. Do this
now while it is fresh in your mind.
Item Analysis: Example
Note: the values in the table are percentages
of correct responses, High/Low/Total Group
Classroom Item Analysis, “So What” . . .
 Item analysis is important because a poor or bad item may add to the
unreliability of the test. Why? Because if your purpose in the test was
to distinguish between those students who know the material and those
who do not, a bad item fails to work toward this end (e.g., items
answered correctly by all, or items that no one answers correctly.)
 A rough "rule-of-thumb" is that if the item difficulty is more than .75, it is
an easy item; if the difficulty is below .25, it is a difficult item.
 It is my feeling that items with difficulties less than 20% or more than
80% (especially if they fail they discriminate) deserve your attention.
Such items should either be revised or replaced. Perhaps you have a
non-functioning distracter. Perhaps your question misleads the best
students. An exception might be at the beginning of a test where
easier items (90% or higher) may be desirable for initial confidence
building.
Taking item analysis to the next level . . .
Item Analysis Programs
If you get into item analysis on a large scale (many students/items) consider:
 Independent vender computer software “Test Packages” on test construction
which often include item analysis features. These may not be good long term
since the companies that produced them come and go (no tech support).
 The “Test Development Software Package” provided by the publisher of the
textbook you are using.
 Test item analysis using the “Microsoft Excel” spreadsheet program.
 Your school district may have a centralized test scoring service (usually limited
to selected response types of exams). This likely means that you would need to
require that your students use a bubble sheet because it would be scanned.
These computer programs may use more complex calculations for indices (aka,
indexes) related to item difficulty or discrimination but the concept will be the same.
For example, item discrimination might be presented in terms of a correlation
coefficient (with a Greek letter or an unusual sounding name, of course).
Remember how to interpret a correlation coefficient? The scale is “-1 to +1” so you
would interpret it much like the “D” statistic.
Classroom Item Analysis . . .
Examining the quality of individual items; “p” and “D” statistics
 When the test is completed, we have two basic questions:


Was the difficulty of the item appropriate or was it too hard or too
easy? I can compute the item difficulty index by dividing the
number of correct answers by total answers:
 p = percent correct
 The higher the value; the easier the item
If the item was difficult, was this difficulty okay? Does it discriminate
between those who know and those who don’t? To answer this we
compute the item discrimination index. In this case we look at each
item in terms of did the students who do the best on this test tend to
get it right more so than the students who did the worst. The book
divides the class in half (H & L); I tend to like thirds (H, M, L).
Compare the percent correct between H & L groups
 D = H% - L%
 The higher the value, the more the item discriminates . . . notice
it go negative or positive. What’s a “+” or “-” mean?
Cheating
Graphic from “A Taxonomy of Text and Exam Fraud” by Dennis Maynes, 2008
Say it ain’t so, Joe . . .
Cheating and Academic Integrity



Most teachers would rather not discuss this; perhaps a form of denial.
Prevalence
 All levels of education
 Student View “No big deal – everybody does it.”
Practices
 Plagiarizing, especially Internet-based
 Cut and paste without attribution
 Other projects completed outside class
 Assistance from parents, siblings, family friends, fellow students range
from incidental to complete authorship
 Cheating on exams, tests (crib notes, look at another’s test, pass notes)
 Let’s look at some cheating examples; a major source was
 Gary K. Clabaugh & Edward G. Rozycki, Preventing Cheating and
Plagiarism, 2nd Edition (2003) Oreland, PA: NewFoundations
Press
Taking note of “crib” notes . . .
So called because of how one “cradles” the small sheets.
OLD SCHOOL:
 Get into the classroom prior to test and write crib notes on the desktop or the
back rest of seat in front. Variation, tape note folder on underside of desktop.
 Make sure the crib notes can be easily destroyed. Example, chew gum during
exam; if a teacher becomes suspicious eat the piece of paper. Do NOT write on
yourself, it is impossible to hide if a teacher questions you.
 Print off crib notes in size 6 font about 4” wide by ½” long. Acquire transparent
pen (BIC recommended). Roll the paper around the tube of ink and slide it in.
 Remove label from a clear beverage, such as Snapple. Put notes on the back of
the label, then paste it back in place using transparent glue. During the exam
take slow "thoughtful" swigs out of the bottle.
 Wear laced up shoes on the day of the exam (with laces remove). Attach your
crib notes behind each tongue of the shoe.
NEW SCHOOL:
 Crib notes on an electronic organizer: a mini tape recorder can be fitted with a
tiny earphone that can be hidden by combing hair over the ear; programmable
calculators; use of a cell phone in a lavatory.
Other Cheating Gambits . . .







Signal answers among group. Keep an eye out for the rearrangement of items
on desktops, coded body movements, and hand signals.
A student who has postponed a test or who takes it in a later section, enlists a
confederate to smuggle an extra copy of the exam out of the exam room.
Fraternity approach. Smuggle a copy of the test from the room at the end of the
test, note the answer sequence when the test is reviewed then keep both the
test and the answers on file.
Student sits for the exam, pretends to be taking the test, but completes nothing.
Then they turn in the blank exam and makes sure the instructor notices it has
been returned. When the tests are graded and the student gets a zero because
they didn't take the test, he or she complains bitterly that their test has been lost.
When tests are returned students either alter their answers from wrong to right
or fill in answers they deliberately left blank. Then they complained that their test
was marked incorrectly.
Students fake injury or illness to postpone tests for which they are unprepared.
Students show up late, feign surprise and claim that they misunderstood when
the test was to be given.
Cheating
Suggestions for Prevention
1.
Create positive classroom atmosphere - reasonable standards,
sensible workloads, keep content interesting, keep upbeat attitude yourself.
2.
Talk to students about cheating – define cheating behavior, identify
consequences; immediate/long term. Is cheating rewarded in society?
3.
Reduce anxiety and uncertainty - be clear and informative about
upcoming tests, what will be tested, how.
4.
Be vigilant – consider changing seating, walk among the students during
testing.
5.
When cheating occurs, confront it – teachers’ disregard of cheating
is one reason students say they feel so free to do it.
6.
Determine if your school has a policy; support it - if the policy is
nonfunctional, then help develop new policy.
Practical Advice
1. Use summary checklists regarding each
aspect of test administration to remind you
of considerations at each phase you value.
2. Learn to do simple item analysis, use
occasionally.
3. Take cheating seriously.