Measurement - Outline Measuring variables 1 2 Measuring Variables Accurately and Consistently “If you can’t measure it, it doesn’t exist” Number System + Psychological Variables Some things can be directly measured Scales of Measurement Implications of Measurement ¡ Behavioral / Observational (watch them) (monitor physiological responses) ¡ Self-Report (ask them) ¡ Physiological Some things can not be directly measured ¡ Create a construct that can be measured and approximates the variable of interest A construct HYPOTHESIZED CONSTRUCT 3 A construct is a hypothetical variable which is Anxiety not directly observable ¡ ‘Intelligence’ ¡ ‘Happiness’ Threat of aversive event Researchers estimate each participant’s level of Unfamiliar situation/change the construct by measuring variables that are somewhat related to the underlying construct. No opportunity for escape Measuring variables 5 Almost all measurements include some degree of error or noise We must evaluate the accuracy and consistency of the measurement Physiological measures: •Sweating •Increased heart rate •Increased blood pressure •Rapid, shallow breathing Behavioral measures: •Inhibited behavior (unable to perform simple tasks) •Pacing, rapid eye movements, startle-reflex •Stammering, other speech problems Other indicators: •Responses to “anxiety assessment” •Expressed need to be with others OBSERVABLE EVENTS Validity and Reliability 6 Validity (accuracy) ¡ Webster: “soundly based on facts or evidence” ¡ genuine, credible, true Reliability (consistency) ¡ Repeat, replicate ¡ If the characteristic being measured is stable, a reliable test yields consistent results 1 Reliability Measuring Reliability 7 The extent to which the test is consistent in its evaluation of the same individual over repeated administrations ¡ Can the variables be measured reliably? you measure them again, do you get the same result? ¡ If Calculate correlations to assess reliability: ¡ Test-retest ¡ Split-Half ¡ Parallel or Alternate Forms ¡ Item-Total ¡ Coefficient Alpha (Cronbachs) Validity Construct Validity 9 Accuracy or correctness Can be very narrow ¡ applied to a particular variable Can be very broad ¡ applied to an entire study or program of research Not directly measurable ¡ evaluated Is the operational definition of a variable accurate? ¡ Are you measuring what you think you are measuring? Face Validity ¡ Does your operational definition appear to measure the construct? based on logic and reasoning Assessing Construct Validity Assessing Construct Validity (cont’d) Procedure ¡ Did anything about the procedure add noise or error to the measurement? (e.g., fatigue, embarrassment, boredom, suspicion) ¡ What was it like to be a participant? ¡ If an independent variable was manipulated, was there an appropriate control group? What steps were taken to ensure that the only difference between the two groups was the level of the independent variable? Method Match ¡ Is this an appropriate method to measure the construct? ¡ Is there anything about the method that adds noise or error to the measurement? ¡ Are behaviors being directly observed? ¡ Are physiological states being directly monitored? ¡ If you are asking about an attitude or a belief, is it something that the participant knows? can remember? will tell the truth? 2 Method Match Additional Construct Validity 13 Thing you want to measure Method used Actions Observation Ideas Self-Report Physical State Monitor Construct Validity Does it measure present or future performance? ¡ Concurrent and Predictive Validity Does it relate appropriately to other measures? ¡ Convergent and Divergent (Discriminant) Validity Review – construct validity 24 We want to measure how many people are living in an area with a high number of undocumented immigrants ¡ We can ask. What sources of noise or error would we be dealing with? ¡ What else can we do? When less is more 25 People's emotional responses to events are influenced by both ¡ ¡ The actual outcome Their thoughts about "what might have been” Which is better?Medvec, et al. (2002) 26 To earn a B+ in a class? To earn a C- in a class? Coming second in a race, or coming third? Medvec, V., Madey, S. F., & Gilovich, T. (2002). When less is more: Counterfactual thinking and satisfaction among Olympic medalists. In T. Gilovich, D. Griffin, D. Kahneman (Eds.) , Heuristics and biases: The psychology of intuitive judgment (pp. 625-635). New York, NY US: Cambridge University Press. Retrieved from EBSCOhost. 3 “What might have been” Method 27 The most compelling alternative for the silver medalist is winning the gold. The most compelling alternative for the bronze medalist is finishing without a medal. 28 Reviewed televised footage of 1992 Summer Olympics ¡ No sound – just video Twenty undergraduates coded the emotional responses of the athletes ¡ 10-point agony to ecstasy scale undergrads were uninterested and uninformed about sports ¡ 10 students rated each scene – high inter-rater reliability (they agreed with each other’s ratings) ¡ The Results Results 30 Bronze medalists displayed more positive emotional reactions than silver medalists Those who are objectively better off but have a more-positive “what might have been” feel worse than those who are not as well off but have a more-negative “what might have been” Medvec, et al. (2002) 10-point "agony to ecstasy” scale 31 8 7 6 5 Silver Bronze 4 3 2 1 0 Immediate Internal and External Validity Medal Podium Statistical Validity 33 Internal Validity ¡ Was it the independent variable that caused a change in the dependent variable? ¡ Was the study conducted well? ¡ Were alternate explanations ruled out? ¡ Were all potential confounds eliminated? External Validity ¡ Does this study reflect what typically happens in the world? ¡ Do the results generalize to other situations? ¡ Do the results generalize to all people? Generally, as one goes up, the other goes down Drawing valid conclusions based on appropriate statistical analysis Every time you draw a conclusion, you might make a mistake ¡ Seeing a relationship that isn’t there that is there ¡ Missing a relationship How strong is the relationship (effect size)? How large is the margin of error? 4 Frequency Association Causal Claims Claims Claims More thoughts on Reliability and Validity 34 38 More is better ¡ 20-item scales are generally more reliable than 10-item scales ¡ Observations at multiple different times and locations will be more reliable than a single session Construct Validity If a measure is valid, it must also be reliable ¡ Reliability is necessary for validity ¡ Reliability is not sufficient for validity (you can reliably measure something that has low construct validity) Internal Validity Statistical Validity External Validity Wake up your clickers Measurement - Outline 45 Measuring Variables Accurately and Consistently Number System + Psychological Variables Scales of Measurement Implications of Measurement Our abstract number system 46 Identity ¡ Each number has a particular meaning Magnitude ¡ Numbers have an inherent order from smaller to larger Equal intervals ¡ The difference between units is the same anywhere on the scale True Zero ¡ Zero means nothing, nil, all gone! Psychological Variables Scales of Measurement 47 49 Identity Magnitude Equal Intervals Not all variables in psychological research fulfill the criteria of the abstract number system Which criteria a variable meets determines which mathematical operations (+, -, *, /) can be conducted True Zero Nominal Scale Ordinal Scale Interval Scale Ratio Scale 5 Nominal Scales Nominal Scales: Example 50 51 Male Naming Scales Categorical Variables Property of Identity ¡ each level has a particular meaning Not measured with an inherent order Cat-lover Democrat Female Non-binary Non-gendered Dog-lover Republican Independent May be assigned a number to assist with coding, but the number assigned is arbitrary Ordinal Scales 53 54 Property of Identity Property of Magnitude ¡ levels of a variable are arranged in an order levels represent more of the variable than do lower levels ¡ higher 1st ‘Place’ in a race 2nd 3rd We can rank order variables measured on ordinal scales Socioeconomic class lower middle upper Interval Scales Interval Scales - Examples 56 Property of Identity Property of Magnitude Property of Equal Intervals ¡ the distance between consecutive levels is equal across the scale We can meaningfully add and subtract variables measured on interval scales 57 Personality Tests ¡ Example: Introversion – Extroversion ¡ Answer 10 questions about your preferences ÷A ÷A score of 9 is 2 more than a score of 7 score of 7 is 2 more than a score of 5 Intelligence Quotients ¡ Average of 100 points ÷A score of 110 is 10 points higher than 100 and the same 10 points lower than 120 6 Ratio Scales Ratio Example 59 60 Property of Identity Property of Magnitude Score on an exam Property of Equal Intervals Property of True Zero ¡ allows for ratios to be made - i.e., a person who has study for 5 hours has studied for half as long as someone who has studied for 10 hours The student who scored 100 points scored twice as many as the one who scored 50 points. We can meaningfully multiple and divide variables measured on a ratio scale Mathematical Operations Deciding on a level of measurement 62 Rank Order Nominal Scale Ordinal Scale Interval Scale Ratio Scale 63 Add and Subtract Multiply and Divide Sometimes levels of measurement are inherent within the construct. Sometimes it’s a function of how we choose to measure the construct. ✔ ✔ ✔ Sometimes it’s a function of our ability to ✔ ✔ measure the construct. ✔ Implications of Measurement Ratio 64 How you measure the variables directly influences what conclusions you can draw ¡ Psychological Continuous measures are rarely true ratio or Interval interval scales ¡ Researchers need to be careful about the conclusions they draw The questions you want to answer dictate how you will chose to measure the variables How you measure the variables dictates how you analyze the results Ordinal Categorical Nominal 66 7
