advertisement

Chapter 5: Reliability Concepts – – – – Definition of Reliability Test consistency Classical Test Theory X = T + Obtained True Score Score E Random Error – – – – – Estimation of Reliability Correlation coefficients (r) are used to estimate reliability The proportion of variance attributable to individual differences Directly interpretable Reliability of .90 accounts for what % of the variance? • – – – – – – Conceptual True Score Variance Fine for true score variance T = X E Convert formula to variances (2) Partial out the ratio of obtained score variance and error variance Substitute the ratio of obtained score variance for 1 Reliability is the ratio of error variance to obtained score variance subtracted from 1 – – – – – – Types of Reliability Test-retest Alternate Forms Split-Half Inter-item consistency Interscorer – – Test-Retest Reliability Coefficient of stability is the correlation of the two sets of test scores • – Alternate (Equivalent) Forms Reliability Coefficient of equivalence is the correlation of the two sets of test scores – Split-half Reliability – – Coefficient of internal consistency is the correlation of the two equal halves of the test. Reliability tends to decrease when test length decreases – – Spearman-Brown Formula A correction estimate The S-B formula is computed with the following ratio: # new items # original items – – – Spearman-Brown Example Reducing the test length reduces reliability What is the new estimated reliability for a 100-item test with a reliability of .90 that is reduced to 50 items? – – – – – Inter-item Consistency The degree to which test items correlate with each other Two special formulas to look at all possible splits of a test a) Kuder-Richardson 20 b) Coefficient Alpha – – Inter-scorer reliability Tests (or performance) are scored by two independent judges and the scores are correlated - What are fluctuations attributed? • – Possible Sources of Error Variance Error differences associated with differences in test scores – Time Sampling – Item Sampling – Inter-Scorer Differences – – Time sampling Conditions associated with administering a test across two different occasions – – – Item Sampling Conditions associated with item content Content heterogeneity v. homogeneity – – Inter-scorer differences Error associated with differences among raters • – Factors affecting the reliability coefficient Test length – The greater the number of reliable test items, the higher the reliability coefficient – Larger test length increases the probability of obtaining reliable items that accurately measure the behavior domain Heterogeneity of scores Item Difficulty Speeded Tests (Timed tests) Based on speed of work, not consistency of the test For example, consistency of speed, not performance Test situation – Conditions associated with test administration Examinee-related – – – – – – – – – – – – Conditions associated with the test taker Examiner-related – conditions associated with scoring and interpretation Stability of the construct – dynamic v. stable variables stable variables more reliable Homogeneity of the items – The more homogeneous the items, the higher the reliability – – – – Interpreting Reliability A test is never perfectly reliable A method for interpreting individual test scores takes into account random error We may never obtain a test-taker’s true score • • • • Standard Error of Measurement (SEM) Provides an index of test measurement error SEM interpreted as standard deviations within a normally distributed curve. SEM is used to estimate true score by constructing a range (confidence interval) within which the examinee's true score is likely to fall given the obtained score SEM Formula St is the standard deviation rtt is the reliability – • • – – – – – SEM Example For example, X = 86, St = 10, rtt = .84 What is the SEM? What is the Confidence Interval (CI)? Within 1 standard deviation, there is a 68% chance that the true score falls within the confidence interval – 2 SDs = 95% – 3 SDs = 99% – – – – – – – – Generalizability Theory Extension of Classical test theory Based on domain sampling theory (Tryson, 1957) Classical Theory emphasizes test error Generalizability Theory emphasizes test circumstances, conditions, and content Test score is considered relatively stable Estimates sources of error that contribute to test scores Variability is the result of variables or error in the test situation – – – – Importance of Reliability Estimates accuracy/consistency of a test Recognizes that error plays a role in testing Understanding reliability helps a test administrator decide which test to use – Strong reliability contributes to the validity of a test