VALIDITY AND RELIABILITY OF QUESTIONNAIRES BY – JASKARAN SINGH CONTENTS Introduction Steps in questionnaire designing Validity Concept of validity Types of validity Steps in questionnaire validation Reliability Types and measurement of reliability Conclusion JASKARAN SINGH INTRODUCTION Questionnaire: Important method of data collection used extensively Advantages of questionnaire Less expensive Offers greater anonymity Disadvantages Application is limited Response rate is low Opportunities to clarify issues is lacking JASKARAN SINGH Ideal requisites of a questionnaire: Should be clear and easy to understand Layout is easy to read and pleasant to eye Sequence of questions easy to follow Should be developed in an interactive style Sensitive questions must be worded exactly JASKARAN SINGH Steps in questionnaire designing JASKARAN SINGH Steps in questionnaire designing JASKARAN SINGH Validity JASKARAN SINGH The concept of validity Validity is the ability of an instrument to measure what it is intended to measure. Degree to which the researcher has measured what he has set out to measure (Smith, 1991) Are we measuring what we think we are measuring? (Kerlinger, 1973) Extent to which an empirical measure adequately reflects the real meaning of the concept under consideration (Babbie, 1989) JASKARAN SINGH Why validity ? Validity is done mainly to answer the following questions: Is the research investigation providing answers to the research questions for which it was undertaken? If so, is it providing these answers using appropriate methods and procedures? JASKARAN SINGH Questions to ponder(Think about) Investigator Readers of report Experts in the field Logic Statistical tests JASKARAN SINGH Logical thinking Justification of each question in relation to objective of study Easy if questions relate to tangible matters Difficult in situations where we are measuring attitude, effectiveness of a program, satisfaction etc Everybody’s logic doesn’t match . . No statistical backing JASKARAN SINGH Statistical procedures By calculating coefficient of correlations between questions and outcome variables JASKARAN SINGH Types of validity Validity Content validity Face validity Criterion related Concurrent Predictive JASKARAN SINGH Construct validity CONTENT VALIDITY Uses logical reasoning and hence easy to apply Extent to which a measuring instrument covers a representative sample of the domain of the aspects measured Whether items and questions cover the full range of the issues or problem being measured Example, if we want to test knowledge on American Geography it is not fair to have most questions limited to the geography of England. JASKARAN SINGH Validity CONTENT What it measures How it is accomplished Whether the test covers a representative sample of the domains to be measured Ask experts to assess the test to establish Steps in questionnaire validation JASKARAN SINGH CONTENT VALIDITY Two phases Experts: Enhancement of content of questionnaire (Six or more experts) Researcher: Conceptualization and domain analysis(finding common and variable parts) How do experts evaluate validity Method 1: Average Congruency Percentage (ACP) [Popham, 1978] Experts compute the percentage of questions deemed to be relevant for them Take the average of all experts If the value is > 90 . . . Valid Eg: 2 experts . . (Expert 1-100%, Expert 2-80%) Then ACP = 90% JASKARAN SINGH JASKARAN SINGH JASKARAN SINGH JASKARAN SINGH Reliability JASKARAN SINGH RELIABILITY Definition: It is the ability of an instrument to create reproducible results Each time it is used, similar scores should be obtained A questionnaire is said to be reliable if we get same/similar answers repeatedly Reliability measured in aspects of: STABILITY • Done to ensure that same results are obtained when used consecutively for two or more times • Test-retest method is used INTERNAL CONSISTENCY • To ensure all subparts of a instrument measure the same characteristic (Homogeneity) • Split-half method EQUIVALENCE • Used when two observers study a single phenomenon simulataneously • Inter-rater reliability JASKARAN SINGH Test-Retest reliability (for stability) Test administered twice to the same participant at different times Used for things that are stable over time Easy and straight-forward approach Useful for questionnaires, checklist, rating scales etc Disadvantages Practice effect (mainly for tests) Too short intervals in between (effect of memory) Some traits may change JASKARAN with time SINGH Statistical calculation Administration of instrument to a sample on two different occasions Scores compared and calculated by using correlation coefficient formula (pearson) JASKARAN SINGH Inter-rater reliability (Equivalence) Inter-Rater Reliability refers to statistical measurements that determine how similar the data collected by different raters are. A rater is someone who is scoring or measuring a performance, behavior, or skill in a human or animal. Used when a single event is measured simultaneously and independently by two or more trained observers R= Number of agreements Number of agreements + Number of disagreements JASKARAN SINGH Summary of Reliability TEST RETEST SPLIT HALF INTERRATER What it measures Stability over time Equivalency of items Agreement between raters How it is accomplished Administer the same test to the same people at two different times Correlate performance for a group of people on two equivalent halves of same test Have multiple researchers measure same instrument and determine percentage of agreement between them JASKARAN SINGH Conclusion Validated questionnaire It is one which has undergone a validation procedure to show that it accurately measures what it aims to, regardless of who responds, when they respond, and to whom they respond or when self-administered and whose reliability has also been examined thereby: Reducing bias and ambiguities Better quality of data and credible information JASKARAN SINGH In a nutshell . . . . A questionnaire can be reliable but invalid . . . But a valid questionnaire is always reliable . . . JASKARAN SINGH JASKARAN SINGH
