Session 3: Framework for a Coding Guide October 27, 2009 2:00-3:30 PM (EDT) Instructor: Chad Nye 1 Life’s Elixir of Youth, Beauty, Wisdom & Painless Coding Dr. Chad’s Wonder Drug 2 A wholesome blend of natural extracts of thirty different herbs and root vegetables that promotes and revitalizes the capacity for learning and memory, strengthens the nerves, and effectively wards off cataleptic neuroplexy AND coding anxiety. 3 Inclusion and Exclusion Criteria Reduce Reviewer Bias: Criteria are determined before beginning Information Retrieval ‘Standardize’ the process and resulting decisions 4 Inclusion and Exclusion Criteria: Potential Categories Study Design Population Intervention Outcomes Measures Publication Sources 5 Coding for Fun and Profit Dr. Chad’s Wonder Drug taken 2 times a day will result in the complete life experience of the life well lived!!! 6 Coding is a 2 Step Process Stage 1 – Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria 7 Inclusion Criteria Abstract & Title Stage Studies will meet at least one of the following criteria: 1. Participant The participant sample of study must include only adults 18 years or older, with a diagnosis of Autism Spectrum Disorder. 2. Intervention The focus of the study intervention must center on the topic of employment. 8 Inclusion Criteria Abstract & Title Stage Studies can be advanced to the next stage if: 1. They meet the criteria OR 2. If these criteria are not clear from the title or abstract 9 Coding is a 2 Step Process Stage 1– Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria Stage 2– Full Text Identify the studies that will be included in the review itself Use full set of criteria 10 Inclusion Criteria Full Text Stage Research Design Studies use an experimental (RCT), quasiexperimental design (QED), and single subject experimental design (SSED) Participants Individuals with an autism spectrum disorder who are 18 years of age or older and not enrolled in a secondary school program (e.g., high school) 11 Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention The treatment must be directed towards the provision of vocationally oriented services intended to produce employment outcomes for individuals with autism spectrum disorders. 12 Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention d. Outcome Measures The study must report subsequent attainment of an employment placement and provide specific data about the duration and/or retention of that placement. Employment will be considered to encompass full or part-time (25% time or more) placements. Interventions of any length duration will be included provided adequate 13 description is provided. Inclusion Criteria Full Text Stage a. Research Design b. Participants c. Intervention d. Outcome Measures e. Publication Status Eligible studies may be either published or unpublished, written in any language, and conducted in any country. 14 Inclusion/Exclusion Decision Tree Potentially relevant reports identified (n=583) Reports excluded by title and abstract review (n=523) Reports progressing to next level (n=60) Reports used to code studies for inclusion (n=20) Reports excluded at Phase II screening (n=40) 15 Study Coding* Purpose of Coding 1. Identify all those variables that you believe might make difference in the observed results. 2. Organize those individual variables into categories of characteristics in order to evaluate the quality of the study 3. Try to determine what is really important about the study 4. Quantify both text and data for analysis 16 Coding Form 5. Guide for locating and recording the who, what, where, how, when, and how much of a study (aka-variables) 6. A record of the ‘interpretation’ of the variables that can be independently replicated (aka-a monitor on the accuracy of judgments) 7. Provides the basis for the data arrangement at the analysis stage 17 Study Coding Categories of Study Characteristics Design Participant Treatment Outcome 18 Design Characteristics Type of Design RCT QED Pre-Experimental Single Subject Blinding Attrition Sampling Method 19 Participant Characteristics Information is collected for BOTH Experimental AND Control groups. Number of Participants Age Gender Severity Language Setting 20 Treatment Characteristics Type of Tx Method Length of Tx Program Tx Mode: Group-Individual Number of Tx Sessions Length of Tx Sessions Dosage 21 Outcome Characteristics Outcome Variable(s) Outcome Measure(s) Assessor of Outcome Assessor Qualifications Time Post Tx to Outcome Measure Immediate Follow-Up 22 CodeBook A dictionary of all the terms used in the Coding Form A guide for all coders to follow to reduce error and bias A reference source for accurate replication 23 A. Publication Source 1. Journal 2. Dissertation 3.Thesis 4. Unpublished Paper 5. Book 6. Other: ________ B. Subject Characteristics (page(s): ____________) Group Pretest Posttest Attrition Age Grade(s) (n) (n) (n) (n) (n) Males (n) Tx: _22___ _22__ ______ ______ ______ _____ Cp: ______ _____ ______ ______ ______ _____ Co: _22___ __18_ __ 4___ ______ ______ _____ 24 C. Socioeconomic Status (page(s):___________ 1. Low 2. Middle 3. Upper 4. Low-Middle 5. Middle Upper 6. Labeled Mixed 7. Unclear 8. ___________ 9. NR Comments:__________________________________ 25 D. Sample Source (page(s)_________) 1. Public School 2. Secular Private School 3. Religious Private School 4. Other:_________________ 5. NR Comments 26 E. Study Setting (page(s) ______) 1. Urban 2. Suburban 3. Rural Geographic Setting: ______________________ Comments: 27 F. Author’s Labeling of Participants’ Achievement Levels (page _____) 1. Average Achieving 2. High Achieving 3. Underachieving 4. At-Risk 5. Unclear 6. Other:___________________ 7. NR G. Race/Ethnicity (page ______) 1. African American 2. American Indian 4. Hispanic/Latino 3. Asian 5. White 6. Other: ____________________ 7. NR 28 H. Intervention Characteristics 1. Type of intervention (page __________________) 2. Total length of intervention program: __________ or NR (page _____) 3. Length of time PER intervention session:________ or NR (page _____) 4. Number of sessions: ________ per ________ (page _____) or NR 5. Describe the intervention(s) intended for delivery to the study groups including how and when it was administered.______________________________________ 29 6. Fidelity of implementation data reported? (page(s)______) 0=No 1=Yes If yes, describe the intervention(s) intended for delivery to the study groups including how and when it was administered: ________________________________________________ ________________________________________________ ________________________________________________ ________________________________________________ 7. Intervention implemented as described? (page(s) _____) 1=No 2=Yes 3= Somewhat 9=NR 30 I. Outcome Measure(s) Type of Measure Instrument Variable(s) Who Administered the Measurement Instrument? 1. Norm Referenced Test (page____) _________ a. Parent b. Teacher _________ c. Researcher d. Other _________ e. NR 2. Criterion Referenced Test (page____) _________ a. Parent b. Teacher _________ c. Researcher d. Other ________ e. NR 4. Rating/Survey (page____) _________ a. Parent b. Teacher _________ c. Researcher d. Other e. NR _________ a. Parent b. Teacher _________ c. Researcher d. Other _________ e. NR 5. Class Grade (page____) 31 J. Design Characteristics Recruitment Pool (page_____) 1. Referral 2. Criterion 3. Norm Reference Test 4. Existing Group 5. Cohort 6. Volunteer 7. Other ___________________ 99. NR Research Design Type (pg____) 1. Group (Cluster, Place-based) Randomized Design 2. Individual Randomized Design 3. Quasi-Experiment: Interrupted Time Series Design 4. Quasi-Experiment: Regression Discontinuity 5. Quasi-Experiment: Nonequivalent Comparison Group Design 6. Single Subject Design 7. Other Designs: _____________________________________ 32 K. Assignment Procedures (If applicable, otherwise skip to section I) Subject Assignment (page(s) _____) 1. Individual Random 2. Group Random 3. Individual Match Random 4. Quasi-random 33 2. Method of Random Allocation (page(s) _____) Method of Random Allocation Who Implemented the Random Allocation? 1. Random Numbers (page____) a. Parent b. Teacher c. Researcher d. Other e. NR 2. Coin Flip (page____) a. Parent b. Teacher c. Researcher d. Other e. NR 4.Other:_______(page____) a. Parent b. Teacher c. Researcher d. Other e. NR 34 Was concealment of allocation used as part of the randomization procedures? 1=no 2= yes 3=NR a. If yes, were the allocation procedures concealed from any of the following people? 1. Researcher (pg _____) 2. Participant (pg _____) 3. Intervener (pg _____) 4. Assessor (pg _____) 35 I. Quasi-Experimental Assignment Procedures (If applicable, otherwise skip to section J) 1. Was any of the following research design techniques used to equate groups prior to the intervention? • • • • a. Matching on pretest proxy (pg _____) b. Matching on a pretest (pg _____) c. Matching on other variables (pg _____) d. No Matching (pg _____) 2. Was statistical equating used prior to the intervention? 1=no 2=yes 3.NR Comments:________________________________________ 36 J. Effect Size Characteristics Comparisons: Group 1: _TX___vs. Group 2:_Ctl___ Outcomes _Reading___ __________ ___________ Groups __1__ _2_ __1__ __2__ __1_ __2__ Mean _22__ __18_ _____ _____ ____ _____ SD _3.55 _4.22_ N at Posttest _____ _____ _____ _____ ____ _____ _____ _____ ____ _____ F value Chi-square _____ _____ _____ _____ _____ ____ _____ _____ ___ _____ _____ _____ ____ p value t value __________ __________ _____ _____ _____ _____ _____ _____ _____ _____37 Method of Analysis (pg _____) a. Intention to Treat: b. Treated Participants Only 38 Excerpt from Master Coding Guide for Study Coding ------------------------------------Code the Fantuzzo Study 39 Data Extraction Practice 40