Methods for Creating Reliable and Reproducible Analyses Jennifer Blackford, PhD Assistant Professor of Psychiatry VKC Stats & Methodology Core Training Coordinator Goals • Introduce a new idea • Persuade you that reproducible analyses are necessary • Convince you that achieving reproducible analyses is possible • Provide the necessary tools for you to begin creating reliable and reproducible analyses Outline 1. 2. 3. 4. 5. 6. Reliability in research methods Reliability in analyses Dangers of point and click software What are “reproducible results”? Barriers to reproducible results Methods for creating reproducible analyses 7. Practical implementation Reliability in Research Methods What is reliability? re·li·a·ble [ri-lahy-uh-buh l] adjective that may be relied on; dependable in achievement, accuracy, honesty, etc.: reliable information. From Random House Dictionary How do we ensure reliability of our research methods? Examples? How reliable is “reliable enough”? a) b) c) d) 70% 80% 90% 100% What if the method is not reliable? a) b) c) d) use it anyway find a different method revise method to increase reliability develop a new method Reliability in (Statistical) Analyses Scientific Method Steps 1. 2. 3. 4. 5. Generate hypothesis Perform experiment and collect data Analyze data Interpret data and draw conclusions Publish results What does “analysis” include? Statistical Analysis • Data entry • Data cleaning – Data entry errors – Missing data – Recoding • Data preparation – Checking data distributions – Handling outliers – Transforming variables – Creating new variables • Performing statistical analyses • Recording results • Creating graphs Neuroimaging Analysis • Image file pre-processing – – – – Convert image files to standard format Correct head motion within and across trials Align structural and functional image Transform images into a “standard” space • Perform statistical analyses – Input experimental design – Assign event-timing information to image files – Run statistical analyses • Record Results • Create Graphs Should we expect the same reliability (or reproducibility) from our analytic methods as we do from our other research methods? a) No, analyses aren’t really methods b) No, analytic methods are too complex c) Maybe, but everything is fine, no need to change methods d) Yes, all parts of the scientific process should be reliable What are the “common” practices? • Enter data in Excel • Use Excel for data cleaning, etc • Generate descriptive statistics (counts, means, etc) in Excel • Import data into SPSS for analysis • Use point-and-click options to run statistical analyses • Type data back into Excel to create graphs Now let’s evaluate reliability…. • Enter data in Excel • Use Excel for data cleaning, etc • Generate descriptive statistics (counts, means, etc) in Excel • Import data into SPSS for analysis • Use point-and-click options to run statistical analyses • Type data back into Excel to create graphs The Dangers of Point-and-Click Software Excel • • • • Data entry inconsistencies and errors No record of what steps were performed Sorting dangers Data export issues – There is a bug in how Windows processing Excel files which results in missing data being exported. Cryer (2001). Problems using Microsoft Excel for Statistics. “Spreadsheet from Hell” Dan Byrne Excel—Statistical Issues • Missing data are handled inconsistently and sometimes incorrectly • Excel uses poor algorithms to find the standard deviation • Excel does not treat tied observations correctly when ranking • Regression is especially problematic-computations are often erroneous due to multiple incorrect procedures Cryer (2001). Problems using Microsoft Excel for Statistics. Other Point-and-Click • There are also dangers with other programs that offer user-friendly pointand-click capabilities, such as SPSS and SAS. • Neuroimaging analysis programs, like Brain Voyager and SPM have the same issues too. Has this ever happened to you….. Vignette #1 You sit down to finish writing your manuscript. You realize that you need to clarify one result by running an additional analysis. First, you re-run the primary analysis……but the primary results don’t match what you have in your paper. Vignette #2 You get an email request from the graduate student of a colleague at another university. She has conducted an experiment similar to one you published five years ago. She has a question and would like for you to run an analysis on a subset of subjects. When you go to the data folder, you find thirty program files and can’t remember which one has the analysis. Where is that final dataset???? Would you like to increase the reliability of your analytic methods? a) No need, my current methods are sufficient. I’m finding significant results already. b) Maybe, but I think it will be hard and I don’t have the energy for something new. c) Yes! Show me how, I’ll do anything. What are “reproducible results”? Reproducible Results • Some consider the use of statistical methods that can not be replicated as a breach of research ethics. • There is a new movement in statistics called “reproducible results” or “literate statistical programming”. Claerbout’s Principle “An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.” Note: The same “Principle” obviously applies to tables, standard errors, and so on. deLeeuw, J (2001). Reproducible Results: The Botton Line. Literate Statistical Practice Literate Statistical Practice (LSP, Rossini, 2001) describes an approach for creating self-documenting statistical results. It applies literate programming (Knuth, 1992) and related techniques in a natural fashion to the practice of statistics. In particular, documentation, specification, and descriptions of results are written concurrently with writing and evaluation of statistical programs. Rossini & Leisch (2003). Literate Statistical Practice. Barriers to Reproducible Results Barriers What are some barriers to “reproducible results”? 1. Awareness We didn’t even know there was a problem. 2. Knowledge What are the right methods? 3. Change You mean I wasn’t doing the right thing? 4. Effort It’s too hard. I don’t have time. I don’t have the energy to learn something new. Benefits 1. Reliable Results The results you produce today will be the same results you will produce tomorrow. 2. Time Savings/Efficiency In the (not so) long run, you will save much time. 3. Flexibility When you don’t “point-and-click” you gain many new analytic options. 4. Institutional (Lab) Memory Baby Steps We’ve already addressed the first barrier Awareness Now, we’ll address the second barrier Knowledge Methods for Creating Reproducible Analyses Caveats 1. This will take discipline. – On the front end, it does take more time to create a system or structure for your analysis. It is much faster to use Excel to do one quick analysis. 2. You might have to learn something new. – This can be a major barrier. Learning takes a commitment of time and effort. Overarching Principle: Organization • Organization for everything is key. • This principle is as true for your research study as for your data analysis. – electronic folder structure – file naming conventions – analytic programs • Things can get out of control.* Data Entry • Enter your data into a stable structure. • Good examples of stable structures – a relational database (like Microsoft Access) • VKC provides database development • VUMC REDCap (Research Electronic Data Capture) https://www.mc.vanderbilt.edu/starbrite/ – SPSS or SAS tables Data Cleaning and Preparation The second step is to create methods or procedures for documenting every step How-To Guides http://www.icpsr.umich.edu/access/dataprep.pdf How-To Guides By Fred Davidson, 1996 What About Data Analysis? Now for the scary word…. programming Which statement best describes you? a) I have never programmed before. b) I have programmed a little….10 years ago. c) I have some basic programming skills (and feel comfortable learning more) d) I am a code junkie! A code example Final Product • The final product is a single file that includes all of the code necessary to reproduce all of the analyses. • The code file should be heavily commented so that another reader can follow the workflow. Note: Ultimately, the file will also print out text and graphics. Specifically… • Data import • Data cleaning/formatting/manipulation – You must have an audit trail of every change made to the data. • Descriptive statistics • Inferential/predictive statistics – Include all of the workflow steps, including analyses that didn’t work or that you don’t keep so that you’ll remember what you did later Practical Implementation An illustration using SPSS SPSS • Starting simply, SPSS provides a relatively easy way to create a document that keeps track of all of your statistical steps. • You can begin with point-and-click, saving the underlying code, while progressing to programming. • SPSS journaling (edit->options->session journal, record syntax in journal) can track your syntax selections SPSS Example Protect Yourself • Track your changes – version control software (like Subversion) tracks of each iteration of document or program • RCS software: http://www.componentsoftware.com/csrcs/ – Screen capture software can capture your mouse movements: SnagIt (www.techsmith.com) • Keep prior operating systems – With Microsoft Virtual PC you can have other operating systems on your computer • http://www.microsoft.com/windows/products/winfamily/virtual pc/default.mspx Jim Steiger Protect Yourself • Back up, back up, back up – Disk space is cheap, your time is not – Amazon S3 “storage for the internet” – Jungle Disk http://www.jungledisk.com/ • Acronis True Image Home – www.acronis.com – Note: first, go to Computer Management and make certain that the Volume Shadow Copy service starts automatically before you install the software – make several copies of the Emergency Boot Disk, because you can boot into a Windows system that will not load windows and restore the whole hard drive (or just the system settings) from OUTSIDE WINDOWS. Jim Steiger Future Steps • SAS Enterprise Guide • With programs like R and Sweave/LaTex, you can incorporate the entire text of your report with results and graphics • Upcoming Seminar – Terri Scott – May 16, 2008.