Research Integrity: The Importance of Mentor/Mentee Relationships Tina L. Cheng, MD, MPH July 12, 2016 1 Tina L. Cheng • No Disclosures 7/12/2016 2 Outline • What is Research Integrity? • What is the Problem? • Research Misconduct – WHO? – WHAT? – WHEN/WHERE? – WHY? • As Mentors, HOW can we Ensure Research Integrity? Question #1 What is the most common cause of scientific publication retraction? A. Error B. Fraud or Suspected Fraud C. Duplicate Publication D. Plagiarism E. Other July 12, 2016 4 Question #2 • Your mentee is writing his paper analyzing data from your large study. He gives you a draft of the paper that includes the methods section of your previously published paper verbatim. He says that since the methods are the same he didn’t rewrite the methods paragraphs. • Do you agree? Why or why not? July 12, 2016 5 Question #3 In the PAST THREE ACADEMIC YEARS, how many times have you observed or had other direct evidence of researchers in your department (or equivalent organizational unit) allegedly committing research misconduct (falsification, fabrication, or plagiarism) in proposing, performing, or reviewing research, or in reporting research results? ____ Zero times ____ One time in the past three academic years ____ Two times in the past three academic years ____ Three times in the past three academic years ____ Four or more times in the past three academic years Question #4 • Your meticulous project coordinator, PC, has worked with you for three years on a randomized trial involving interviews with patients at baseline, 6 and 15 months. Your team reviews the patient tracking log recording completion of interviews at your weekly meeting. PC’s attendance becomes erratic and a covering research assistant tells you she cannot find copies of ten 6 month interviews that PC stated she had done. Calling of patients reveals that PC never completed the interviews that she recorded in the log as completed. What do you do now? July 12, 2016 7 Case • The Players: – Dr. Eugene Braunwald, mentor – Dr. John Roland Darsee, mentee – 47 researchers, co-authors over 3 years Wright DE et al. Sci Eng Ethics, 14:323-336, 2008 7/12/2016 8 Darsee Case • Occurred in the early 1980s • Scrutiny by Congress, press • Led to federal research misconduct regulations by the Public Health Service and the National Science Foundation • Office of Research Integrity (ORI) established – 800 formal cases in 18 years Wright DE et al. Sci Eng Ethics, 14:323-336, 2008 7/12/2016 9 Responsible Conduct of Research (RCR) Training • NIH and NSF guidelines require institutions receiving federal funds to provide ongoing training and education in RCR • The areas to be covered include: – – – – – – – – Data Acquisition and Management Responsible Authorship Responsible Peer Review Research Misconduct Mentoring Collaborative research Conflicts of Interest The scientist as a responsible member of society Office of Science & Technology, White House, 2000 Research misconduct is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results. • Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record. • Fabrication is making up data or results and recording or reporting them. • Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit. NIH and Research Integrity Research integrity includes: • the use of honest and verifiable methods in proposing, performing, and evaluating research • reporting research results with particular attention to adherence to rules, regulations, guidelines, • and following commonly accepted professional codes or norms. http://grants.nih.gov/grants/research _integrity/index.htm Johns Hopkins Medicine and Research Integrity • Includes the destruction, absence of, or accused person’s failure to provide research records accurately documenting the questioned research • Does not include honest error or honest differences of opinion Johns Hopkins Medicine and Research Integrity • Each of the following must be met to support a finding of research misconduct: – Significant departure from the accepted practices of the scientific community – Committed intentionally, knowingly, or recklessly – Allegation has been proven by a preponderance of the evidence http://www.hopkinsmedicine.org/Resea rch/OPC/Research_Integrity/ Outline • What is Research Integrity? • What is the Problem? • Research Misconduct – WHO? – WHAT? – WHEN/WHERE? – WHY? • As Mentors, HOW can we Ensure Research Integrity? Three Study Designs • Scientist Self Report Surveys of Misconduct • Scientist Surveys Reporting Observation of Misconduct • Review of Office of Research Integrity (ORI) Cases of Misconduct 7/12/2016 16 Scientist Self-Report Martinson et al Study, 2005 • Anonymous survey of 4160 early and 3600 mid-career scientists funded by NIH • Response rates: 43% Early career, 52% Mid career • 33% had engaged in at least one of the top ten behaviors during the previous three years Martinson B et al. Scientist Behaving Badly. Nature 435: 737-738, 2005 Scientist Self-Report Martinson et al Study 2005 Martinson B et al. Nature 435: 737-738, 2005 Scientist Self-Report Martinson et al Study 2005 Martinson B et al. Nature 435: 737-738, 2005 How Many Scientists Fabricate & Falsify Research? A Systematic Review & Meta-Analysis 2% of scientists admitted to have fabricated, falsified or modified data or results at least once Fanelli D. PLoS One 2009; 4:e5738 Scientist Observation & Reporting of Misconduct, Gallup Survey 2006 7/12/2016 21 Scientist Observation & Reporting of Misconduct, Gallup Survey 2006 • Anonymous random survey to 4298 principal investigators of NIH R01s at 4298 unique departments at 605 universities, institutes, hospitals • Response rate 51% 7/12/2016 22 Question In the PAST THREE ACADEMIC YEARS, how many times have you observed or had other direct evidence of researchers in your department (or equivalent organizational unit) allegedly committing research misconduct (falsification, fabrication, or plagiarism) in proposing, performing, or reviewing research, or in reporting research results? ____ Zero times ____ One time in the past three academic years ____ Two times in the past three academic years ____ Three times in the past three academic years ____ Four or more times in the past three academic years Scientist Observation & Reporting of Misconduct, Gallup Survey 2006 • 164 scientists (7.4% of sample) reported observing 201 incidents of meeting the federal definition of misconduct over 3 yrs • ~ 3 incidents/100 researchers/yr • If this rate applied to entire population of scientists supported by NIH (~155,000)= 2300-4600 incidents per year. • 58% reported to institutional officials 7/12/2016 24 How Many Scientists Fabricate & Falsify Research? A Systematic Review & Meta-Analysis Fanelli D. PLoS One 2009; 4:e5738 JHU SOM Misconduct Cases JHU School of Medicine • Approximately 2500 full time and 1300 part time faculty • Approximately 1500 graduate students • Approximately 1600 postdoctoral trainees (non house staff) • Approximately 900 house staff Question #1 What is the most common cause of scientific publication retraction? A. Error B. Fraud or Suspected Fraud C. Duplicate Publication D. Plagiarism E. Other July 12, 2016 28 Question #1 What is the most common cause of scientific publication retraction? A. Error B. Fraud or Suspected Fraud C. Duplicate Publication D. Plagiarism E. Other 21% 43% 14% 10% Fang FC et al. PNAS 2012;109;17028-33 July 12, 2016 29 Retractions in Scientific Publications Number of retracted articles by cause Fang FC et al. PNAS 2012;109; 17028-33 Retractions in Scientific Publications Percent of publications retracted Fang FC et al. PNAS 2012;109;17028-33 Retractions in Scientific Publications Fraudlent vs. Error • Journal Impact Factor was higher for fraudulent papers (p<.001) • Fraudulent papers were more likely to be written by a first author who had written other retracted papers than error retraction papers (53% vs 18%, p<.0001) • More fraud than error among retracted papers from the USA compared to the rest Steen RG J Med Ethics of the world (p<.05) 2011;37:113-117 Outline • What is Research Integrity? • What is the Problem? • Research Misconduct – WHO? – WHAT? – WHEN/WHERE? – WHY? • As Mentors, HOW can we Ensure Research Integrity? WHO had Misconduct? Gallup Survey 2006 Titus SL et al. Nature 2008;453:980-82 7/12/2016 34 WHO had Misconduct? ORI Case Reviews Fang FC et al. MBio. 2013 Jan 22;4(1):e00640-12 7/12/2016 35 WHAT Misconduct? Gallup Scientist Survey 2006 • Misconduct observed – 60% falsification or fabrication w or w/o plagiarism – 36% plagiarism only 7/12/2016 36 Question #2 • Your mentee is writing his paper analyzing data from your large study. He gives you a draft of the paper that includes the methods section of your previously published paper verbatim. He says that since the methods are the same He didn’t rewrite the methods paragraphs. • Do you agree? Why or why not? • If published, self plagiarism, copyright infringement July 12, 2016 37 WHAT Misconduct? Plagiarism • • • • Common! Plagiarism of ideas Plagiarism of text Self plagiarism Time Editor-at-Large, CNN host 7/12/2016 38 WHAT Misconduct? Self Plagiarism • Double dipping: redundant and duplicate publications • Salami slicing: data fragmentation, partitioning one study into multiple smaller publications • Copyright infringement Roig M. Avoiding plagiarism, • Text recycling self plagiarism and other 7/12/2016 questionable writing practices: a guide to ethical writing, http://www.cse.msu.edu/~alexli u/plagiarism.pdf 39 WHAT Misconduct? Gallup Scientist Survey 2006 • A post doc changed the numbers in assays in order to ‘improve’ the data. • A colleague duplicated results between three different papers but differently labelled data in each paper. • A co-investigator on a large, interdisciplinary grant application reported that post doc falsified data submitted as preliminary data in a grant. • A colleague used Photoshop to eliminate background bands on a western blot to make the data look more specific than they were. Titus SL et al. Nature 2008;453:980-82 WHAT Misconduct? Wright ORI Review of Trainee Misconduct • Closed ORI cases of trainee misconduct 19902004 • 33 post docs, 10 graduate students and 2 additional trainees • All but three cases involved either or both fabrication and falsification • 77% admitted to misconduct – most signed a Voluntary Exclusion Agreement with ORI precluding them from receiving federal funds for research from 3-5 years – 63% required retractions of publications – 41% fired or dismissed, 43% resigned Wright DE et al. Sci Eng Ethics 2008 14:323-336 WHEN/WHERE was the Misconduct? Wright ORI Review of Trainee Misconduct • How was it discovered? – 39% others could not replicate the data – 36% case witnesses – 25% found when researchers wanted to examine the source data and it could not be located. WHEN/WHERE was the Misconduct? Gallup Survey 2006 Titus SL et al. Nature 2008;453:980-82 7/12/2016 43 WHY did they do it? Wright ORI Review of Trainee Misconduct • Sociopathology • Increasing pressure on researchers • Arrogance, already knowing the right answer without bothering to do the experiment • Ignorance of research standards and ethical norms, poor mentoring WHY did they do it? • • • • Career advancement and promotion Competitiveness Laziness Ability to get away with it WHY did they do it? Wright ORI Review of Trainee Misconduct • 53% of cases described their stress levels as a factor that caused or contributed to their misconduct – 62% felt pressure to perform well – 38% felt time-related stress such as submitting a grant, publication or publication deadline or complete dissertation – 17% felt unreasonable pressure from the mentor to get desired or quick results WHY did they do it? Wright ORI Review of Trainee Misconduct “Even though I had already secured a position…and had 18 publications, an NIH fellowship and several awards for my prior work, I believed myself to be a complete failure as a scientist…I think that was going through my mind, had led me to believe that, if I could just show one piece of ‘promising’ data on a group meeting, my supervisor would let me continue working on the problem and produce real data that be presented and published…” WHY did they do it? Wright ORI Review of Trainee Misconduct “There was much excitement over this [surprising and promising preliminary result] and I began to feel a self-imposed pressure to keep the positive data coming in. It was at this time that I began to substitute buffer for the control SSS. At the time I realized I was making a grave error in judgment, but as the excitement over the results grew and grew I felt more pressure to manipulate the system…over time I lost more and more control and felt like I could not stop falsifying experiments.” WHY did they do it? • Association between scientific misconduct and perceptions of inequities in resource distribution processes Martinson B et al. Nature 435: 737-738, 2005 WHY did they do it? Scientist Self Report Survey • Being expected to obtain external funding • Receiving federal research funding • Private industry involvement were all associated with significantly higher reports of misconduct and neglectful or careless behaviors 7/12/2016 Martinson et al. Acad Med 2009;84:1491-99 52 WHY did they do it? Scientist Self Report Survey • “The free play of university and individual self-interests, combined with, and contributing to the intense competition for research funding, may be undermining scientific integrity.” Martinson et al. Acad Med 2009;84:1491-99 7/12/2016 53 7/12/2016 54 Outline • What is Research Integrity? • What is the Problem? • Research Misconduct – WHO? – WHAT? – WHEN/WHERE? – WHY? • As Mentors, HOW can we Ensure Research Integrity? Question #4 • Your meticulous project coordinator, PC, has worked with you for three years on a randomized trial involving interviews with patients at baseline, 6 and 15 months. Your team reviews the patient tracking log recording completion of interviews at your weekly meeting. PC’s attendance becomes erratic and a covering research assistant tells you she cannot find copies of ten 6 month interviews that PC stated she had done. Calling of patients reveals that PC never completed the interviews that she recorded in the log as completed. What do you do now? July 12, 2016 56 Johns Hopkins Medicine and Research Integrity • Any faculty member, trainee or staff employee of the School of Medicine who suspects that research misconduct has occurred has an obligation to report that suspicion to the director of the department or division affected, or to the Dean of the SOM. • If the report is made to the director of a department, the department director must report the allegation to the Dean or the Dean’s designee in a timely fashion, regardless of the department director’s assessment concerning whether the activity in question constitutes possible misconduct. http://www.hopkinsmedicine.org/Research /OPC/Research_Integrity/ Institute of Medicine, National Research Council Report 2002 Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct • Mentoring is key to producing responsible researchers 7/12/2016 58 HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct Mentoring and Research Misconduct: An Analysis of Research Mentoring in Closed ORI Cases • Did the mentor review source data? • Did the mentor teach specific research standards? • Did the mentor minimize stressful work situations? Wright DE et al. Sci Eng Ethics 2008 14:323-336 HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct • 73% mentors did not look at raw data • 62% mentors did not have set standards for recording data • 53% of trainees reported stress levels as a factor in misconduct HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct What the investigators said: “Mentor/PIs should provide a more formal process of initial training for their graduate students as they join a research project. This should include coverage of IRB regulations and the responsibility inherent in maintaining the integrity of research. The Board also recommends that [M/PI] should have more contact with the graduate students throughout the research project…” HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct What the investigators said: “Although outright fraud can circumvent virtually any review process, we believe that every laboratory head must take the responsibility to ensure that procedures are in place… so that the possibility of fraud is minimized. These include that a) every manuscript receive adequate review by senior members of the laboratory, that the PI is directly informed of the resulting criticisms, and that the PI reviews the final manuscript…and b) every effort should be made to provide opportunities for each investigator to present primary unedited data to an appropriate group or subgroup of the laboratory for criticism and feedback.” HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct What the investigators said: “The committee believes that it is good practice for the mentor to examine the primary laboratory notebooks for a student conducting his/her PhD thesis research in the mentor’s laboratory. More than just checking the validity of results, it helps the mentor better understand some of the details and nuances of the work which will help with both the thesis and the publications.” HOW to Ensure Research Integrity? Wright ORI Review of Trainee Misconduct • Regular review of trainee raw data • Standard setting • Attention to trainee stress levels HOW to Ensure Research Integrity? Research Integrity Measures • Survey of 6698 researchers with NIH grant support • 67% response rate 7/12/2016 Survey of Research Integrity Measures Utilized in Biomedical Research Laboratories. American Institutes for Research, 2003 66 HOW to Ensure Research Integrity? Research Integrity Measures • Data stored digitally (42%) or notebooks (39%) retained for mean of 12.9 years after publication • Lab meetings to discuss research 30x/yr (median) with each researcher presenting 6 (median) to 12 (mean) times a year • PI supervises five researchers spending 2 hrs/week with each Survey of Research Integrity Measures 7/12/2016 Utilized in Biomedical Research Laboratories. American Institutes for 67 Research, 2003 HOW to Ensure Research Integrity? Research Integrity Measures • Less than 5% of PIs utilized written guidelines for – authorship, reproducibility, prevention of fragmentation of publications, prevention of multiple submissions of the same manuscript to more than one journal simultaneously, promotion of sharing data, methods, reagents and other materials and for the proper handling of correction or retraction of publications 7/12/2016 Survey of Research Integrity Measures Utilized in Biomedical Research Laboratories. American Institutes for Research, 2003 68 HOW to Ensure Research Integrity? Gallup Survey 2006 • Scientist recommendations to prevent or detect misconduct – Close supervision including reviewing data, reproducing results, audit, quality control procedures – Place responsibility on the principal investigator – Open communication – Whistleblower protections 7/12/2016 69 HOW to Ensure Research Integrity? • Proactive quality assurance mechanisms – Review data collection forms, lab notebooks, computerized records – Double data collection and double data entry for a sample – Audio or videotape research encounters – Unannounced observations of research work 7/12/2016 70 HOW to Ensure Research Integrity? Redefine misconduct as distorted reporting • “any omission or misrepresentation of the information necessary and sufficient to evaluate the validity and significance of research, at the level appropriate to the context in which the research is communicated.” • Authors should be held accountable for what they write, and for recording what they did. • Writing guidelines for many biomedical techniques or study designs – EQUATOR Network (equator-network.org) – Minimum Information for Biological and Biomedical Investigations (mibbi.sourceforge.net). 7/12/2016 71 Fanelli D. Nature. 2013;494(7436):149 HOW to Ensure Research Integrity? Recommendations from DHHS ORI Six recommendations are made to institutions for fostering a culture of integrity: 1. adopt a zero tolerance for research misconduct 2. protect whistleblowers 3. implement a clear system for reporting alleged research misconduct 4. train mentors on establishing and maintaining research rules and minimizing opportunities to commit research misconduct 5. develop continuing mechanisms for reviewing and evaluating the research and training environments 6. model ethical behavior Titus SL et al. Nature 2008;453:980-82 7/12/2016 73 Mentors: C’s • Content mentor • Coaching mentor Career mentor • Critiquing/Challenging mentor • Connecting mentor • Cheerleader mentor • Zeldich’s Dimensions of Good Mentoring • “Advisors: people with career experience willing to share their knowledge • Supporters: people who give emotional and moral encouragement • Tutors: people who give specific feedback on one’s performance • Masters: employers to whom one is apprenticed • Sponsors: sources of information about and aid in obtaining opportunities • Models of identity: of the kind of person one should be…” 7/12/2016 Titus & Ballou. Sci Eng Ethics 6/3/2012 75 https://members.aamc.org/ eweb/upload/Compact%2 0Between%20Postdoctorl %202006.pdf https://www.aamc.org/download/49868/data/ gradcompact.pdf 7/12/2016 76 Compact Between Postdoctoral Appointees & Their Mentors (AAMC 2006) Commitment of Mentors (11 bullets) • I will ensure that a mutually agreed upon set of expectations and goals are in place at the outset and I will work with the postdoc appointee to create an individual career development plan. • I will strive to maintain a relationship that is based on trust and mutual respect. I acknowledge that open communication and periodic formal performance reviews…will help ensure that the expectations of both parties are met. • I will promote all ethical standards for conducting research including compliance with all institutional and federal regulations as they related to responsible conduct in research, privacy and human subjects research, animal care and use, laboratory safety, and use of radioisotopes. I will clearly define expectations for conduct of research in my lab and make myself available to discuss ethical concerns as they arise. 77 Compact Between Postdoctoral Appointees & Their Mentors (AAMC 2006) Commitment of Postdoctoral Appointees (11 bullets) • I acknowledge that I have the primary responsibility for the development of my own career. • I will perform my research activities conscientiously, maintain good research records, and catalog and maintain all tangible research materials that result from the research project. • I will respect all ethical standards when conducting my research including compliance with all institutional and federal regulations as they related to responsible conduct in research, privacy and human subjects research, animal care and use, laboratory safety, and use of radioisotopes. I recognize that this commitment includes asking for guidance when presented with ethical or compliance uncertainties and reporting on breeches of ethical or compliance standards by me and/or others. • At the end of my appointment, in accordance with institutional policy, I will leave behind all original notebooks, computerized files, and tangible research materials so that other individuals can carry on related research… 78 Compact Between Biomedical Graduate Students & Their Advisors (AAMC 2008) Commitment of Graduate Students (13 bullets) • I will participate in my institution’s Responsible Conduct of Research Training Program and practice those guidelines in conducting my thesis/dissertation research. • I will maintain a detailed, organized, and accurate laboratory notebook. I am aware that my original notebooks and all tangible research data are the property of my institution but that I am able to take a copy of my notebooks with me after I complete my thesis/dissertation. • I will discuss policies on authorship and attendance at professional meetings with my research advisor. 79 Compact Between Postdoctoral Appointees & Their Mentors (AAMC 2006) This compact serves both as a pledge and a reminder to mentors and their postdoctoral appointees that their conduct in fulfilling their commitments to one another should reflect the highest professional standards and mutual respect. 80 An Emerging Culture for Health Care Darrell Kirch, MD, AAMC, 2012 NIH and Team Science “The scale and complexity of today's biomedical research problems demand that scientists move beyond the confines of their individual disciplines and explore new organizational models for team science. Advances in molecular imaging, for example, require collaborations among diverse groups–radiologists, cell biologists, physicists, and computer programmers. NIH wants to stimulate new ways of combining skills and disciplines in the physical, biological, and social sciences to realize the great promise of 21st century medical research.” 7/12/2016 82 Research Integrity Challenges in Team Science • More team members • Rely on team members to know different content areas • Rely on team members to teach and supervise trainees • More “farming out” parts of research to others 7/12/2016 83 What Do Mentoring & Training in the RCR Have To Do With Scientists’ Misbehavior? • Survey of 1479 early-career (T32/F32 post doc fellows) and 1768 mid career scientists (initial R01) • There was NO consistent relationship between training in research integrity and behaviors that may compromise integrity of science Anderson MS et al Academic Medicine 82:853, 2007 Responsibilities of the Mentor • 56% of 3,257 researcher admitted inadequate monitoring of research projects because of work overload. (Anderson MS et al Academic Medicine 82:853, 2007) • Only 33% of 2,000 research lab directors who supervised on average 4.7 individuals had a mentor who had prepared them to be a good mentor. (Robhard D, Survey of research integrity measures utilized in biomedical research labs. ORI report 2003) Conclusion • Fostering an environment that promotes research integrity is important for public trust • A culture of research integrity is essential • Mentoring is a critical component to producing responsible researchers 7/12/2016 87 References • Previous Research Integrity Lectures: http://www.hopkinsmedicine.org/Research/OPC/Resear ch_Integrity/responsible_conduct_DeanRILectures_1 2_13.html. 7/12/2016 88 Panel Discussion • Arlene Butz, Sc.D. • George Dover, M.D. • Pamela Zeitlin, M.D., Ph.D. • Sheila Garrity, J.D., M.P.H., M.B.A. – Moderator 7/12/2016 89