CSC 426 Team Presentation
Week 8 Spring 2013
Jinghui CHENG
Jean HOP
Raj PARTHASARATHY
RESEARCH ETHICS
the rules of conduct recognized in respect to a
particular class of human actions or a particular
group, culture (dictionary.com)
Why You Should Care
1
1. Promote the aims of research
– Knowledge - Truth - Avoidance of Error
2. Promote values of collaborative work
– Trust – Accountability – Mutual Respect – Fairness
3. Help build public support for research
– Integrity – Quality
4. Promote important moral and social values
– Social Responsibility – Health – Safety
1 “What is Ethics in Research & Why is it Important?” David B. Resnik, NIH, May 1, 2011. www.niehs.nigh.gov
Why You Should Care
1. Promote the aims of research
– Knowledge - Truth - Avoidance of Error
2. Promote values of collaborative work
– Trust – Accountability – Mutual Respect – Fairness
3. Help build public support for research
– Integrity – Quality
4. Promote important moral and social values
– Social Responsibility – Health – Safety
F F P: The BIG Three + One
Falsification
Fabrication
Plagiarism
Misrepresentation
FALSIFICATION
Outright falsification rare but exists
• 2002 Bell Labs: Jan Hendrick Schon
• 2005 U of VT: Eric Poehlman
Falsification GREY Area
•
•
•
•
Culling of “spurious” data
Removal of badly behaved points
Decisions on controls and statistical tests
Etc
FABRICATION
Outright fabrication rare but exists
• 2002 Bell Labs: Jan Hendrick Schon
• 2005 U of VT: Eric Poehlman
• 2005 Seoul University Woo Suk Hwang
MISREPRESENTATION
• Present results in a way that inaccurately
reflects the outcome
• Understate the work of others
• Imply a higher confidence in results than is
merited
SPECIAL CASE: Online changes
Why You Should Care
1. Promote the aims of research
1
– Knowledge - Truth - Avoidance of Error
2. Promote values of collaborative work
1
– Trust – Accountability – Mutual Respect – Fairness
3. Help build public support for research
1
– Integrity – Quality
4. Promote important moral and social values 1
– Social Responsibility – Health – Safety
PLAGIARISM
Outright plagiarism rare but exists
• 2002 Historian Stephen Ambrose
• 2013 Science Journalist Jonah Lehrer
• 2013 German Education Minister
Annette Schavan
Plagiarism GREY Areas:
• Self Plagiarism
• Unconscious Plagiarism
SELF PLAGIARISM
Why can’t I reuse MY work?
1. Published or put out for publication
– Copyright on your EXPRESSION of your idea
2. Theoretically ideas become tired recycleds
“UNCONSCIOUS” PLAGIARISM
• Paraphrase of another’s work
• Unclear citation as to which parts of the work
are another’s
• Paraphrase of paper structure
• Citing same background literature
• Similar experiments
AUTHORSHIP
Included as authors but did not contribute
• Owed a favor
• Research team head
RELATIONSHIPS
Required for collaboration and just plain living
Power Imbalances
• Graduate Students
• Lab heads, Department Chairs
Sharing data/Confidentiality
Future Competition
MORE NOT ALWAYS BETTER!
What about “dangerous” information?
Example:
2012 Two papers published in Nature and Science
due to implications for terrorism.
Why You Should Care
1. Promote the aims of research
1
– Knowledge - Truth - Avoidance of Error
2. Promote values of collaborative work
1
– Trust – Accountability – Mutual Respect – Fairness
3. Help build public support for research
1
– Integrity – Quality
4. Promote important moral and social values 1
– Social Responsibility – Health – Safety
PUBLIC SUPPORT
Funding, funding, funding!
Examples
• West Nile Disease versus Valley Fever funding
• Polio funding
Why You Should Care
1. Promote the aims of research
1
– Knowledge - Truth - Avoidance of Error
2. Promote values of collaborative work
1
– Trust – Accountability – Mutual Respect – Fairness
3. Help build public support for research
1
– Integrity – Quality
4. Promote important moral and social values 1
– Social Responsibility – Health – Safety
CONFLICT OF INTEREST
Universities associated with BIG business
• 1974 Monsanto and Harvard
Human Testing (not us)
EXAMPLES
• Dr B’s Meat Example
• Pharmatec and U of FL
• Boots Pharmaceuticals and Betty Dong
• Apotex and Nancy Olivieri (U of Toronto)
ETHICS ISSUE: SORTING IT OUT
1.
2.
3.
4.
What is the problem or issue?
What is the relevant information?
What are the different options?
How do ethical codes or policies as well as legal
rules apply to these different options?
5. Are there any people who can offer ethical advice?
KEY IS TO BE ABLE TO JUSTIFY YOUR DECISION
ETHICS CHECKLIST
See pg 224 in text book for checklist
MEASUREMENT
Part 1: Construct Validity
Part 2: Reliability
Construct Validity
• One facet of measurement
• How do we take ideas or theories and
measure experimental results?
• How do we know we are measuring the right
things?
Construct Validity
• Operationalization: eliminate subjectivity by
by operationally defining concepts
• Operationally – define concepts by how you
will measure them
• Defining variables as MEASURABLE factors
OPERATIONALIZATION
• Example:
Variable
Factor
Age
Number of years and months
Gender
Male/Female/Other
Distance
Number of miles
• How well do we measure what we purport to be
measuring?
CONSTRUCT VALIDITY
• Definitionalist Perspective:
• Define construct so tightly as to define
operationalization at the same time
• Relationalist Perspective:
• Everything related; Not black and white
• Can’t really use operational definitions to
define constructs
CONSTRUCT VALIDITY
• Translation Validity
• Does the chosen operationalization reflect
the construct?
• Criterion-related Validity
• How will the operationalization perform?
CONSTRUCT VALIDITY: Translation
• Face Validity
• On its face, does it appear to be good?
• Subjective judgment call therefore weak
• Can strengthen by expert review/feedback
• Content Validity
• Check against content domain
• Checklist approach
CONSTRUCT VALIDITY: Criterion
Given what we know theoretically and using
operationalization, can we …
• Predict (Predictive)
• Distinguish between groups (Concurrent)
• Find similar results with theoretically similar
operationalizations (Convergent)
• Find dissimilar results with theoretically dissimilar
operationalizations (Discriminant)
ALL WE NEED IS BOTH!
Convergent AND Divergent
Convergent, Divergent – correlation analysis
How high is high?
How do we decide?
THREATS-NON SOCIAL
•Inadequate Preoperational Explication of Constructs
Concepts not well defined prior to measurement phase
•Mono-operation bias
Only one version of variable used limiting the breadth of the
experiment
•Mono-method bias
Only one method used to observe or measure a key
concept
•Interaction of treatment and testing
Testing becomes part of treatment and the interaction skews the
results
THREATS-NON SOCIAL
•Interaction of Different Treatments
Other reasons besides chosen treatment might be influencing
results
•Restricted Generalizability Across Constructs
Might be a situation of working only in this experiment
THREATS-SOCIAL
•Hypothesis Guessing
Participants respond based on what they think the goal of the
experiment is.
•Evaluator Apprehension
Knowledge of experiment affects participant
•Experimenter Expectancies
Researcher wants experiment to go a certain way
HANDLING CONSTRUCT VALIDITY
• Theoretical representation: Nomological Net
http://www.socialresearchm
ethods.net/kb/nomonet.php
HANDLING CONSTRUCT VALIDITY
• Application: Multitrait Multimethod Matrix
• Developed in the 1950s
• Not used in practice due to multi method
constraints
• Too much “subjective” judgement
• All about correlation
HANDLING CONSTRUCT VALIDITY
http://www.socialresearchm
ethods.net/kb/nomonet.php
Pattern Matching Correlation
• Overall estimate of construct validity
• Matches predicted, theoretical
• Compare concept map with observed pattern
http://www.socialresearchm
ethods.net/kb/nomonet.php
RELIABILITY
• Reliability = consistency = repeatability
RELIABILITY
•
Reliability = consistency = repeatability
•
True Score Theory says …
Observations = True Measurement + Random Error
var(X) = var(T) + var(eX)
– Variability due to variability in the True and Random
measures
RELIABILITY
• Across individuals – not related to individual
measure
• Can’t compute actual Reliability
var(T)
------var(x)
Because we do not know var(T)
•
So have to estimate
ESTIMATING RELIABILITY
• Can estimate
• Always between 0 and 1
•
.x means x percent attributable to true measurement and
(1 - .x) is attributable to error
TYPES OF RELIABILITY
•
Test-Retest Reliability
• Administer same test twice to same group of
individuals with a period of time between the tests
•
Parallel Forms Reliability
• Administer different versions of a test tool
TYPES OF RELIABILITY
•
InterRater Reliability
• Assess the degree to which different raters agree on
their assessment
•
Internal Consistency Reliability
• Assess consistency of results across items in the test
RELIABILITY AND VALIDITY
•
Can view as continuum
ON BEING A SCIENTISTRESPONSIBLE CONDUCT IN
RESEARCH
Presented by: Raj Parthasarathy
CSC 426 Spring 2013
May 28
ON BEING A SCIENTIST- RESPONSIBLE
CONDUCT IN RESEARCH
ON BEING A SCIENTIST-RESPONSIBLE
CONDUCT IN RESEARCH
• Free book/lengthy paper written by the ‘Committee on
Science, Engineering & Public Policy’, ‘National Academy of
Sciences’, ‘National Academy of Engineering’ & ‘Institute of
Medicine’ (1995).
• A treatise on do and don’ts in scientific research including
topics such as allocation of credit, and error and negligence.
•
Not possible to cover every detail in the available time today.
I have selected some of the most interesting and important
sections for presentation today.
Introduction
• What keeps the researcher rooted to their research?
• Pursuit of an experience so interesting that one cannot wait to
get up every morning and continue to work on it
• Ability to associate with colleagues that have made significant
contributions to the subject matter
• Knowing that their work may have a direct and impact on society
• Interest of others in the findings and implications of the research
SOCIAL FOUNDATIONS OF SCIENCE
• Scientific research is a social enterprise. It cannot be
done without drawing on the work of others or
collaborating with others.
• Of course, the object of research is to extend human
knowledge beyond what is already known.
• But such contribution enters the domain of science only
after it is presented to other researchers so that they can
independently judge its validity.
HOW IS VALIDITY JUDGED?
• Presentation at seminars and conferences, and to
scientific journals (which in turn send the papers to be
scrutinized by reviewers)
• After a paper is published or a finding is presented, it is
judged by other scientists in the context of what they
already know from their research/knowledge.
• After this continuum of discussion and deliberation
the ideas of individuals are eventually incorporated into
scientific worldview.
REVIEWS ARE IMPORTANT
• The process of review and revision (when a
paper is submitted to a journal, as we all
know) is critically important. It minimizes the
influence of individual subjectivity by
requiring that research results be accepted by
other scientists.
• Powerful inducement for researchers to be
critical of their own conclusions
SOME TOPICS WE HAVE DISCUSSED IN
CLASS
• We have already discussed the following topics in
class through presentations, and at length:
• Experimental techniques and treatment of data:
• moral of the story: methods for the treatment of
data and experimental techniques are NOT
infallible. Therefore researchers must be careful
in this regard, so as to assure the validity of the
results. Also need to be forthcoming about the
procedures used.
Values in Science
• Values in science:
• Scientists must display curiosity, intuition, creativity
• Some religious values, social beliefs and personal
beliefs may distort the work of researchers. For
example, the field of eugenics used the techniques of
science to try to demonstrate the inferiority of certain
races.
• Despite such cautionary episodes, it is clear that
values cannot—and should not—be separated from
science.
CONFLICTS OF INTEREST
• Sometimes values conflict. For example, a particular
circumstance might compromise—or appear to compromise—
professional judgments.
• One example: A scientist might receive a manuscript or
proposal to review that discusses work similar to but a step
ahead of that being done by the reviewer.
• Virtually all institutions that conduct research and many
scientific journals have policies and procedures for
managing conflicts of interest. These policies and procedures
are designed to protect the integrity of the scientific process
CONFLICTS ON INTEREST
• Interesting case:
• John, a third-year graduate student, is participating in a department-wide
seminar where students, postdocs, and faculty members discuss work in
progress. An assistant professor prefaces her comments by saying that the
work she is about to discuss is sponsored by both a federal grant and a
biotechnology firm for which she consults. In the course of the talk John
realizes that he has been working on a technique that could make a major
contribution to the work being discussed. But his faculty advisor consults
for a different, and competing, biotechnology firm.
• How should John participate in this seminar?
• What, if anything, should he say to his advisor-and when?
• What implications does this case raise for the traditional openness and
sharing of data, materials, and findings that have characterized modern
science?
PUBLICATION AND OPENNESS
•
Until the latter half of the seventeenth century. many scientists sought to
keep their work secret so that others could not claim it as their own.
Prominent figures of the time, including Isaac Newton, were loathe to convey
news of their discoveries for fear that someone else would claim priority—a
fear that was frequently realized.
•
Henry Oldenburg (the secretary of the Royal Society of London) won over
scientists by guaranteeing rapid publication and also pioneered the practice of
sending submitted manuscripts to experts who could judge their quality (today
known as the practice of peer review).
•
First to publish a view or finding, not the first to discover it, tends to get
most of the credit for the discovery. Once results are published, they can be
freely used by other researchers to extend knowledge. But until the results
become common knowledge, people who use them are obliged to recognize the
discoverer through citations. In this way scientists are rewarded through peer
recognition for making results public.
PUBLICATION AND OPENNESS
•
During the initial stages of research, a scientist deserves a period of privacy in
which data are not subject to disclosure.--allows researchers to advance their work to the
point at which they have confidence both in its accuracy and its meaning.
•
After publication, scientists expect that data and other research materials will be
shared with qualified colleagues upon request. At this stage, a scientist who is
unwilling to share research materials with qualified colleagues runs the risk of not being
trusted or respected.
•
Publication in a peer-reviewed journal remains the standard means of disseminating
scientific result. Posters, abstracts, lectures at professional gatherings, and proceedings
volumes are being used more often to present preliminary results before full review.
•
It is inappropriate for a scientist to release important and controversial results
directly to the public before submitting them to the scrutiny of peers. If the
researcher has made a mistake or the findings are misinterpreted by the media or the
public, the scientific community and the public may react adversely. Also, dangers in
application.
ALLOCATION OF CREDIT
• Need to follow the principle of fairness : credit is explicitly
acknowledged in three places: in the list of authors, in the
acknowledgments, and in the list of references or citations.
Conflicts over proper attribution can arise in any of these places.
• Failure to cite the work of others can give rise to more than
just hard feelings. Citations are part of the reward system of
science. They are connected to funding decisions and to the
future careers of researchers.
• MUST GIVE CREDIT WHERE AND WHEN IT IS DUE.
ERROR AND NEGLIGENCE
•
All scientific results must be treated as susceptible to error—researchers
are not perfect. An author who is willing to take credit for a paper must
also bear responsibility for its contents (example, when errors are found in
the paper).
•
Even the most responsible scientist can make an honest mistake. When
such errors are discovered, they should be acknowledged, preferably in the
same journal in which the mistaken information was published. Scientists
who make such acknowledgments promptly and openly are rarely condemned
by colleagues.
•
Mistakes made through negligent work are treated more harshly. If
scientists cut corners for whatever reason, they are placing their reputation, the
work of their colleagues, and the public's confidence in science at risk. (This
explains the time it takes to translate research work into a publishable paper in
a peer reviewed journal!)
MISCONDUCT
•
Misconduct goes beyond error and negligence—involves deception.
•
Examples: Making up data or results (fabrication), changing or misreporting
data or results (falsification), using the ideas or words of another person
without giving appropriate credit (plagiarism)
•
Misconduct strikes at the heart of the values on which science is based.
Anyone who engages in any of these practices is putting his or her scientific
career at risk. Even infractions that may seem minor at the time can end up
being severely punished.
•
Misconduct in science is unlikely to remain internal to the scientific
community. Its consequences are too extreme: it can harm individuals outside
of science (as when falsified results become the basis of a medical treatment), it
squanders public funds, and it attracts the attention of those who would seek to
criticize science. As a result, federal agencies, Congress, the media, and the
courts can all get involved.
RESPONDING TO VIOLATIONS OF ETHICAL STANDARDS
•
Violations of ethical standards must be taken very seriously.
•
In most cases the best thing to do is to discuss the situation with a trusted friend or
advisor. In universities, faculty advisors, department chairs, and other senior
faculty can be invaluable sources of advice in deciding whether to go forward
with a complaint. Rash complaints without basis can break the career of a
sincere researcher.
•
An important consideration is deciding when to put a complaint in writing. Once in
writing, universities are obligated to deal with a complaint in a more formal manner
than if it is made verbally. Putting a complaint in writing can have serious
consequences for the career of a scientist and should be undertaken only after
thorough consideration.
•
Many universities and other research institutions have designated an
ombudsman, ethics officer, or other official who is available to discuss situations
involving research ethics. Such discussions are carried out in strictest confidence
whenever possible.
THE SCIENTIST IN SOCIETY
• In addition to everything discussed in this
presentation, scientists have additional
responsibilities to society.
• Even scientists conducting the most fundamental
research need to be aware that their work can
ultimately have a great impact on society.
Construction of the atomic bomb and the
development of recombinant DNA two examples of
how seemingly arcane areas of science can have
tremendous societal consequences.
GUIDELINES ON RESEARCH
PRACTICE IN COMPUTER SCIENCE
Compiled by Justin Zobel, Department of
Computer Science, RMIT University,
Melbourne, Australia
MAY 1999
Presented by: Raj Parthasarathy
Introduction
• Developed by the major organizations responsible
for research and science in Australia, such as the
AVCC (The Australian Vice-Chancellor’s
Committee) and NHMRC (National Health and
Medical Research Council).
• Principal topics: Data Storage and Retention,
Authorship, Publication, Supervisor of Students,
Disclosure of Potential Conflicts of Interest, and
Research Misconduct.
Introduction
• Some topics repeated from what I presented in
the previous document presented today (viz.
“on being a scientist-responsible conduct in
research”).
• Therefore in the interests of time I will discuss
only those that are new or different and will
not repeat what was already discussed.
Principal Recommendations
•
Research workers should only participate in work which conforms to
accepted ethical standards and which they are competent to perform.
•
The minimum requirement for authorship of a publication is participation in
conceiving, executing or interpreting a significant part of the outcomes of the
research reported.
•
Supervisors must not publish a student's work without giving appropriate
credit (usually authorship) to the student.
•
Researchers should not referee a paper or examine a thesis where there is a real
or perceived conflict of interest, or where there is some reasonable likelihood
that it will be difficult for the referee to maintain objectivity.
Data storage and retention
• 2.1 Data (including electronic data) must be recorded in a
durable and appropriately referenced form.
• 2.2 The department or research unit must establish
procedures for the retention of data and for the keeping of
records of data held.
• 2.3 Data must be held for sufficient time to allow
reference. The minimum period for retention is at least 5
years from the date of publication.
• 2.4 Wherever possible, original data must be retained
Authorship, Publication & Misconduct
• All co-authors of a publication must acknowledge their authorship in
writing in a “signed statement of authorship”, which must specify that
the have seen the version of the paper submitted for publication.
• Publications must include information on the sources of financial
support
• Researchers shall report relevant negative results as well as positive
results.
• Interference : A researcher or reviewer shall not intentionally damage
any research-related property of another (becomes scientific
misconduct)
ACM (Association for Computing
Machinery) Policy, Code of Ethics
and Professional Conduct Guidelines
Presented by: Raj Parthasarathy
Introduction
• Relates to moral imperatives and professional
responsibilities that members and researchers
may face in their daily lives.
• Since many are repeats from the previous two
documents presented today, I will discuss only
what is new or different (in the interests of
time).
Moral Imperatives & Professional
Responsibilities
• Be fair and take action not to discriminate in any manner.
• Give proper credit for intellectual property (Specifically, one must
not take credit for other's ideas or work, even in cases where the work
has not been explicitly protected by copyright, patent, etc.)
• Give comprehensive and thorough evaluations of computer systems
and their impacts, including analysis of possible risks (especially,
any signs of danger from systems must be reported to those who have
opportunity and/or responsibility to resolve them).
Moral Imperatives & Professional
Responsibilities
• Improve public understanding of computing and its consequences
(Computing professionals have a responsibility to share technical
knowledge with the public by encouraging understanding of
computing, including the impacts of computer systems and their
limitations. This imperative implies an obligation to counter any false
views related to computing).
• Access computing and communication resources only when
authorized to do so (No one should enter or use another's computer
system, software, or data files without permission. One must always
have appropriate approval before using system resources, including
communication ports, file space, other system peripherals, and
computer time).
Leadership Imperatives & Code
Compliance
• Articulate social responsibilities of members of an organizational
unit and encourage full acceptance of those responsibilities
(Research must serve public interest and thus fulfill social
responsibility).
• Design and build information systems that enhance the quality of
working life (When implementing a computer system, organizations
must consider the personal and professional development, physical
safety, and human dignity of all workers).
• Create opportunities for members of the organization to learn the
principles and limitations of computer systems
Summary & Concluding Comments
• Being a researcher is a complex role involving ethics, moral behaviors, and
legal and societal responsibilities.
• The research system exerts many pressures on beginning and experienced
researchers alike. Principal investigators need to raise funds and attract
students. Faculty members must balance the time spent on research with the
time spent on teaching. Industrial sponsorship of research introduces the
possibility of conflicts of interest.
• Ultimately, a successful researcher will be one that incorporates a judicious
mix of all of the above elements in his/her day to day activities, while at the
same time making significant contributions to advance the body of
knowledge in his/her chosen field of research.
• THANK YOU FOR LISTENING.
Record Keeping in Experiments
Jinghui Cheng
5/21/2013
• Zobel, J. (1998, February). Reliable research:
Towards experimental standards for computer
science. In Proceedings of the Australasian
Computer Science Conference (pp. 217-229).
Springer-Verlag, Perth, Western Australia.
An Important Ethical Standard
• The Office of Research Integrity
– Integrity and Misconduct in Research: Report of the Commission on Research
Integrity
– http://ori.hhs.gov/images/ddblock/report_commission.pdf
• NH&MRC/AVCC
– Statement and Guidelines on Research Practice
– http://www.nhmrc.gov.au/guidelines/publications/r0024
• American Psychological Association
– Record Keeping Guidelines
– http://www.apa.org/practice/guidelines/record-keeping.pdf
• National Institutes of Health
– Guidelines for Scientific Record Keeping in the Intramural Research Program
– http://sourcebook.od.nih.gov/ethic-conduct/recordkeeping.pdf
NIH Guidelines for
Scientific Record Keeping
• Elements of Good Laboratory Notes
–
–
–
–
–
–
–
–
–
–
Who did it (the person making the record).
What you did.
When you did it (clearly stating month, date and year).
Why you did it.
What project the research was part of.
How you did it (including the methodology).
What materials were used.
The findings.
Your interpretation.
The next step.
Why Keeping Records?
• Fundamental motivations:
– It provides evidence of precedent.
– It is the only evidence resulting from an
experiment.
– It can be re-used in the future.
• Additional motivations:
– rigor, elucidation, reproduction, verification
Recordkeeping in Computer Science
• Depend on the type of research
– Meet the needs of keeping records
– Ensure that the records themselves are verifiable
• Principle:
– To provide corroborating testimonies
• Elements:
– notebooks, code, logs
Workshop
• What records to keep when…
Workshop
• What records to keep when…
– Scenario 1:
– You proposed a new face recognition
algorithm. It could be used in a
photo organizing applications (e.g.
iPhoto) to group photos by person.
You plan to use 1,000 images to
evaluate your algorithm against two
baseline ones. You want to compare
their efficiency (run time) and
effectiveness (false positive and false
negative error rates).
Workshop
• What records to keep when…
– Scenario 2:
– You are evaluating the usability of
cellphone keyboards. You plan to ask
30 participants to input a sentence
on iPhone and a Blackberry phone
(counterbalanced). You want to
compare the efficiency (time on
task), effectiveness (error rate), and
user satisfaction (using a
questionnaire) of two kinds of
keyboards.
Special Issues with Human Subjects
• The Belmont Report
– Respect for Persons
• “the requirement to acknowledge autonomy and the
requirement to protect those with diminished
autonomy”
– Beneficence
• “(1) do not harm and (2) maximize possible benefits
and minimize possible harms.”
– Justice
• “equals ought to be treated equally”
•
http://www.hhs.gov/ohrp/policy/belmont.html
Special Issues with Human Subjects
• When Keeping Records
– Be careful about privacy and confidentiality
• Informed Consent
– Illinois law prohibits the recording of
conversations without the consent of everyone
who is a party to the conversation
• Institutional Review Board (IRB)
– A review committee established to help protect
the rights and welfare of human research subjects
– http://research.depaul.edu/IRB/IRB_Home.html
Thank you!!
• Questions?
• Comments?