SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 1 of 11
Title:
Longitudinal Case Study Evaluation of Graphical User Interfaces
Principal Investigators
XX
Abstract
Graphical user interfaces have radically changed the way people access data on the
personal computers, cell phones or the internet. Information visualization now allows
users to find patterns, identify outliers and make discoveries. In our research we are
developing novel information visualization techniques and interfaces.
The reports of usability studies and controlled experiments are helpful to understand the
potential and limitations of our tools, but we need to consider other evaluation
approaches that take into account the long exploratory nature of users’ tasks, the value of
potential discoveries or the benefits of overall awareness of changes in the data. We also
need reports of successful adoption and demonstrated utility.
We seek to recruit domain expert collaborators (our “study participants”) who will be
involved in long-term studies to study how they utilize our interfaces. The study will
mostly be qualitative. We seek to demonstrate the utility and effectiveness of our
interface by collecting evidence of success due to usage from our collaborators. These
collaborators will first undergo training sessions on how to use the advanced interface.
The interface will then be given to our collaborators for use in their professional
environments. Periodic meetings to discuss the usage, utility, satisfaction (or lack
thereof) of the interfaces will be conducted. The meetings can be face-to-face,
teleconferencing, or web meetings. In addition, a history of their actions using the
interface will be automatically logged on their own computers. The logs will be accessed
by the investigators only with the permission of our collaborators. The collaborators will
fill out an exit questionnaire at the end of study to assess their subjective experience with
the interface. Each participant in those studies will work with us for about 4-8 weeks.
Subject Selection
During this project, we will recruit adult volunteer professionals (e.g. analysts,
journalists, professors, medical researchers, etc.). They will be expert in their domain,
and have some use of our interfaces. They will recruited vis personal emails, among
people who contact us in person or by email to ask information about the tools we are
developing. No specific characteristics (e.g. gender, age, ethnicity, etc.) are required of
the participants. It is important to note that we see our subjects are collaborators who
are interested in using advanced interfaces for their professional work. We seek to
observe how they use our interfaces and how our advanced interface design help them in
their daily professional work. There will be no monetary compensation. Instead, we will
offer co-authorship to our collaborators in the papers reporting our results in scientific
1
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 2 of 11
publications. There will be no obligation for our collaborators to publish with
identifiable information, or to publish at all. At any time, our collaborators can choose to
terminate the study and collaboration with the investigators.
Because of the long-term, time-consuming, and detailed nature of this type of qualitative
study, we intend to recruit only 3-6 participants per year.
Procedures
Our collaborators will first sign a consent form (sample attached), and receive a training
of the interface. Collaborators will be informed of the type of data that is being collected
prior to giving consent.
Once the investigators feel that collaborators are trained and are competent in using the
interface, a copy of our software will be given to the collaborator for him/she to install in
their professional environment. Depending on the work environment, the investigators
may ask to observe collaborators in their work environments. The investigators and the
collaborators will schedule regular meetings (weekly or bi-weekly depending on the
participant task and availability) to discuss problems, possible discoveries, or frustrations
in using the interface in collaborators’ working environment. The regular meetings may
be recorded in audio. To facilitate these meetings, collaborators will be ask to write
down their usage experiences in a journal. The goal is to obtain anecdotes of success and
of failure when the advanced interface is put to use in a real environment.
While in use, the interface will automatically log the usage (e.g. submitted queries,
browsing strategies used) of a collaborator on the collaborator’s computer. Note that no
personal or identifiable information will be collected, only the actions performed within
the interface are logged. The results of the logging are stored in the collaborators
computer, and the investigators are only to access the log with the collaborator’s
permission. The location of the log is made known to the collaborators, so they can
remove or edit the log anytime they wish.
During any part of the study, the collaborators are encouraged to ask questions about the
interface or the study. The collaborators may choose to stop and terminate the study.
Since the goal of the study is to collect narratives of the usage of the interface in a realworld setting, and progress in understanding data can take days or weeks, the entire study
is planned for 4 to 8 weeks. Each session in the regular meeting are expected to take
about 1 hour each. Each collaborator can choose to spend as much time as they wish
with the interface in their work.
There will be no monetary compensation to our collaborators. Instead, summary
narratives collected from the study may be published in scientific publications and the
collaborators are offered to be co-authors. Collaborators are free to choose whether to be
included as co-authors, or to have their portion of the study published. No information
about the study or the collaborator will be published without the collaborator’s consent.
There is also no obligation to publish or to be a co-author. At any time prior to any
2
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 3 of 11
publication of a scientific paper, a collaborator may choose to withdraw from being a coauthor, or to refuse the publication of certain portions of the paper.
The Informed Consent Form is attached.
Risks and Benefits
There are no risks involved in using the interface. The study ask collaborators to use their
own personal computer running our novel interface. Our collaborator may rest at any
time, or quit at any time. While using the interface in their work environment, the
collaborators are not forced to use the interface for a minimum amount of time. If the
collaborator decides not to use the tool, we will interview them to ask them why (i.e.
what the tool not useful? Not reliable? Too hard to use? Other reasons?) Unless the
participant expressly choose to be mentioned by name in the papers we write their name
will not appear anywhere in our writings. Participants will be provided copies of the
papers before they are submitted for publication if the paper mentioned the study they
participated in.
Confidentiality
All results will be given to our collaborators prior to making them available or published.
The collaborators can refuse the publication of certain personal or identifiable
information, their narratives, their logs, or any combination of them. Since the
collaborators are not receiving monetary compensation, we do not collect social security
numbers or any other private information.
The logged data from collaborators’ actions using the interface will be stored only in each
collaborator’s computer. The location of the log is shown to the collaborators, and they
are made aware that they may choose to delete the log at any time. Furthermore, the logs
are only to be accessed by the investigators with the corresponding collaborator’s
permission.
E-mail addresses obtained from volunteering collaborators will be kept during the study
in the investigators regular email system.
Interviews will be conducted in the setting chosen by the collaborators (i.e. their own
office, or the PIs’ office) and they will choose the degree of privacy they need to report
on their work with the tool.
Information and Consent Forms
Subjects will be informed about the purpose of the research. The consent form will be
presented and signed before the study starts. All participants will receive a copy of the
consent form for their records. The procedures will be explained before the study, and
subjects can ask any questions they wish.
During the length of the study, our
collaborators are encouraged to ask questions and to communicate with us. At the end of
the study an exit questionnaire will be provided, including an additional consent form
section qith more specific questions about how we can report on their work (see Exit
3
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 4 of 11
questionnaire). A copy of the exit questionnaire and consent form will be provided to the
participants.
Conflict of Interest
No potential conflict of interest.
HIPAA Compliance
N/A
Research Outside of the United States
N/A
Research Involving Prisoners
N/A.
4
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 5 of 11
Longitudinal Case Study Evaluation of Graphical User Interfaces
Informed Consent Form
DESCRIPTION: You are invited to participate in a research study in data visualization graphical user
interfaces. Name of the particular interface(s) to be used:
PROCEDURE: After you fill a questionnaire and participate in a training session you will use the interface
in your normal tasks with your own data. We will meet with you at regular intervals chosen by you to
document your use of the tool. Your computer interactions may be logged and saved on your own
computer for later data analysis.
RISKS AND BENEFITS: There are no known risks associated with participation in this study. The
expected benefits are better human computer interaction and information visualization techniques. This
research is not designed to benefit you personally. However, you may be acknowledged in our papers
and presentations, and your use of the tool describe. You have the right to refuse being identified by
name or to have the result of your analysis described in our papers.
TIME INVOLVEMENT: This study will typically take 4 – 8 weeks. The initial training session will take
2 to 4 hours. Each subsequent meeting should take about an hour. Other than these, you are asked to
use the interface naturally in your own work environment during work time. The actual committed time
will depend on how much you use the interface.
PAYMENTS: You will receive no monetary payments for your participation.
SUBJECT'S RIGHTS: If you have read this form and have decided to participate in this study, please
understand that your participation is voluntary and you have the right to withdraw your consent or
discontinue participation at any time without penalty. You have the right to refuse to answer particular
questions, and you may ask any question you wish. Your individual privacy will be maintained in all
published and written data resulting from the study. If you have questions about your rights as a study
participant, are dissatisfied at any time with any aspect of this study, or wish to report a research-related
injury, please contact - anonymously, if you wish – the Institutional Review Board Office, XXXX
YOUR IRB INFO HERE XXX
I state that I am over 18 years of age and wish to participate in a program of research being conducted by
XXX (xxx@email) at the University of XXX.
I understand that if the results of the study are to be published, I have the right to choose to be identified by
name or to have the results of my work described:
(please initial): __ Yes __ No
I give consent to have my name and institution mentioned (as partnering in an ongoing case study) in
HCIL presentations about the tool being evaluated. I am aware that can withdraw this authorization at any
time by sending a written request to the PIs.
(please initial): __ Yes __ No
I give consent to be recorded by audio or video during the meeting portions of the study:
(please initial): __ Yes __ No
I give consent for my actions to be logged when using the interface for research purposes (I am told where
the log is, and how to delete it if I wish to):
(please initial): __ Yes __ No
I have reviewed the entry and exit questionnaires:
(please initial): __ Yes __ No
SIGNATURE _____________________________ DATE ____________
Name (PRINT) ____________________________ Institution (Print)__________________________
Approval Date:
Expiration Date:
5
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 6 of 11
PERSONAL EMAIL TEMPLATE (to be sent to potential participants)
Dear XX
Thank you for your interest in our research. To evaluate our work we are conducting
multiple longitudinal case studies with participants who use our tools to conduct their
own work (i.e. analyzing their own data, at their own pace).
During our first meeting we will give you an entry questionnaire to record your plans and
expectation. We will also setup a schedule of visits at regular intervals chosen by you,
during which we will collect information about your use of the tool and the results of
your ongoing analysis. We will let you install our software on your own computer and
provide training and assistance during the duration of the study. You will be able to
choose to have your name, institution and results mentioned in our papers and
presentations (or not).
Other current participants include XXX who is working on XXX and YYYY
For more detailed information about the tool please visit our website at
www.cs.umd.edu/hcil/nameoftool
Let us know if you are interested
Catherine and Ben
6
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 7 of 11
ENTRY QUESTIONNAIRE
Name:
____________
Institution:
____________
Date:
____________
Tool(s) to be tested: ___________________________
Summarize the question you have or the problem you are trying to solve:
What data you will analyze:
What prior analysis you or others have already conducted with that data:
Give examples of the type of discovery, finding, insight you HOPE to make during the
data analysis.
What would you consider to be a successful outcome of this analysis in your own
profession work (e.g. a new drug discovered, money saved, a scientific paper submitted, a
newspaper article published, a change in work practices, new hypotheses, improve
confidence in data quality, better awareness, etc.)
- Moderate success:
- Significant success:
7
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 8 of 11
Schedule of interviews and visit
What is the estimated start and end date of the analysis you plan to conduct?
What would be an acceptable schedule of visits for HCIL researchers to come for short
interviews (biweekly personal or phone interviews are preferred)
8
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 9 of 11
EXIT QUESTIONNAIRE
Name:
____________
Institution:
____________
Date:
____________
Start and end date of the case study: ______________________________
Tools being used: to analyze the data
HCIL tools ______________________________________
Other tools ______________________________________
Summarize in a paragraph how you used the tools HCIL provided (describe how it was
used in conjunction with other tools if appropriate, and give indications of the amount of
effort spent)
Summarize the discovery made or the finding/insights gained during the data analysis
(provide references if necessary)
9
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 10 of 11
Could those discoveries/findings have taken place without the use of the tools provided?
[ ] Yes
[ ] Yes probably, but it would have been difficult
[ ] Yes possibly, but it would have been extremely difficult
[ ] Most likely No
[ ] Definitively No
Comments:
Please rate the utility of the tool in your data analysis
For this particular case study the tool was:
Not useful at all: 1 2 3 4 5 6 7 Extremely useful
In general the tool is likely to be:
Not useful at all: 1 2 3 4 5 6 7 Extremely useful
Would you be likely to use such a tool in the future if it was available?
[ ] Yes [ ] May be [ ] No
If any, what features should be added or modified for you to use it on a regular basis?
Can you give examples of other potential uses for analysis in your work?
If a discovery was made or significant insight was gained
What professional output is likely to be produced (e.g. none, a scientific paper
submission, a report produced, a presentation to colleagues, a white paper, a new
direction of work, etc.)
How does this compare to your original expectations before starting with the tool.
Well below my expectations 1 2 3 4 5 6 7 Well above my expectations
10
SAMPLE IRB Application for MILC STUDIES - 4/11/2010 – Catherine Plaisant
Page 11 of 11
Longitudinal Case Study Evaluation of Graphical User Interfaces
Research being conducted by XXX (XXX email) and tel XXX
CASE STUDY EXIT Consent form
As researchers we hope to be able to report on your case study in our scientific papers
and public presentations. Please specify how you want us to report on your case study:
I consent to have my case study described in generic terms that do not identify my name,
institution or my discoveries and findings.
(please initial) ___ Yes
___ No
I consent to have my case study described in scientific papers or presentations, with
mention of my name and institution in the credits or in the body of the paper. HCIL
will use the name and institution provided below.
(please initial) ___ Yes
___ No
I consent to have a general layman description of my discoveries and findings
mentioned in scientific papers or presentations
(please initial) ___ Yes
___ No
If you answered YES to any of the above questions, please answer the following
question:
I request the right to review the materials to be published or presented . I will provide
consent by email within a week of receiving the materials.
(please initial) ___ Yes I want to review the materials
___ No, this is not needed.
A copy of the exit questionnaire and consent form will be provided to me.
SIGNATURE _____________________________ DATE ____________
Name (PRINT) ____________________________ Institution (Print)__________________________
Approval Date:
Expiration Date:
11