Recommendations
for a Campus
Survey Tool
Orlando Leon, Enterprise Architect Education, ITS
Beth Berrean,
Communications/Website Services
Manager, ISU
Opinder Bawa, Chief Technology
Officer, ITS
Overview
• Charge
• Analysis
• Current Progress
• Next Steps
Our Charge
• Should we purchase and support Qualtrics as
an enterprise solution for the Campus?
• Should UCSF decide on a single solution for all
survey needs?
• Should UCSF rationalize availability and support
of existing survey solutions currently within the
Campus?
Analysis: Sample Use Cases
Encountered
• Education
– Student Learner Survey
– Evaluation
– Feedback
– Registration
• Research
– Radiology Orthopedics Study
– mOst Survey Tool
– ATHENA
– Clinical Trial Qualification Surveys
– Research Questionnaires
Analysis: Additional Use Cases
Encountered
• Administrative
– Polls
– Voting
– Usability Satisfaction/Feedback Surveys
– Interview Candidate Feedback
• Clinical
– Nursing PI Surveys
– Patient Satisfaction
Analysis: Existing Survey Tools
Currently
supported by
Primary Use
Costs
Checkbox
SOM ISU, SOM
EdTech
Surveys for education
and some one-off
surveys for SOM
TCO $7600/year for
unlimited users
RedCap
Cancer Center & ITS
ARS
Clinical research or by
individual PI’s who
need to store ePHI
Unknown, but it is an
open-source product
Salesforce
SOM ISU
Athena, Nursing
Performance
Improvement Group
Project-based
SurveyMonkey
Ad-hoc
Simple surveys
$780/user/year for
highest level of
support
Zoomerang
Nursing
Alumni outreach
$599/user/year
Findings on Tools: Qualtrics
• Able to collect ePHI
• SSO integration with MyAccess
• Mobile versions of surveys “out of the box”
• Integration with Salesforce objects
• Supports 48 languages
• ADA compliance-checking tool
• Panels
• Triggers
• Robust user/group/organization/library features
• Programmatic interface to administration tools
• Training and Support
Considerations
• In conversations with stakeholders (likely
survey users—SON, SOM MedEd, Nursing PI,
SOM TAC), people asked us to consider:
– Product Features
– External/Vendor Support
– Internal Product Ownership
– Integration with Existing Systems
– Total Cost of Ownership
– Expertise in Survey Design vs Knowledge of the
Technology
Recommendations
• Continue support for RedCap research,
Salesforce for research and clinical needs
• Replace Checkbox with Campus license for
Qualtrics
• Create a strategic marketing plan to help move
all qualified users into Qualtrics
• Create product oversight group for each part of
the Campus that intends to use the tool
(Schools, Centers, Departments, Administrative)
as first line of support for tool
Cost Projection
• 2011-2012 – est $48,000
–
$20,000 - Annual license cost
–
$20,000 - 0.25 FTE @ $80k/year for brand administration,
organization configuration and custom template creation
–
$5,000 – 0.05 FTE @ $100k/year for MyAccess
integration
–
$6,000 – 0.10 FTE @ $60k/year for marketing and
communications
• 2012-2013+ est $24,000*
–
$20,000 - Annual license cost
–
$4,000 - 0.05 FTE @ $80k/year for brand administration
* The assumption is that each functional unit using Qualtrics will
administer their own users and organization configuration, while the
overall brand administrator administers the overall Campus/global
configuration and setup of new functional units
Scope of License
The term of the license for the Qualtrics Software is one year,
beginning January 16, 2012 and includes an unlimited number of
surveys. The Qualtrics license may be used by academic areas
within University of California San Francisco for projects that are
academic in nature. Academic projects may serve research
(instructional or academic) and administrative purposes. The
license may not be used for purely corporate or "for-profit"
purposes. Such corporate projects would include purely clinical and
patient related surveys that are not related to academic research.
Affiliated hospital and clinics may purchase Qualtrics access
according to Qualtrics corporate pricing model. A 10% discount will be
applied to any license purchased for corporate purposes.
11
Current Progress
• BAA reviewed and signed
• License negotiated, reviewed, and signed
– License terms
– Features
• Advisory and Oversight Committee
– SON, SOM/UME/GME, SOD, SOP, Graduate
Division, Academic Senate, Memory and Aging,
Center for Health Professions, CTSI, Library, SAA,
ITS
• Configuration and Deployment
– MyAccess/SSO, Wiki/FAQ, Branding,
Marketing/Communications, Administration
12
Next Steps
• Review and Consultation with University
Relations and UCSF Legal
• Finalization of division and general use policies
• Marketing and Communication to UCSF
community
• Group Training Sessions
13
Findings on Tools: CheckBox (v4)
• User interface: OK
• No end user (survey creator) training program
• As currently implemented at UCSF,
– Not able to collect ePHI
– No support for multi-lingual surveys
– Allows importing of data (Medical Students since
2004)
– Integrated with AD (cannot integrate with MyAccess)
• Key Feature: Actively Used
– The SOM MedEd group uses this tool widely and
SOM ISU also uses it. It is stable, and there is a
upgrade path.
Findings on Tools: RedCap
• Able to collect ePHI
• User interface: complex
• ARS offers classes
• As currently implemented at UCSF,
– No support for multi-lingual surveys
– No native accessibility features
– Not integrated with MyAccess
– Mobile compatible?
• Key Feature: Validated Instruments
– Data collection instruments and forms reviewed for
research relevance, accuracy in coding and function
by the RedCap Library Oversight Committee.
Findings on Tools: Salesforce Health Quality Surveys (HQS)
• Able to collect ePHI
• Integrated into mobile application development
platform
• Open-source survey definition language
supported by Harvard, MIT, and the Boston
Children's Hospital
• Key Feature: Integration with Salesforce
– Integrated with multiple systems (including multiple
EHR systems) and the UCSF IDR
• “Anything is possible.”
Findings on Tools: Others
• SurveyMonkey
– ADA compliant
– Free
• Zoomerang
• SurveyGizmo
– Mobile versions built into platform
– Professional services available
– Integration with Salesforce
Next Steps
• Endorse recommendation
• Decide where within the enterprise the product
should be managed
• Negotiate a BAA and license agreements with
Qualtrics
• Create a Product Oversight Committee
• Develop a marketing/communications plan for
helping internal constituents identify survey
needs and tools
19