Recommendations
for a Campus
Survey Tool
Orlando Leon, Enterprise Architect Education, ITS
Beth Berrean,
Communications/Website Services
Manager, ISU
Opinder Bawa, Chief Technology
Officer, ITS
Overview
• Charge
• Analysis
• Recommendations
• Next Steps
Our Charge
• Should we purchase and support Qualtrics as
an enterprise solution for the Campus?
• Should UCSF decide on a single solution for all
survey needs?
• Should UCSF rationalize availability and support
of existing survey solutions currently within the
Campus?
Analysis: Sample Use Cases
Encountered
• Education
– Student Learner Survey
– Evaluation
– Feedback
– Registration
• Research
– Radiology Orthopedics Study
– mOst Survey Tool
– ATHENA
– Clinical Trial Qualification Surveys
– Research Questionnaires
Analysis: Additional Use Cases
Encountered
• Administrative
– Polls
– Voting
– Usability Satisfaction/Feedback Surveys
– Interview Candidate Feedback
• Clinical
– Nursing PI Surveys
– Patient Satisfaction
Analysis: Existing Survey Tools
Currently
supported by
Primary Use
Costs
Checkbox
SOM ISU, SOM
EdTech
Surveys for education
and some one-off
surveys for SOM
TCO $7600/year for
unlimited users
RedCap
Cancer Center & ITS
ARS
Clinical research or by
individual PI’s who
need to store ePHI
Unknown, but it is an
open-source product
Salesforce
SOM ISU
Athena, Nursing
Performance
Improvement Group
Project-based
SurveyMonkey
Ad-hoc
Simple surveys
$780/user/year for
highest level of
support
Zoomerang
Nursing
Alumni outreach
$599/user/year
Findings on Tools: CheckBox (v4)
• User interface: OK
• No end user (survey creator) training program
• As currently implemented at UCSF,
– Not able to collect ePHI
– No support for multi-lingual surveys
– Allows importing of data (Medical Students since
2004)
– Integrated with AD (cannot integrate with MyAccess)
• Key Feature: Actively Used
– The SOM MedEd group uses this tool widely and
SOM ISU also uses it. It is stable, and there is a
upgrade path.
Findings on Tools: RedCap
• Able to collect ePHI
• User interface: complex
• ARS offers classes
• As currently implemented at UCSF,
– No support for multi-lingual surveys
– No native accessibility features
– Not integrated with MyAccess
– Mobile compatible?
• Key Feature: Validated Instruments
– Data collection instruments and forms reviewed for
research relevance, accuracy in coding and function
by the RedCap Library Oversight Committee.
Findings on Tools: Salesforce Health Quality Surveys (HQS)
• Able to collect ePHI
• Integrated into mobile application development
platform
• Open-source survey definition language
supported by Harvard, MIT, and the Boston
Children's Hospital
• Key Feature: Integration with Salesforce
– Integrated with multiple systems (including multiple
EHR systems) and the UCSF IDR
• “Anything is possible.”
Findings on Tools: Qualtrics
• Able to collect ePHI (after signing a BAA)
• SSO integration with MyAccess
• Mobile versions of surveys “out of the box”
• Integration with Salesforce objects
• Supports 48 languages
• ADA compliance-checking tool
• Panels
• Triggers
• Robust user/group/organization/library features
• Programmatic interface to administration tools
• Training and Support
Findings on Tools: Others
• SurveyMonkey
– ADA compliant
– Free
• Zoomerang
• SurveyGizmo
– Mobile versions built into platform
– Professional services available
– Integration with Salesforce
Considerations
• In conversations with stakeholders (likely
survey users—SON, SOM MedEd, Nursing PI,
SOM TAC people asked us to consider:
– Product Features
– External/Vendor Support
– Internal Product Ownership
– Integration with Existing Systems
– Total Cost of Ownership
– Expertise in Survey Design vs Knowledge of the
Technology
Recommendations
• Continue support for RedCap research,
Salesforce for research and clinical needs
• Replace Checkbox with Campus license for
Qualtrics
• Create a strategic marketing plan to help move
all qualified users into Qualtrics
• Create product oversight group for each part of
the Campus that intends to use the tool
(Schools, Centers, Departments, Administrative)
as first line of support for tool
Cost Projection
• 2011-2012 – est $48,000
–
$20,000 - Annual license cost
–
$20,000 - 0.25 FTE @ $80k/year for brand administration,
organization configuration and custom template creation
–
$5,000 – 0.05 FTE @ $100k/year for MyAccess
integration
–
$6,000 – 0.10 FTE @ $60k/year for marketing and
communications
• 2012-2013+ est $24,000*
–
$20,000 - Annual license cost
–
$4,000 - 0.05 FTE @ $80k/year for brand administration
* The assumption is that each functional unit using Qualtrics will
administer their own users and organization configuration, while the
overall brand administrator administers the overall Campus/global
configuration and setup of new functional units
Next Steps
• Endorse recommendation
• Decide where within the enterprise the product
should be managed
• Negotiate a BAA and license agreements with
Qualtrics
• Create a Product Oversight Committee
• Develop a marketing/communications plan for
helping internal constituents identify survey
needs and tools
15