Qualtrics Survey
Research Suite:
Orlando Leon, Applications Leader Education, ITS
A Case Study of Innovating
Business Processes and Systems
at UCSF
Overview
• Introduction to Qualtrics
• Statistics about Qualtrics at UCSF
• Use Cases
• Future Roadmap
• Resources
• Q&A
Introduction: Cloud-based Surveys
“Easy enough for an intern, sophisticated enough
for a PhD.”
Introduction: Key Features
• Able to collect ePHI
• SSO integration with MyAccess
• Mobile versions of surveys “out of the box”
• Integration with Salesforce objects
• Supports 48 languages
• ADA compliance-checking tool
• Panels
• Triggers
• Robust user/group/organization/library features
• Programmatic interface to administration tools
• Training and Support
Introduction: Who May Use It
• Staff
• Faculty
• Students/Learners
• Affiliates
• Others
Introduction: Scope of License
The term of the license for the Qualtrics Software is one year,
beginning January 16, 2012 and includes an unlimited number of
surveys. The Qualtrics license may be used by academic
areas within University of California San Francisco for
projects that are academic in nature. Academic projects may
serve research (instructional or academic) and administrative
purposes. The license may not be used for purely corporate or "forprofit" purposes. Such corporate projects would include purely clinical
and patient related surveys that are not related to academic research.
Affiliated hospital and clinics may purchase Qualtrics access
according to Qualtrics corporate pricing model. A 10% discount will be
applied to any license purchased for corporate purposes.
7
Statistics about Qualtrics at UCSF
• 45+ Divisions
• 1,643 Users
• 7,596 Surveys
• 331,338 Auditable Responses
• 69,957 Responses on currently most active
survey
* Campus License since January 2012 – Officially Announced to
UCSF Summer 2012
Use Cases
• Education
– Student Learner Survey, Evaluation, Feedback,
Registration, etc
• Research
– Radiology Orthopedics Study, mOst Survey Tool
– ATHENA, Clinical Trial Qualification Survey,
Research Questionnaires, etc
• Administrative
– Polls, Voting, Usability Satisfaction/Feedback
Surveys, Interview Candidate Feedback, etc
• Clinical
– Nursing PI Surveys, Patient Satisfaction, etc
Innovative Use Cases (1)
• Shared Asset Libraries Among Departments
• Customizable Reports with Live Updates [link]
• Accreditation Review Objectives
• Collection of Data (ie pictures, homework, quizzes,
research materials)
• Student Elections [link]
• Quizzes/Scoring
• 360 Evaluations + Follow-up Evaluations
• Online Training [link]
• Access Requests [link]
• Event Registration
Innovative Use Cases (2)
• Online Form Integration (ie Drupal)
• Dynamic Classroom or Meeting Polling with Live
Results with Mobile Devices (ie Clickr) [link]
• Service Checklist (ie onboarding, change control,
surgery, etc) [link] [link2]
• Shopping List with Pictures [link]
• Service Tracking (ie quick questionnaire for frontline staff) [link]
• Decommissioning of FileMaker Pro Web Forms
• Application Forms [link]
• Interview Evaluations [link]
• Reservation Requests [link]
Innovative Use Cases (3)
• Service Requests (ie. computer/supply purchase
requests, etc) [link]
• Branded Customer Service Feedback forms [link]
• Integration within Web Applications using Custom
REST API Calls (ie SLR)
• Collaboration without Sharing Passwords
• Collaboration Outside of UCSF
• Panels and SSO-based Contextual Information [link]
• Quizzes and Learning Evaluation
• Annual Evaluations (ToC)
• ID Badge Scanning [link]
Lessons Learned
• Limitations
• Growing Pains
• New Features and Enhancements
– Some free, some not
Future Roadmap
• Funding
• Offline Response Gathering
• 360 Reviews
• Evaluations
• Crowdsourcing of Survey Innovations
• Deeper Integrations with Salesforce
Resources
• Sharecase Booth A2
• Recorded Training Sessions
• UCSF Informational Website
– http://tiny.ucsf.edu/surveysoftware
• Qualtrics Website
– http://www.qualtrics.com
• Qualtrics User Conference - February 2014
– http://www.qualtrics.com/summit
• Other Survey Tools at UCSF
– REDCap, Salesforce
• Qualtrics Chatter Group
15
Questions?
16
Demo
https://ucsf.qualtrics.com
http://tiny.ucsf.edu/qualtrics_kitchensink
http://tiny.ucsf.edu/qualtrics_sharecase
https://ucsf.us.qualtrics.com/SE/?SID=SV_3C0A0QVPbEsHOv2
https://ucsf.us.qualtrics.com/SE/?SID=SV_50fnPOSxU2srWAI
https://ucsf.us.qualtrics.com/SE/?SID=SV_4ZcwwpjdoQMbfW4
https://ucsf.us.qualtrics.com/SE/?SID=SV_6VxQn3fGl6x6UMk
https://ucsf.us.qualtrics.com/SE/?SID=SV_brUenOV1gE4wXyc
https://ucsf.us.qualtrics.com/SE/?SID=SV_bJm24deQNGh1V08
https://ucsf.us.qualtrics.com/SE/?SID=SV_5o8SkNlPBmB7VJi
18
Business Case for Enterprise
Survey Tool
• Should we purchase and support Qualtrics as
an enterprise solution for the Campus?
• Should UCSF decide on a single solution for all
survey needs?
• Should UCSF rationalize availability and support
of existing survey solutions currently within the
Campus?
Business Case for Enterprise
Survey Tool: Considerations
• In conversations with stakeholders (likely
survey users—SON, SOM MedEd, Nursing PI,
SOM TAC), people asked us to consider:
– Product Features
– External/Vendor Support
– Internal Product Ownership
– Integration with Existing Systems
– Total Cost of Ownership
– Expertise in Survey Design vs Knowledge of the
Technology
Introduction: Cost
• It is FREE (funded by SAA and ITS)
21
Analysis: Cost Projection
• 2011-2012 – est $48,000
–
$20,000 - Annual license cost, based on “Carnegie
Classification”
–
$20,000 - 0.25 FTE @ $80k/year for brand administration,
organization configuration and custom template creation
–
$5,000 – 0.05 FTE @ $100k/year for MyAccess
integration
–
$6,000 – 0.10 FTE @ $60k/year for marketing and
communications
• 2012-2013+ est $24,000*
–
$20,000 - Annual license cost
–
$4,000 - 0.05 FTE @ $80k/year for brand administration
* The assumption is that each functional unit using Qualtrics will
administer their own users and organization configuration, while the
overall brand administrator administers the overall Campus/global
configuration and setup of new functional units
Analysis: Implementation Details
• BAA reviewed and signed
• License negotiated, reviewed, and signed
– License terms
– Features
• Advisory and Oversight Committee
– SON, SOM/UME/GME, SOD, SOP, Graduate
Division, Academic Senate, Center for Health
Professions, CTSI, Library, Memory and Aging, SAA,
ITS
• Configuration and Deployment
– MyAccess/SSO, Wiki/FAQ/Listserv/Chatter,
Branding, Marketing/Communications, Administration
23
Analysis: Implementation Details
• Review and Consultation with University
Relations and UCSF Legal
• Division and general use policies
• Marketing and Communication to UCSF
community
• Group Training Sessions
• On-site Visit and Training
24
Introduction: Who May Use It
• Staff
• Faculty
• Students/Learners
• Affiliates
• …
Resources
• Sharecase Booth #2
• Qualtrics UCSF Site Visit
– Wednesday, October 17, 2012 (9AM-12PM) –
Parnassus HSW301 –
http://QualtricsAtUcsf2012.eventbrite.com
• Introduction to Qualtrics
• Show Me the Cool Stuff
• How Do I…
• UCSF Informational Website
– http://tiny.ucsf.edu/surveysoftware
26
Sample Surveys
27
Sample Surveys
28
Sample Surveys
29
Demo
https://ucsf.qualtrics.com
http://tiny.ucsf.edu/qualtrics_kitchensink
http://tiny.ucsf.edu/qualtrics_sharecase
https://ucsf.us.qualtrics.com/SE/?SID=SV_3C0A0QVPbEsHOv2
https://ucsf.us.qualtrics.com/SE/?SID=SV_50fnPOSxU2srWAI
https://ucsf.us.qualtrics.com/SE/?SID=SV_4ZcwwpjdoQMbfW4
https://ucsf.us.qualtrics.com/SE/?SID=SV_6VxQn3fGl6x6UMk
https://ucsf.us.qualtrics.com/SE/?SID=SV_brUenOV1gE4wXyc
https://ucsf.us.qualtrics.com/SE/?SID=SV_bJm24deQNGh1V08
https://ucsf.us.qualtrics.com/SE/?SID=SV_5o8SkNlPBmB7VJi
30