STANFORD UNIVERSITY
School of Medicine
Office of Academic Affairs
Clinical Excellence Survey
Table of Contents
Overview ....................................................................................................................................................... 2
Flow chart ..................................................................................................................................................... 2
Setting Up a Qualtrics Account (first use only) ............................................................................................. 2
Download Standard Template Survey file from OAA Site and Import to Qualtrics ...................................... 3
Activate Your Survey ..................................................................................................................................... 8
Email Your Survey ......................................................................................................................................... 8
Set up Reminder Email ................................................................................................................................ 13
Monitor survey completion ........................................................................................................................ 16
Reporting .................................................................................................................................................... 17
Collaborate.................................................................................................................................................. 20
Closing the CES Survey ................................................................................................................................ 21
Deleting the CES form ................................................................................................................................. 23
Appendix ..................................................................................................................................................... 25
Accessing your Library for invitation emails and reminders ............................................................... 25
Deleting email reminders before they are sent by the system .......................................................... 26
Entering Evaluations from Hard Copies (Two Approaches) ................................................................ 27
Fixing the respondent category type filter on the Results/Report ..................................................... 30
CES Instructions
Page 1
Version 09/19/14
Overview
The Qualtrics survey tool is an easy-to-use, full-featured, web-based tool for creating and
conducting online surveys. The survey tool is available at
http://stanforduniversity.qualtrics.com and is integrated with Stanford’s Kerberos (SUNet ID
authentication) system.
Flow chart
Here’s the simple flow chart of the process:
Task 1:
Set up Qualtrics
account, if
applicable
Download
QSF file from
OAA website
Set up Email Reminder 1 and
Reminder 2
Create New
Survey in
Qualtrics
Upload QSF file
to Qualtrics
Email Survey
Activate Survey
Task 2:
Monitor survey
completion
Create reports and
summaries; convert to
PDF, optional
Upload Reports
to Fast Fac
Task 3:
President
approval of
A&P action
Delete survey
from
Qualtrics
Setting Up a Qualtrics Account (first use only)
1. Go to http://stanforduniversity.qualtrics.com (see #3 below for discussion of an
alternate portal if you are having difficulty logging in or setting up your account –
especially if you receive the error message, “Could not log in as this user").
You will be prompted to log in to Stanford Webauth using your SUNet ID and
password, unless your browser is already open and you have already logged into a
Webauth site.
CES Instructions
Page 2
Version 09/19/14
2. First time users only: You will see an entry screen that allows you to create a new
Qualtrics account or migrate an existing Qualtrics account. Click on the appropriate
button:
o I don't have a Qualtrics account if you do not have a Qualtrics account (i.e.,
you have not used Qualtrics before). Qualtrics creates a new Qualtrics
account associated with your SUNet ID. You will see the Qualtrics "Create
Survey" page. (For future logins, you will only need to complete steps 1 and
2.)
o I already have a Qualtrics account if you currently have a Qualtrics account,
that you would like migrated to a SUNet ID account.
You will see an entry screen where you can enter your existing Qualtrics
account (e.g., jsmith@stanford.edu). Enter your existing account and click on
the Migrate Account button. Qualtrics creates a new Qualtrics account
associated with your SUNet ID and moves your existing surveys, results,
settings, etc., from the old account to the new account. You will see the
Qualtrics "My Surveys" screen. (For future logins, you will only need to
complete step 1.)
3. Some SUNet IDs may require use of the alternate portal
http://stanfordmedicine.qualtrics.com. Either portal will provide functional access
to the Qualtrics software and will allow you to use the online CECCE template.
However, you can only collaborate with people who have an account in the same
portal. You will want to make sure you use the same portal as others in your office
or with those you will collaborate with on CECCE or other surveys.
Download Standard Template Survey file from OAA Site and Import to
Qualtrics
1. Go to http://med.stanford.edu/academicaffairs/administrators/forms.html
2. Right-click (ctrl-click in the Mac) your mouse on “.qsf – Qualtrics CECCE template” and
select “Save Link As” (or “Save Target As”). A pop-up window will appear to
download/save the file. The file should be named “CES_Template_June_2014.qsf” or
similar (it may have been replaced by OAA with a more recent version). Pay attention to
where the downloaded file is placed. For example, in Windows it will normally be
placed in the “Downloads” folder inside your “My Documents” folder on your C: drive.
Also pay attention to the “Save as type” or “Save as” file option in the pop-up window.
Be sure to save the file type as “All Files”. DO NOT select “Text Document” or “.QSF
document” – these file types are usually the default choices. Make sure you select “All
Files.” For Mac Users - if you don’t see “All Files” as an option, consider trying a
different browser, e.g., FireFox. It is important to save as “All Files.” (It may be
tempting to select “.QSF document,” and the name and extension of the resulting saved
file may look normal, but we’ve seen some subtle problems with function when the file
is saved this way.)
CES Instructions
Page 3
Version 09/19/14
a.
4. Click on the “Create Survey” tab in Qualtrics.
CES Instructions
Page 4
Version 09/19/14
1. Click on “Click to Continue” link under Quick Survey Builder.
2. Enter a Survey Name and call it Temp. You can leave the Folder blank.
3. Click on Create Survey button. You will be taken to a blank survey for Dr. John Smith.
CES Instructions
Page 5
Version 09/19/14
4. Click on the Advanced Options link and select Import Survey.
5. Click on the Browse button (or “choose” button in Mac), locate the Clinical
Excellence Survey Template “_______.qsf” file you downloaded from the OAA
website.
6. And then click Import.
7. Your CES survey should now appear on the screen. HOWEVER, the survey name will
appear as CES_Template_June_2014 (the same name as the template file name).
CES Instructions
Page 6
Version 09/19/14
Please follow these quick steps to rename the imported survey according to your
preference:
a. Rename the survey by clicking on the small arrow next to the survey name:
b. A pop-up box will appear. Enter the name of the survey, e.g., John Smith –
CES - 2014
c. Now you will have to delete the blank survey you initially created called
“Temp.” Click on “My Surveys” tab.
d. You will see the “Temp” survey file.
e. Click on the delete button and follow the pop-up screen instructions.
f. And you’re done. We have had some recent discussions with Qualtrics about
whether a more direct process might be possible. If there are changes to the
system, we will release an updated manual.
PLEASE DO NOT MODIFY THE SURVEY IN ANY WAY – if you suspect that you have
accidentally modified the survey, please delete the survey (see the section at the end
of this document), and re-import the template .QSF file as above. Contact the Office
of Academic Affairs if you have concerns or questions.
CES Instructions
Page 7
Version 09/19/14
Activate Your Survey
1. Before you can collect responses, you need to activate your survey. Click on the Launch Survey
button under Edit Survey tab.
2. You will see a link to “Activate your survey to collect responses”. Click on the link.
Email Your Survey
1. Click on the Email Survey button under Distribute Survey tab.
CES Instructions
Page 8
Version 09/19/14
2. In the drop down menu for “To:”, select “Enter email addresses…”
3. Enter email addresses – one email address per row. You should not enter First Name or Last
Name. After entering five email addresses, the Qualtrics system will automatically add more
rows.
CES Instructions
Page 9
Version 09/19/14
4. In the drop down menu for When to distribute the form, select “Send Now” as shown above.
5. In the Message field, enter the Subject line. For example, Dr. John Smith – clinical performance
evaluation – response requested.
CES Instructions
Page 10
Version 09/19/14
6. In the body of the email, remove the language indicated below in the figure:
7. REPLACE THE REMOVED TEXT WITH THE FOLLOWING:
If you prefer to respond via hard copy, you may print the form at
http://med.stanford.edu/academicaffairs/documents/administrators/forms/ClinicalExcellenceE
valuationForm060414.doc (Word) or
http://med.stanford.edu/academicaffairs/documents/administrators/forms/ClinicalExcellenceE
valuationForm060414.pdf (PDF) and fax to [fax number], return via email to [email address], or
mail to [address].”
CES Instructions
Page 11
Version 09/19/14
8. Insert the following text above the automated message, filling in details as appropriate:
The Department of ______ at the Stanford University School of Medicine is reviewing Dr. [full
name] for [reappointment as/promotion to] [rank] in the [line]. We ask your assistance in
completing an online survey to evaluate Dr. [last name]’s clinical performance. Your response
will be kept confidential within our Appointments and Promotions review process.
The survey includes 26 brief performance questions and should take no more than a few
minutes to complete. It would be most helpful to receive your response by DATE.
Thank you very much for your help. Please notify [FAA Name] at [email address; 650-XXX-XXXX]
if you are having difficulty completing the survey.
9. Here’s what the body of the email should look like once it is sent out:
The Department of Pathology at the Stanford University School of Medicine is reviewing Dr. John
Smith for promotion to Associate Professor in the MCL. We ask your assistance in completing an
online survey to evaluate Dr. Smith's clinical performance. Your response will be kept
confidential within our Appointments and Promotions review process.
The survey includes 26 brief performance questions and should take no more than a few minutes
to complete. It would be most helpful to receive your response by January 15, 2011.
Thank you very much for your help. Please notify Cynthia Llanes at cllanes@stanford.edu; 650724-3725 if you are having difficulty completing the survey.
Follow this link to the Survey:
Take the Survey
Or copy and paste the URL below into your internet browser:
https://stanforduniversity.qualtrics.com/WRQualtricsSurveyEngine/?Q_SS=5C59T10HRLnWtww
If you prefer to respond via hard copy, you may print the form at
http://med.stanford.edu/academicaffairs/documents/administrators/forms/ClinicalExcellenceEv
aluationForm060414.doc (Word) or
http://med.stanford.edu/academicaffairs/documents/administrators/forms/ClinicalExcellenceEv
aluationForm060414.pdf (PDF) and fax to 650-999-9999, return via email to
cllanes@stanford.edu, or mail to Cynthia Llanes, Dept of Pathology, Lane Building, L235, Stanford
CA 94305-5324.
10. When you’re done editing the message, click on the Save As button at the upper right.
CES Instructions
Page 12
Version 09/19/14
11. Select “My Library” from the drop down menu, if it is not the default, and enter the title of
email, for example: Smith, John reappointment or John Smith reappt.
12. Before sending the message, copy the entire text of the email body. This is a convenience
issue; you will need to paste this text in your reminder emails described in the next section of
this manual.
13. Send the message using the “Send Now” button at the lower right of the screen. Then proceed
immediately to the “set up reminder email” section below.
Set up Reminder Email
Immediately after you send out the initial email to evaluators, you should set up the reminder emails.
Qualtrics allows you to send the reminder at various times, as early as “Now” to “28 days” later. You can
even customize it to the exact date and time. We suggest you set up the reminder so that it will be sent
in “7 days” and another reminder in “14 days”. Here’s how to set it up:
1. Under Distribute Survey tab, click on the Email History button (the page may not change when
you do this – you may already be on the right page).
CES Instructions
Page 13
Version 09/19/14
2. Then click on the arrow icon under the Actions column:
3. Select Send Reminder or Thank You.
4. Select “Send Reminder” radio button.
5. Select “Send in 7 days” from the drop down menu under When.
6. Fill out the Subject Line as “Dr. Smith – clinical performance evaluation – response requested”
CES Instructions
Page 14
Version 09/19/14
7. If you recall from the previous section (#12), you copied the email body text. Please paste that
text in the email body, replacing the system-generated text.
8. OPTIONAL: Insert the following text above the email message you just pasted.
REMINDER
If you have already submitted your clinical evaluation of Dr. [full name], please disregard
this message. [This message is directed at any evaluators who submitted hard copy
forms. If you did not get any, then you can delete this line.]
9. When you’re done editing the message, click on the “Save As” button.
10. Select “My Library: [your name]” folder and type in the name of this reminder (e.g. “John Smith
Reminder”).
11. Finally, click on the Send button.
Now, set up another email reminder to be sent in 14 days by following the instructions below:
1. Under Distribute Survey tab, click on the Email History button (the page may not change when
you do this – you may already be on the right page).
2. Then click on the arrow icon under the Actions column:
3. Select Send Reminder or Thank You.
4. Select “Send Reminder” radio button.
5. Select “Send in 14 days” from the drop down menu under When.
6. Fill out the Subject Line as “Dr. Smith – clinical performance evaluation – response requested”
7. Click on the “Load a Saved Message” field and select “My Library” and the name of the email
reminder you created (e.g. “John Smith Reminder”).
8. Then click on the Save As button and save it as John Smith Reminder 2.
9. Then click on the Send button.
CES Instructions
Page 15
Version 09/19/14
Monitor survey completion
Now that your survey has been emailed and reminders have been set up, you can check back in a few
days to see the progress. You can see how many responses you have received by clicking on the “My
Surveys” tab and looking at the “Responses” column.
If you want to see more granular information, then click on the Survey Name under My Surveys.
Then click on the Distribute Survey tab, then the Email History icon.
You will see the number of emails sent out, failed, started, finished, and the date it was sent.
CES Instructions
Page 16
Version 09/19/14
Reporting
Once you have received enough responses and the reminder dates have passed, you can now create
your reports. NOTE: if you do not get enough responses, be sure to contact Office of Academic Affairs to
get further instructions on the next steps.
1. Click the View Results tab. Then, click on the specific respondent group whose response you
wish to view (we have selected the “Allied Healthcare Providers” in the example below – see
#12 below at the end of this section for a note regarding use of the “All Responses” report):
2. Upon clicking the Allied healthcare provider report, you will be taken to this screen:
CES Instructions
Page 17
Version 09/19/14
3. You will notice that there is a drop down menu for the other categories on the left hand side.
This gives you quick access to the other category reports.
4. Click on the Word document icon on the right hand side.
5. You will get this pop up screen.
6. Select “Open with Microsoft Office Word (default), then click OK button.
7. HINT: The Word document is approximately 10 pages or so for each category. To decrease the
number of pages, we suggest you “Select All” and resize the document font to size 10.
CES Instructions
Page 18
Version 09/19/14
8. Download the Word document for each category type by selecting the category from the drop
down menu on the left side of the navigation.
NOTE: The PDF version creates too many page breaks and therefore the reports are too long.
The Word version is the best option.
9. Again, resize the font size for each Word document to size 10. This will help decrease the
number of pages.
10. Optional and if you have Adobe Professional software, you can convert these Word docs to PDF
format and combine all the reports as one PDF file. This will make it easier to upload to Fast|Fac.
11. Be sure to include a cover page for your CES report which shows the following sample
information:
CES SUMMARY
Dr. John Smith, MD, promotion to Assoc Prof (UTL) with tenure
Department of Radiology
February 2011
Report prepared by Kendall Yi
Received
Solicited
Category
11
13
Physicians (not trainees)
5
6
Trainees of the candidate
3
3
Clinical administrators (for example, nurse manager or clinic manager)
3
3
Allied healthcare providers (e.g., nurse practitioner or physician assistant)
0
0
Other
12. NOTE: The “All Responses” category in Qualtrics reports/results will include all responses in a single
report. If the responses received for your candidate are all strongly positive, you may include this
comprehensive report in place of the individual category reports. This will save significant space/paper
CES Instructions
Page 19
Version 09/19/14
in the long form. If there are negative comments or scores, we ask that you provide the individual
category reports.
Collaborate
The Collaborate option in Qualtrics allows you to share or co-author a survey in your account with
another individual or group. Collaborating on a survey allows you to give others access to a survey at a
level you specify.
1. Go to My Surveys tab. You will see a list of surveys. Click on the collaborate button for the
survey you wish to share.
2. You will be asked to select the user to collaborate with. If the individual is a Qualtrics user, type
in their name.
Note: If they are not a Qualtrics user, type in their email address. A screen will then
appear asking you to create an email message to be sent to them. When they access this
link a trial account is automatically generated for them. Although they'll have limited
permissions, they will be able to see the survey you shared with them. They'll be asked
to provide their name and a password for their account.
3. Enter the name. If the person has a Qualtrics account, his/her name will appear as a selection
option. Once a name or email address is added, click on the Add button.
4. You will be given a set of permissions that you can turn on or off for the person you‘re
collaborating with. For instance, you can set up the permissions so the individual is able to view
CES Instructions
Page 20
Version 09/19/14
the results but unable to deactivate the survey. Select the choices under the Edit and View
Results columns.
You should not check off Activate/Deactivate or Copy or Distribute boxes.
Closing the CES Survey
Assuming you have enough CES responses, deadline has passed, you have sent two reminders, you have
created the reports, and submitted the long form to OAA, below are the steps to close the survey.
Please keep in mind that anyone who tries to fill out the survey after it has closed will get this message:
CES Instructions
Page 21
Version 09/19/14
1. Click on the My Surveys tab
2. Click on the Name of the survey
3. Click on the Survey Options button
4. Change the following survey options under the Survey Protection section:
a. Check the radio button for “This survey is valid from ______ to ____”.
b. Enter yesterday’s date as an end date.
It should look something like this (see Survey Protection section):
CES Instructions
Page 22
Version 09/19/14
5. Click on the Save Changes button.
Deleting the CES form
1. After the appointment, reappointment or promotion has been approved by the President,
delete the survey from Qualtrics. Go to My Surveys tab. You will see a list of surveys. Click on
the Delete button for the survey you wish to delete.
2. A pop-up box will appear. You will need to enter “delete” and then click on the Delete button to
proceed.
CES Instructions
Page 23
Version 09/19/14
CES Instructions
Page 24
Version 09/19/14
Appendix
Accessing your Library for invitation emails and reminders
How to access your “Library” – where you have set up invitation emails and reminders.
a. Click on the Library Tab, then click on the number of library messages you have under
Message Libraries _ your name. See below:
b. Once you click on the number, you’ll be able to view, edit, copy, delete, and create new
messages.
c. You may want to go to your library to copy an existing message in your library and add it
to your reminder library by creating a new message and pasting the message you copied
from your invite emails library.
CES Instructions
Page 25
Version 09/19/14
Deleting email reminders before they are sent by the system
After setting up a reminder, you will not be able to change the distribution time. You will, however, be
able to cancel (delete) the distribution and set up a new reminder, if you wish. You can do this by going
to Distribute Survey tab, then Email History page, then click on the “bell” icon.
You will get this pop-up menu:
Put your mouse over the line bar under the Actions column. You’ll get the option to “Delete
Distribution”.
CES Instructions
Page 26
Version 09/19/14
Select Delete Distribution. Another pop-up screen will appear asking you to confirm deletion. Select
“Delete”. It will remove the reminder message.
Entering Evaluations from Hard Copies (Two Approaches)
There are a couple ways you can enter evaluations that you have received as hard copies (outside the
Qualtrics system). Hopefully you will not receive too many of these.
a) One simple way to get this information into the system is to send the survey link to yourself
and then complete the survey using the hard copy responses received. You would do this by
going to Distribute Survey tab, then Email Survey button, similar to what you did to send out
the survey to the respondents, but this time you enter your email address. This approach
works if you have only one hard copy response to input (because the survey options are set
up to prevent “ballot box stuffing,” the software will only allow you to take the survey once
as an email respondent).
b) The approach described below works if you have many hard copy responses to input. This
should be done at the very end of the process, when reasonable time has passed since the
final reminder, and you are not expecting to receive any more electronic responses. You will
change the survey options to allow you to enter multiple responses manually (without an
email solicitation). You will then immediately close the survey.
Here are the steps to manually input multiple hard copy responses and then immediately close the
survey:
1. Click on the My Surveys tab
2. Click on the Name of the survey
3. Click on the Survey Options button
CES Instructions
Page 27
Version 09/19/14
4. Change the following survey options under the Survey Protection section:
a. Click on Open Access
b. Click on Password Protection, then enter a password (example below is “hello2015”)
c. Deselect Prevent Ballot Box Stuffing
d. And make sure you are still within the valid dates of the survey.
It should look something like this (see Survey Protection section):
5. Click on the Save Changes button.
6. Click on Distribute Survey tab.
CES Instructions
Page 28
Version 09/19/14
7. Click on the Survey Link button at the upper left, then the URL link in the center:
8. Enter the password:
(If you don’t know the password, go back to the Qualtrics control panel, click on Edit Survey tab, then
click on the Survey Options button.)
9. Fill out the survey for each hard copy you have. At the end of each submission you’ll get this
message:
CES Instructions
Page 29
Version 09/19/14
Go back to the URL on the distribute survey tab, and click on it to enter the password and to fill
out another survey.
10. After you have completed entering all the hard copy surveys, go to the Edit Survey tab, click on
the Survey Options button, then change the survey valid end date to yesterday’s date.
11. Once you do this, anyone who goes to the survey link will see this message:
Note: The submissions you entered today will not disappear or be removed because you changed the
survey valid end date.
Fixing the respondent category type filter on the Results/Report
1. If you inadvertently removed the filter of the Results/Report (red graphic in middle of page), it
will cause data to be included in the wrong respondent category types. Here’s how to fix it.
a) Click on the link “Add a Subgroup to This Report” within the tab (category type) that is
missing the filter.
CES Instructions
Page 30
Version 09/19/14
b) In the drop down menus, select If “Question” “Q2 PLEASE TELL US…” “[CATEGORY TYPE YOU
NEED]” IS “SELECTED”.
c) Then click Apply Changes.
d) Depending on your local environment, you may get a lightbox stating “Generating your
document…” followed by a pop-up box to Open or Save the word document. Alternately,
the software may simply display a new report with the appropriate respondent category
data.
CES Instructions
Page 31
Version 09/19/14