- D
advertisement
20 November 2013 An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas Prepared for the ULS Leadership Program by Luke Ferdinand, John Fudrow, Karen Calhoun and Jeff Wisniewski This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. 2 Agenda Topic Arrival Introduction to the workshop Needs assessment - Luke Box lunches Surveys – John Short break Focus groups Short break Personas research – Jeff Wrap up and get on shuttle Time 10:25-10:35 am 10:35-10:45 am 10:45 am-12:00 pm 12:00 -12:30 pm 12:30-1:30 pm 1:30-1:40 pm 1:40 -2:40 pm 2:40 –2:50 pm 2:50 – 3:50 pm 3:50 – 4:00 pm Needs Assessment Defining and Addressing The Needs of an Organization and Its Audience Luke Ferdinand ULS Leadership Program 20 November 2013 4 Objectives Define Needs in Organization Define Needs Assessment Understand the methods and tools used in Needs Assessment Know potential risks in Needs Assessment methods ULS Research Methods Workshop 5 What is a Needs Assessment? "A systematic process for collecting information and making justifiable decisions" - Ryan Watkins Ryan Watkins, 2013, “Needs = Gaps in Results”, ULS Research Methods Workshop http://ryanrwatkins.com/na/naintroduction.html, 6 What is a need? “Needs are the differences between your current achievements and your desired accomplishments.” Ryan Watkins, 2013, “Needs = Gaps in Results”, ULS Research Methods Workshop http://ryanrwatkins.com/na/naintroduction.html, 7 Needs Assessments Help Identify Data that define your needs Prioritization of your needs Criteria for implementing solutions Information necessary to justify selection of one or more activities to improve performance Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html ULS Research Methods Workshop 8 Why? “Provide a systematic process to guide decision-making in organizations Provide justification for decisions before they are made. Scalable for any size project, time-frame, or budget. Offer a replicable model that can be applied by novices or experts. Provide a systemic perspective for decision-makers. Allow for interdisciplinary solutions for complex problems. “ Ryan Watkins, 2013, Needs = Gaps in Results”, ULS Research Methods Workshop http://ryanrwatkins.com/na/naessentials.html, 9 Where can we use a needs assessment? Strategic Action 9a : Ground Floor Needs Assessment (obviously) Opportunity to rethink the ground floor Many assumptions and ideas for what's best Gather information from stakeholders, relevant resources Make an informed decision to best meet the needs of organization and community Other opportunities? ULS Research Methods Workshop 10 12 Steps Brought to you by the National Oceanic & Atmospheric Administration http://csc.noaa.gov/needs/ ULS Research Methods Workshop 11 Needs Assessment Steps 1. 2. 3. 4. 5. 6. Confirm the Issues & Audience Establish the Planning Team Establish the Goals and Objectives Characterize Your Audience Conduct Information & Literature Search Select Your Data Collection methods 7. 8. 9. 10. 11. 12. Determine Your Sampling Scheme Design and Pilot the Collection Instrument Gather & Report Data Analyze Data Manage Data Synthesize Data & Create Report ULS Research Methods Workshop 12 1: Confirm Issues & Audience • Summary: • • Questions: • • • Establish purpose of assessment and summarize Is this a new issue or audience for the organization? Is there agreement up the organizational chain this issue or audience needs to be addressed? Risks: • • • Unknown stakeholders Communication issues LACK OF SUPPORT ULS Research Methods Workshop 13 2. Establish the Planning Team • • • • • All steps should be conducted with planning team Ideally members of stakeholder groups and experts Communication is essential Consider resource needs: time, expertise etc. Questions: • • • • Are the stakeholders new or well known? How will geography impact planning? Expertise within the team? Risks • • • Sense of ownership Communication Planning can overtake action ULS Research Methods Workshop 14 3. Establish Goals and Objectives Summary Questions: Identify the desired and actual levels of knowledge or skill Identify cause(s) for the lack of knowledge or skill Devise solution(s) Are the goals widely shared by the audience? Are your objectives measurable? Will the project be considered a success if the objectives are met? Risks: Potential to disengage by different team members who have different priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments ULS Research Methods Workshop Play Time 16 4. Characterize Your Audience Summary Questions: Sample size Skill & knowledge level Educational Level Attitudes and biases How long have you worked with the audience? How much variation is there within the audience? Risks: Assumptions can backfire Broadly generalizing and audience can lead to failure ULS Research Methods Workshop 17 Topics for Audience Characterization Knowledge Training Tools & Techniques Benefits Attitudes & Biases Ability to Attend or Access Cultural Characteristics ULS Research Methods Workshop 18 5. Information & Literature Search Summary Questions: Environmental Scan Readings Surveys to address questions raised in review Has this audience or issue been surveyed in the past? What other ways might we find useful information about this audience or issue? Risks: Results may be outdated Redundant effort, wasted time Missing important information that may make results easier or better ULS Research Methods Workshop 19 6. Select Your Data Collection methods Summary Questions: Observation Personal Interviews Surveys Focus Groups Have all methods been considered? Consider audience characteristics when drafting collection methods How much expertise is there in-house? Seek help! Risks: Can be intrusive or upsetting Time spent designing vs time spent analyzing ULS Research Methods Workshop 20 7. Determine Your Sampling Scheme Summary Sample more than you think you need Questions: What is statistically recommended? What is the population size of audience? Risks: Likely not scientifically sound Too many or too few in sample ULS Research Methods Workshop 21 Exercise: Develop Your Goals Discuss how you would approach Audience Characterization Data Collection Methods Sample Size Measure of Success ULS Research Methods Workshop 22 8. Design and Pilot Your Collection Instrument Summary Questions: Pilot first to identify weaknesses How will you pilot your instrument? What kind of expertise is on your planning team? How important is statistical precision? How will data collection be standardized? Risks: Instrument will not be clear or gather necessary data The audience will not be receptive to the survey instrument Asking too many questions may irritate the respondents ULS Research Methods Workshop 23 9. Gather and Record Data Summary Questions: Find means to incentivize respondents (survey prizes, etc) Ensure anonymity if this is a priority to audience Coordinate with assessment team If audience is outside of library, make use of External Communications group Will you recruit people outside planning team to assist in data gathering? Are you getting the desired response rate? Risks: Data will be biased Language or vocabulary issues Invalid Study ULS Research Methods Workshop 24 10. Analyze Data Summary Questions: Keep findings and interpretation of findings separate in reporting Seek assistance in interpreting data if necessary Risks: Under or over-whelming audience with report Missing trends and patterns Not accounting for possible critical barriers Letting bias slip into the process ULS Research Methods Workshop 25 11. Manage Data Summary Questions: This step involves determining how data will be organized and archived. The importance of this step is often not recognized until it is too late. Will this data ever need to be referred to again? Will the raw data be stored? How will it be stored? Can/should the data be shared? Risks: There may be unforeseen reasons that would necessitate the data being used again Inadequate metadata ULS Research Methods Workshop 26 12. Synthesize Data and Create Report Always address your goals and objectives in synthesis. Report must include problems or errors with the design and the implementation of the survey. An executive summary is often helpful. Questions: Who is the audience for the report? Did you address your objectives? Risks: Potential to disengage by different team members who have different priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments ULS Research Methods Workshop 27 Gather your Data/Report ULS Research Methods Workshop 28 Thank you! Works Cited: http://www.cscnoaa.gov/needs http://ryanrwatkins.com/na (includes link to free NA eBook) Additional Reading: http://en.wikipedia.org/wiki/Needs_assessment ULS Research Methods Workshop Survey Basics and Survey Monkey John Fudrow ULS Leadership Program 20 November 2013 30 When to Use a Survey Explore a Topic Discussion with Target Population Gather Objective Data on Subject Benchmarking of Service Levels ULS Research Methods Workshop 31 Survey Monkey Templates Over 150 pre-made templates Variety of topics Community (2) Customer Feedback (7) Demographics (9) Education (20) Events (7) Healthcare (14) Human Resources (18) Industry Specific (58) Just for Fun (9) Market Research (12) Non-profit (5) Political (11) ULS Research Methods Workshop 32 Surveys We Have Applied ULS General Survey Library Instruction Survey E-book Survey Event Feedback Surveys Internal Planning ULS Research Methods Workshop 33 Types of Survey Questions Open-Ended Closed-Ended ULS Research Methods Workshop 34 Types of Survey Questions Open-Ended i. Single Textbox ii. Multiple Textboxes iii.Comment/Essay Box iv. Numerical Textboxes v. Demographic (U.S. or International) vi. Date and/or Time ULS Research Methods Workshop 35 Types of Survey Questions Close-Ended i. Multiple Choice (Only One Answer Allowed) ii. Multiple Choice (Multiple Answers Allowed) iii.Rating iv. Ranking v. Matrix of Choices (Only One Answer Per Row) vi. Matrix of Choices (Multiple Answers Per Row) vii. Matrix of Drop-down menus ULS Research Methods Workshop 36 Likert Scales and Ratings What is a Likert Scale? A scaled response of a respondents feelings toward a topic based on a presented scale. ULS Research Methods Workshop 37 Likert Scales and Ratings Clear wording of response choices. Number of choices to relevancy 5 to 7 choices allow for a consistent distribution. The third choice should be the undecided or neutral decision. Be careful not to force ranking by the wording of the base question. ULS Research Methods Workshop 38 Anonymity • IRB Authorization Service Improvement Anonymous Ratings • Do not store data publically ULS Research Methods Workshop 39 Survey Delivery Target your audience Multiple avenues for large samples Create Multiple links to analyze effectiveness of delivery methods DON’T SPAM You cannot mass email ULS Research Methods Workshop 40 Statistical Relevance and You We are not statisticians We don’t have the resources nor time to run full analysis Our reports are not statistically relevant Our analysis is focused on report fulfillment We don't properly sample Our audience is often “expert” library users This alters the influence of their input We don't run via SPSS Statistical Package for the Social Sciences ULS Research Methods Workshop 41 Using Survey Monkey (Free Version) https://www.surveymonkey.com/ Create a free account ULS Research Methods Workshop 42 Survey Monkey (Free Version) Limitations 10 Questions 100 Responses No Page Logic 20 Less Templates No Customization ULS Research Methods Workshop 43 Survey Question Creation Create 2 questions you would like to ask of your department or patrons Think about how you would use the results and write that idea in the page description. We will look at several and discuss their efficacy. ULS Research Methods Workshop Focus Groups: the Method and How to Use It Karen Calhoun ULS Leadership Program 20 November 2013 45 Learning objectives 1. 2. 3. 4. Know the advantages and disadvantages of focus groups and when to use them Be able to select and recruit focus group interviewees Know how to prepare a focus group interview guide Have information about: The roles of the facilitator and recorder How to analyze focus group interview data ULS Research Methods Workshop 46 Focus Groups, Defined Qualitative social science research method Feature open, interactive, in-depth small group discussion (typically 6 to 10 people), led by a trained moderator/facilitator Participants carefully selected Evaluate participant thoughts, opinions, practices, values, beliefs, feelings in a nonthreatening, semistructured setting ULS Research Methods Workshop 47 Purpose of a focus group IT IS TO: Collect information and ideas on a pre-selected topic IT IS NOT: To achieve consensus To answer participants’ questions (facilitator ≠ sage on stage) To solve one or more problems To make decisions To provide a forum for participants to gripe or vent frustration (group therapy) ULS Research Methods Workshop 48 Advantages of Focus Groups Generate CC BY NC Francois Proulx http://www.flickr.com/photos/91569742@N00/404909051/ insight into not just what people think or feel, but why Can be comparatively low cost method research method Interactive, not one way Can be used alone or in combination with other research methods ULS Research Methods Workshop 49 Disadvantages/Risks of Focus Groups Findings not representative of entire population of interest Quality of results highly dependent on: Appropriate preparation Skill of moderator Skill of recorder Skill of analyst (preparation of report) Can be challenging to evaluate responses to openended questions ULS Research Methods Workshop 50 Steps to Set Up Focus Group Interviews Decide how many focus group interviews to hold (usually from two to a dozen) Select appropriate facility (see next slide) and time(s) for the event(s) Decide on participant incentives Recruit participants (usually 6 to 10 people in each focus group) Prepare interview guide/script Assign moderator and recorder Conduct the focus group(s) Analyze and report results ULS Research Methods Workshop 51 Focus Group Facility – Minimum Requirements Comfortable room in a convenient location for participants Date and time convenient for participants (faculty or peers = during work hours?; undergraduates = evening?) Table and chairs set up for group conversation Food, refreshments Permission to record the session Suitable means for moderator to conduct the interview (e.g., flip charts, markers, laptop, projector, screen …) The means to record the session (notes + audio recording is one way; another is streaming video to a separate room where notetakers/observers sit) ULS Research Methods Workshop 52 Focus Group Facilities – High End Text: Summary of information provided by a commercial focus group facility – a local one is Direct Feedback Inc. (Pittsburgh area): http://www.dfresearch.com/focus.htm Facility and technical capabilities: •Video streaming •Wireless •Comfortable, well equipped respondent room •On-staff moderating •Assistance with recruiting, audio and video recording … •Client viewing room with oneway mirror and closed-circuit television ULS Research Methods Workshop 53 Recruiting Participants – Selection Choose participants who will be comfortable with one another (but ideally, do not know each other) – “homogeneous strangers” Potential selection criteria: Knowledge of topic Comfortable discussing the topic with others Demographics - gender, age, etc. - if relevant to the topic Status , if relevant to the topic (role/target audience, e.g. faculty, student … or user of service, non-user of service … ) If group members are known to each other, avoid having all members of a clique in the same focus group if you can Also in groups known to each other, try to avoid having an employee and his/her boss in the same group ULS Research Methods Workshop 54 Recruiting Participants – Recruitment Process Sample flyer recruiting volunteers Participants might be: Nominated Randomly selected Members of a definable group Have same job/title Volunteers who fit selection criteria Other Screen potential participants/assign to groups Once you have grouped recruits, confirm participation (see sample letter, next slide) Send reminder 2 days before Over-invite by 10 to 20% (to account for no-shows) Extract from Eliot and Associates 2005 ULS Research Methods Workshop 55 Sample confirmation letter Track invitation emails and responses (along with demographic info you may need later) Gain consent to record session (“Although the session will be recorded, your responses will remain anonymous and no names will be mentioned in the report.”) Ideas for incentives (besides refreshments): Monetary Coupons/gift cards Door prize (drawing at end of session) …? Extract from Eliot and Associates 2005 ULS Research Methods Workshop 56 Exercise #1 Situation You’ve been given the following assignment: Conduct focus groups of the Pitt community to examine attitudes toward, awareness and usage of library instruction services (inside and outside the library, onsite and virtual) Exercise Working as a group at your table, take 10 minutes to sketch out: 1. 2. How will you determine who should be in the focus groups (i.e. the selection criteria)? Once you have selected the criteria, what methods might you use to recruit participants and assign them to groups? Report out (1-2 minutes each table) ULS Research Methods Workshop 57 Preparing and Using an Interview Guide(Script) Prepare 1. pre-group paperwork – consent forms, brief demographic information Opening – engagement questions 2. 3. Introductions, ground rules (see sample, next slide), opening question (make it an easy one; could be round robin) Exploration questions (2 to 4 key questions) Exit/ending questions (e.g., “Of all the things we discussed, what is the most important to you?” and/or “Have we missed anything in our discussion today”?) ULS Research Methods Workshop 58 Sample Introduction and Ground Rules (Interview Guide) Extract from Eliot and Associates 2005 ULS Research Methods Workshop 59 Interview Guides Use Open-Ended Questions Open-ended questions = inquiries that produce rich data = they start conversations and keep them going “When you think about green energy, what comes to mind?” “What do you like most about coming to the library?” Closed-ended questions = impose answers (yes or no, or a choice from a list) Help clarify and confirm Typically used in quantitative research like surveys ULS Research Methods Workshop 60 Other Possibilities for Designing Focus Group Interview Guides They don’t always have to be questions … Can show a brief video/make a proposal then start asking questions Can introduce alternatives and ask them to choose Fill in the blank Draw a picture More ULS Research Methods Workshop 61 Exercise #2 Situation You’ve been given the following assignment: Exercise Conduct focus groups to evaluate user reaction to the high-tech group study rooms on the second floor of Hillman Working at your table, and using the handout from Richard Krueger: Take 5 minutes to select or create 3 open-ended questions for the focus group interview guide Take another 5 minutes to consider how you will open and close the focus group interview Each group report out (1-2 minutes) ULS Research Methods Workshop 62 Help for Facilitators and Notetakers Available from Behind the Scenes, FY14 Planning and Budget Cte. Site Preparation for October 11 “Birds of a Feather” Event Karen Calhoun October 7, 2013 This PPT covers the roles of facilitators and recorders; See also “further reading” at the end ULS Research Methods Workshop 63 You, Moderating the Next Focus Group ULS Research Methods Workshop 64 Analyzing Focus Group Interview Data Is often not easy Devise a method of capturing the comments from the notes/recordings Capture the comments Look for common categories or themes Assign categories to comments Sort comments by category or theme Synthesize Summarize findings for each category/theme Prepare and present report One method described @ Eliot and Associates 2005 (see Methods last slide) ULS Research Workshop 65 Recommended Reading for Focus Group Designers, Facilitators, Recorders and Analysts Connaway, Lynn Silipigni, Debra Wilcox Johnson, and Susan E. Searing. 1997. “Online Catalogs from the Users’ Perspective: The Use of Focus Group Interviews.” College & Research Libraries 58 (5): 403–420. Dixon, Jill. 2005. “Focus Group Facilitation Guidelines”. Centre for Higher Education Quality. Monash University. http://www.uwsuper.edu/cipt/exsite/upload/Focus_Group_Guidelines.pdf Eliot and Associates. 2005. “Guidelines for Conducting a Focus Group.” http://assessment.aas.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf FocusGroupTips.com. 2012. “Focus Group Questionnaire Fundamentals: Basic Questions.” http://www.focusgrouptips.com/focus-group-questionnaire.html Krueger, Richard A. 2002. “Designing and Conducting Focus Group Interviews.” http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf Walden, Graham R. 2006. “Focus Group Interviewing in the Library Literature: A Selective Annotated Bibliography 1996-2005.” Reference Services Review 34 (2) (April 1): 222–241. doi:10.1108/00907320610669461. ULS Research Methods Workshop Personas Jeff Wisniewski ULS Leadership Program 20 November 2013 67 Road Map What’s a persona? Benefits of personas How to create personas How to use personas ULS Research Methods Workshop 68 Personas Personas are “stand ins” or hypothetical archetypes created to represent the primary user segments for your web site Each persona represents a key user type that shares demographic characteristics, needs, behaviours, and environment ULS Research Methods Workshop 69 Personas Imaginary, derived from user research Each has a name and personal details ULS Research Methods Workshop 70 second year graduate student in Biological Engineering currently splits her time between class work, time in the lab, and studying from home since much of her work is either course driven or in the lab, she does not consider herself a particularly heavy library user tends to rely on lectures rather than library resources uses company websites quite frequently for information on the lab products she uses and uses professional association sites for recent papers and information on developments in her field uses the popular search engines initially to get a sense of what types of materials are out there then moves to Web of Science and Compendex for access to journal articles Amy (Soo-Jin) ULS Research Methods Workshop 71 if she can avoid going to the library, she will at home she always connects through remote access to get access to full articles through the databases for which the library has subscription interested in doing exhaustive searches for journal articles on her dissertation topic no one has shown her how to use the full breadth of the resources and functionality of e-Journal on the library web site; she has a sense there are more resources and tools than she knows about uses ILL often to gain access to articles that she cannot access through Pitt subscriptions ULS Research Methods Workshop 72 Customer Segments to Personas Originated in the 1930 when marketers started using fictional characters to represent a customer segment Alan Cooper, a software developer, coined a related term and similar practice: personas. His book The Inmates are Running the Asylum popularized the use of personas and designing for “archetypal users”. ULS Research Methods Workshop 73 Benefits of Personas ULS Research Methods Workshop 74 #1 “User” Centered Design Way to have users attend all your design meetings Each persona has the weight Personas are based on and embody what we know about our library’s web site users Personas keeping it about the user ULS Research Methods Workshop 75 ULS Research Methods Workshop 76 #2. Support Evidence Based Decision Making Way to harness the user research data to inform web site development Easier to remember a persona than pages of facts and figures: path data, survey results, interview summaries etc. Share abstract data in a compelling and memorable way Personas encapsulate evidence ULS Research Methods Workshop 77 #3 Where to Put Design Effort Personas spell out what the site must do to support each personas’ goals and tasks Personas provide focus ULS Research Methods Workshop 78 #4. Communicate to Stakeholders in a Language Understood by Everyone Easy and fun way to communicate design decisions Keeps the focus on the user Avoid “geek” speak Personas speak to everyone ULS Research Methods Workshop 79 #4 Build Consensus and Commitment to the Design Communicate a common direction Reduce the need for extremely detailed specifications. Nuances of behaviours and preferences are captured in the persona and narratives Personas build shared vision ULS Research Methods Workshop 80 How to Create Personas Research! Environmental scan Interviews Ethnographic research ULS Research Methods Workshop 81 Environmental Scanning Identify true peers Literature review Web search Provides a framework ULS Research Methods Workshop 82 Ethnographic Research Gorillas in the Mist Time consuming Expensive Highly useful! ULS Research Methods Workshop 83 Interviews Useful for creating from scratch and for local validation of “borrowed” ULS Research Methods Workshop 84 Finding Subjects Leverage your networks General advertising not useful ULS Research Methods Workshop 85 Logistics Voice recorder of smartphone app Transcription Analysis ULS Research Methods Workshop 86 Interview Intro Explain hat you’re doing and why You’re helping us build a better website Be candid General computer usage habits When you start your browser where is the first place you go? Favorite sites, and why? ULS Research Methods Workshop 87 Information-Seeking Habits If you need to: Find books to take on vacation Write a paper …Where’s the first place you’d go? ULS Research Methods Workshop 88 Library Questions How often do you go to the library? How often do you go to the library website? When do you go to the library website? What do you do when you’re there? (Take them to site) What immediately draws your attention? What information did you look for but not find? Is there something you looked for on the homepage but didn’t find? ULS Research Methods Workshop 89 Now What? Interview Create Refine ULS Research Methods Workshop 90 How Many Personas ? Primary constituencies 5-7 generally recommended ULS Research Methods Workshop 91 How To Use the Personas In the room Frame discussions ULS Research Methods Workshop 92 Exercise In groups, create a persona based on one of the group members Scenario: the personas created will be used to guide a user centered redesign of the ULS’ SharePoint site ULS Research Methods Workshop 93 Questions? ULS Research Methods Workshop 94 Thanks for coming! Please fill out the post-workshop evaluation survey at: https://www.surveymonkey.com/s/FS8PFXK ULS Research Methods Workshop
