Technology and PE Evaluation
Dr Eric Jensen (e.jensen@warwick.ac.uk)
My Background
• Sociology, communication and psychology (PhD in sociology from University of Cambridge).
• Research specialism in evaluating public engagement quality of experience and impacts.
• I am a sociology lecturer at the University of Warwick, where I teach research methods (quantitative and qualitative) Professional Background
External evaluation experience across a wide range of settings: – e.g. for University of Cambridge Public Engagement, National Gallery, Imperial War Museum, Manchester Science Festival, British Museum, Cheltenham Literature & Science Festival, Cambridge Science Festival & Cambridge Festival of Ideas, Durrell Wildlife Conservation Trust, London Zoo, Natural History Museum, National Marine Aquarium, World Association of Zoos and Aquaria, University of Cambridge Museums, etc.
– Particular focus on evaluating public engagement impacts through festivals and events.
Forthcoming books:
– ‘Doing Real Research’ (SAGE)
– ‘From Conservation Education to Public Engagement: Research, Principles and Practice’ (Cambridge University Press)
– ‘Making the Most of Public Engagement Events and Festivals’ (Cambridge University Press)
3
Context for Public Engagement Evaluation: ‘More is better’ changing to ‘quality and impact’?
Why Evaluate?
To build a better understanding of your visiting publics, (e.g. needs, interests, motivations, language).
To inform your plans and to predict which engagement or learning methods and content will be most effective.
To know whether you have achieved your objectives (and why or why not).
To re-­design your approach to be even more effective in future.
Are there limits to what evaluation can tell us about impacts?
– With right methods and approach, I think it is possible to measure most outcomes that would be of interest.
– Limits therefore are in terms of:
• Capabilities / methods knowledge and skill
• Time horizon and available resources
Overview of key challenges pertaining to public engagment evaluation
‘Industry-­‐standard’ visitor surveys and evaluation procedures at science communication institutions often display basic errors and poor practice in design, sampling and analysis.
Many institutions are uncritical consumers (and producers) of evaluation, quick to believe measuring complex outcomes is simple.
As a result:
– Questionable data and conclusions have been fed into the field of public engagement for years. – Gut-­‐based decision-­‐making predominates in public engagement practice
Defining Evaluation
Evaluation = sub-­‐category of 'social research' (thus all principles of social research apply)
Distinguishing feature of evaluation: Focus on objectives / claimed outcomes (practitioners must specify these outcomes)
In order to evaluate them, practitioner objectives should be Specific, Measurable, Achievable, Realistic and Targeted. (Beware of ‘Raising Awareness’ and ‘Inspiring Interest’!)
Impact Evaluation
BEFORE
AFTER
Impact Evaluation: Defining Impact
Impact is the overall net outcomes or results of an activity or intervention (intended or unintended) for individuals or groups
Note that changes or ‘impacts’ can be in negative or dysfunctional directions!
Impacts could include:
Development in learning about a specific topic
Attitude change
A greater sense of self-­‐efficacy
Enhanced curiosity or interest in a subject
Improved skills or confidence, etc.
Good Impact Evaluation
Is SYSTEMATIC
Tells you how and why particular aspects of activity are effective
• NOT a binary ‘good’ / ‘bad’ or ‘successful’ / ‘unsuccessful’ result
You don’t learn anything from binary results
• A ‘successful’ project can always develop the good aspects of their practice further
• There will be specific aspects of an ‘unsuccessful’ project or method that were ineffective (and should be avoided in future projects)
• Either way, it is important to have some specifics!
Getting started on impact evaluation: Reviewing the Toolkit
The Toolkit
•Qualitative Evaluation Methods
•Quantitative Evaluation
Methods
•Combined Evaluation Methods
The Qualitative Toolkit
BEFORE
AFTER
What is ‘Qualitative’ Evaluation?
Data and results are not numbers.
Could be:
– Words
– Images or Drawings
– Video
– Concept Maps
– Emotions
– Etc., etc., etc.
The Qualitative Toolkit
•Interviews
•Focus Groups
•Accompanied Visits and Ethnography
•Qualitative Surveys •Qualitative Analysis of ‘Content’
(e.g. analysing information panels)
Why use Qualitative Evaluation?
Qualitative evaluation goals centre on terms like discovery and exploration.
Allows for flexibility in evaluation design
– Very useful when not much is known about the topic.
– ‘Emergent’ evaluation design strongly encouraged.
– Potential for depth and richness of understanding.
Qualitative Questions
What is the range of experiences or responses that our visitors have at a particular exhibit?
How are publics making sense of the content that we are trying to communicate?
Qualitative Data Analysis
-­‐ Must be systematic to avoid tendency to select quotes based on personal bias and preferences.
-­‐ Can convert qualitative data into quantitative data through content analysis
Indicators of Robust Qualitative Evaluation
Allow for Possibility of Negative Outcomes (not cherry picking positive quotations)
Systematic Data Analysis
–
Must be systematic to avoid tendency to select quotes based on personal bias and preferences.
Quantitative Evaluation Options
Quantitative Evaluation Methods
Used to answer any counting related question: How many? What proportion?
• Most likely to be survey methods (alternatives would be experimental design or direct observation)
– Primary Data Collection
Analysis conducted by the individual or institution that collected the data.
– Secondary (Data) Analysis
Any further analysis of an existing dataset that produces results or conclusions other than those produced by the individual or institution that collected the data. The Quantitative Evaluation Toolkit
BEFORE
AFTER
Measurement
– A key issue is what will be captured on a particular measure (i.e. ‘what counts?’)
– Measurement error is an issue. (i.e. error due to measurement approach/tool)
– Where feasible, important to directly measure relevant variables such as knowledge, e.g. before/after
Key threats
• Response Bias: A bias in data due to the survey instrument rather than respondents’ actual beliefs
• Sampling Bias due to non-­random sampling: Unintentional sampling of subjects that introduces systematic error or bias into the results
Using surveys for impact evaluation
What is a questionnaire?
Standardized method of data collection.
Can be used for both qualitative and quantitative data.
Used to collect data from individuals, not groups or on behalf of someone else.
Surveys are often used for gathering information about recent actions and experiences, or current thoughts and opinions.
What are surveys good for?
Can be used for describing patterns in a large population.
Can determine individuals’ characteristics.
Can be used to assess general population patterns from individuals’ perspectives.
Can compare the perspectives and effects of an intervention on different sets of individuals within a population.
Using Qualitative Items in Surveys:
Alternative Approaches to Impact Evaluation
• Thought-­listing
• Drawings
• Concept Maps
Pre-­ and Post – visit (repeated measures) survey components included:
Annotated drawing of ‘favourite wildlife habitat’ with all the plants and animals which live there
Annotated Diagrams of favourite habitat
Analysis of all paired forms – scoring on basis of 1-­‐3
1= negative change in accuracy of representation (animals/habitat)
2= no change in accuracy 3 = positive change
Pre-­session
Post-­ session
3
Another example of quantitative impact evaluation using open-­‐
ended survey questions
Measuring Biodiversity Literacy in World Zoo and Aquarium Visitors
Aichi Biodiversity Target 1
Target 1: “By 2020, at the latest, people are aware of the values of biodiversity and the steps they can take to conserve and use it sustainably.”
Participating institutions
Survey Design
Single-­‐page design with three main components:
1. Basic demographic information.
2. Two main outcome variables, each measured by open-­‐
ended questions:
• Biodiversity Understanding
• Knowledge of actions to protect Biodiversity
3. A number of potential independent variables also measured.
Pre-­‐visit Survey
Post-­‐visit Survey
Measuring the outcome variables
To measure biodiversity understanding: ‘Please list anything that comes to mind when you think of ‘biodiversity’ (space for up to five responses)’.
To measure knowledge of actions to help protect biodiversity: ‘If you can think of an action that you could take to help save animal species, please list below (space for up to two responses)’.
Data Processing and Analysis
• Dependent variables were content analysed to produce quantitative data:
Ø Biodiversity understanding/literacy -­‐ scored along a continuous scale of understanding* Ø Knowledge of actions to protect biodiversity –
were scored along a continuous scale of personal action^
Inter-­‐coder reliability (Cohen’s Kappa): *= 0.82; ^=0.84
Analysis of biodiversity understanding
1 -­‐ Inaccurate: completely inaccurate descriptions (no accurate elements) –
e.g. ‘open air’, ‘everything in general’ – and/or too vague to indicate accurate knowledge of any kind – e.g. ‘many things’.
2 -­‐ Ambivalent: some evidence of accurate descriptions, some of inaccurate descriptions.
3 -­‐ Some positive evidence: mention of something biological (e.g. ‘species’) but no other accurate elements or detail.
4 -­‐ Positive evidence: some evidence of accurate descriptions, but (1) only mentioning animals or plants, not both (minimal inaccurate elements) and/or (2) using a vague but accurate description – e.g. ‘lots of life’, ‘many species’, ‘variety of species’.
5 -­‐ Strongly positive evidence: strong evidence of accurate descriptions, specifically mentioning both plants and animals (no inaccurate elements) –
e.g. ‘variety of animals, fish and insects’, ‘loss of habitat’, ‘shared environment’, ‘wildlife and plant life in balance’.
-­‐99 -­‐ Missing: no thought-­‐listing data provided; excluded and marked as missing data.
Analysis of conservation ‘actions’
(0) Action or behavior identified not relevant to conservation.
(1) Vague platitudes about need for change (no specific action or behavior mentioned) – e.g., “save ecosystems”.
(2) Specific identification of pro-­‐biodiversity action or behavior, but is at a general level (not feasible to address as an individual) – e.g., “stop hunting”, “stop Chinese medicine”.
(3) Very specific identification of pro-­‐biodiversity action or behavior that can be done at an individual level – e.g., “drive less to reduce effects of climate change”.
(4) Respondent clearly states a personal action or behavior –
e.g., “I recycle my mobile phone for gorillas”.
Headline Results
Significant aggregate increases between pre-­‐ and post-­‐visit in biodiversity understanding and knowledge of actions to help protect biodiversity
Headline Results
• Number of respondents demonstrating at least some positive evidence of biodiversity understanding: increase from pre-­‐visit (69.8%) to post-­‐visit (75.1%) • Number of respondents that could identify a pro-­‐
biodiversity action that could be achieved at an individual level: increase from pre-­‐visit (50.5%) to post-­‐visit (58.8%) Evaluating public engagement impacts with quantitative methods: Pitfalls and Top Tips
Common problems with public engagement evaluation:
Research Design Issues when Evaluating Impacts
Survey Design for Evaluation: Avoiding bad decisions
Designing a good survey
A good survey should:
– Ask questions that allow you to find out as much as possible about your area of research.
– Do not use more questions than you need to address your research question.
– Use standardized, relevant questions that can be understood by your respondents.
– Avoid bad question design, such as the use of biased and leading questions.
Designing a good survey
A good survey should:
– Be consistent in phrasing.
– Avoid using too many different question types.
– Keep questions brief.
– Use plain, easy-­‐to-­‐understand language.
– Minimize ambiguity in question and response options.
– Use a clear, legible font: e.g. Arial 11 or 12 point.
– Format your survey consistently: e.g. bold/italic.
Layout and Sequencing
Don’t include too many questions per page:
– This looks intimidating and may harm your response rate.
Question order can be important: keep it in mind. (e.g. ask open-­‐ended items first)
Questions should go from general to specific and from easy to hard:
– Exception: Consider saving demographic details for the end.
Surveys should be understandable
Survey questions and instructions should be clear.
Jargon and complicated wording should be avoided. Response categories should always offer a ‘don’t know’ option:
– Without a ‘don’t know’ option, respondents may provide inaccurate guesses or select a survey response that does not match their true views.
Unintended Cues can Influence Responses
Cues that you give to respondents can the affect opinions and thoughts they report.
Be careful not to influence responses by accidentally hinting about your expected outcomes, etc.
Unintended Cues
Unintended cues can be imbedded in: – the way questions are written, – survey layout, – whether other people are nearby with a verbally administered survey, – who is collecting the data, – what data collectors are wearing, etc.
Response Categories
Unclear questions or confusing response options may result in respondents: – Guessing.
– Selecting a ‘neutral’/‘don’t know’ option.
– Not answering the question.
Top tips for writing your survey
QUESTION DESIGN
Multiple-­‐choice questions: Select one response
This question type provides pre-­‐determined response options: Respondents must choose one
answer.
Key criteria for this question type is that response options should be:
– Exhaustive: everyone fits into at least one category.
– Exclusive: everyone fits into only one category.
– Unambiguous: response categories mean the same to everyone.
Likert scale questions
This question type should be used when the measured variable has multiple levels:
– E.g. levels of agreement, concern, confidence etc.
The scale should always have a neutral option:
– E.g. Strongly agree, agree, neutral, disagree, strongly disagree (also a ‘don’t know’/’no opinion’, etc.).
Likert scale questions
Science engagement indicators
[Impact Measures]
Scientific self-­‐
efficacy [Impact Measure]
7. Please indicate your Likert scale
level of agreement with the following statements:
I feel capable of understanding science.
Strongly Disagree (1) to Strongly Agree (7) and Prefer not to say or no opinion
Avoiding Survey Bias
Using a biased survey reduces the reliability and validity or your survey research.
You should try to avoid the various forms of bias when designing your survey:
– Editing, getting feedback and pilot testing are essential to reducing survey bias.
Type of Bias
What is it
Example
Researcher Expectancy Effect
Researchers unintentionally introduce bias by designing survey questions and response options around their existing assumptions.
A business’s customer service team expecting positive feedback might unintentionally bias their survey by asking leading questions. Acquiescence Bias
Respondents tend to agree with If all such Likert scale statements Likert scale (level of agreement) are framed positively, the results statements.
may skew towards agreement. Demand
Characteristics
Respondents may alter their answers based on what they think is the researcher’s preferred result.
Being asked to give feedback about a hospital by a uniformed hospital worker may result in more positive responses.
Social Desirability Bias
Respondents may over-­‐report views and behaviours that are widely praised in society and to make themselves look better.
Inaccurately reporting higher levels of recycling or charitable donations in order to appear more caring is typical of this bias. Table adapted from ‘Types of Survey Bias to Avoid’ from Doing Real
Research by Jensen, E. and Laurie, C. (SAGE, 2016).
Survey Design Flaws (Avoid!)
• Demand Characteristics: Participants will alter their responses in accordance with what they believe to be the evaluators’ expected results.
Ø This can happen when questions make the expected outcome clear, or other cues give away researchers’ expectations.
• Expectancy effect: When evaluators unintentionally bias results in accordance with expected results.
(e.g. by asking biased questions)
Survey Design Flaws (Avoid!)
continued
• Acquiescence Bias: A bias from respondents’ tendency to agree with statements
à Control for this by including reverse wording items on agreement scales (e.g. ‘I found the presentation confusing’)
“Put me down for whoever comes out ahead in your poll”.
Survey Design Flaws
Beware of social desirability bias Phrase questions e.g. about their prior knowledge or visiting experience in a way that respondents can answer truthfully without feeling stigmatized or awkward.
– e.g. ‘sure, I read all the information signs’.
Further Survey Biases to avoid
Double-­‐barrelled questions, 2 questions in 1:
– May have 2 different answers.
Leading questions:
– These reveal the researcher’s expected response.
Survey Biases from Self-­‐Report
Many surveys ask respondents to ‘self-­‐report’ information about events, beliefs or attitudes.
Self-­‐report allows for direct access to respondents’ views.
However, self-­‐report can be a source of bias:
– If they are asked to report on behalf of someone else.
– If they are expected to recall information from the distant past.
– If they are asked to predict future behaviour.
Tips for good survey design
• Label each of the response options you use to increase reliability (e.g. 1 – Strongly disagree, 2 – Disagree, 3 – Somewhat disagree, etc.)
• Don’t ask about events in the distant past if you want accurate recall
• Use the your respondents’ language wherever possible
Survey-­‐based Impact Evaluation –
Current Approaches
Common problems: – Oversimplification of impact measurement, e.g. relying on post-­‐visit only self-­‐report
– Proxy reporting
1) Over-­‐simplification
Many are quick to assume that complex concepts can be accurately evaluated through simple questions
Want to know whether a child has learned a lot about science after their day at the science museum? Easy! Just ask them:
‘Did you learn during your visit to the science museum today?’: Yes or No?
Over-­‐simplification (real example)
London’s Science Museum’s internal guidance for evaluation includes the following flawed survey item:
‘To what extent do you agree or disagree with the following statements?’ (Strongly agree to strongly disagree)
‘I have learnt something new today’ (National museum of Science and Industry)
Over-­‐simplification
When our hypothetical child above says ‘yes’ to the self-­‐
reported learning question, they are most likely telling the institution what it wants to hear. Relates back to issues of measurement – this question imposes the unrealistic expectation that respondents can:
– Accurately assess their pre-­‐visit science knowledge
– Identify gains or losses that occurred during the visit
– Accurately self-­‐report their conclusions on a 5-­‐point scale
Actually measuring learning requires (at minimum) direct measurement of visitors’ thinking or attitudes before and after the intervention (or an experimental design).
Over to you!
Response options:
1 -­‐ a lot; 2 -­‐ a fair amount; 3 -­‐ only a little; 4 -­‐ nothing; What is wrong with this?
Response options:
1 -­‐ great extent; 2 -­‐ considerable extent; 3 -­‐ moderate extent; 4 -­‐ slight extent; 5 -­‐ no extent.
What is wrong with this?
Proxy reporting of impacts
Parents reporting for children
Example: Evaluating California Science Center impacts on children
Falk and Needham (2011) sought to measure the Science Center’s impacts on children by asking parents to report on cognitive and affective outcomes. First, parents asked to indicate whether their children had gained an increased understanding of ‘science or technology’ after visiting the Science Center. Falk and Needham (2011: 5) reported that ‘nearly all adults (87%) who indicated that their children had visited the Science Center agreed that the visit increased their children’s science or technology understanding, with 45% believing that the experience increased their children’s understanding “a lot”’. Example: Evaluating California Science Center impacts on children
This survey item raises obvious issues surrounding the unreliability of expecting different parents within a sample to judge what counts as “a lot” of learning. Respondents will likely interpret “a lot” of learning in different ways.
Parents are being asked to provide one assessment regardless of the number of children they may have. – What if parents feel that one of their children learned “a lot”, while another learned “a little” and a third “nothing” at all? Are parents really likely to be making a considered judgment here? Asking parents to provide an off-­‐the-­‐cuff assessment of their child’s learning is highly prone to error, let alone the effects of events that may have happened months or years prior. 2) Parent Feedback
The visitor evaluation survey for the Edinburgh International Science Festival asked adult respondents: “What score would the children in your party give this event/activity(s) out of 10?”
Headline:
Teacher or parent opinion cannot be a valid proxy indicator of student/child impact on thinking, attitudes, etc.
Teacher Feedback Forms
Some teacher comments from a zoo evaluation that cannot be taken as accurate assessments :
“The kids loved it, and they didn't really think about how much they were learning as they looked around.”
“I think it's 100% educational as the Zoo is so involved with highlighting the importance of preserving ecosystems (even the cafes); also watching animals invariably increases understanding of them.”
Teacher Feedback Forms
All of the above are perceptions of the teacher, not measures of impact on the learners involved
So what can teachers validly provide to evaluations?
Valid teacher input:
Their opinions/satisfaction on a range of topics, for example:
ü The service they have received.
ü The education provision on offer and how/if they used it.
ü Comparisons with competing organisations.
ü Their professional opinion on any improvements to education provision.
Some ‘good’ teacher comments:
“The Applied Science sessions were spot on. Did you know the AQA Applied Science course is changing for 2012?”
“Frustrated by website. I tried to book online but could not book a group of 18 students and 2 staff. I found it hard to find the information that used to be really obvious.”
Looking to the future: What can be done to improve overall quality of evidence-­‐based public engagement practice?
Over to you!
Major causes of poor quality evaluation in public engagement?
Using automated technology in evaluation
NEW LOGO | MASTER OPTION 1
FONT MONTSERRAT
Why use technology for evaluation?
ü Less expensive than a market research company
ü Far less staff time needed vs. in-­‐house data collection (effectively no on-­‐going staff time)
ü Better quality data available (with expert input on evaluation/survey design) than is likely from either a market research company or in-­‐house
ü More extensive and timely data can be gathered than would otherwise be possible
ü Real-­‐time and automated, which provides you with answers as the data roll in, with no ongoing logistics to organise for data collection
ü Organisations own their own data
NEW LOGO | MASTER OPTION 1
FONT MONTSERRAT
Qualia project (qualia.org.uk)
Smartphone app-­‐based systems specifically are helpful for gathering feedback from audiences while they are still in close proximity to the event being evaluated
Prototype smartphone app was developed for audiences to improve their experience at the Cheltenham Science Festival
The app included numerous SC activities
Mobile and app-­‐based systems
Mobile technology proved particularly effective as a means of automating evaluation.
Micro-­‐surveying integrated within visitor smartphone app
Questions can be customised to organisations’ specific requirements and recurring events
‘Enjoyable?’
‘Confusing?’
Social Media Analysis
Sentiment analysis: Automated social media analysis (Scale: very negative to very positive)
Key issues
Social media content is not normal conversation data:
– Level of overlap of online and offline behaviour is largely unknown.
App evaluation features:
Scheduler to add events with automatic reminders
Micro-­‐survey designed to gather feedback about the experience through four short questions (e.g. ‘enjoyable?’)
Automatic feedback request was pushed to users when someone indicated they were attending an event
Questions were customized to organizations’ specific requirements and event types
Artory project (artory.co.uk)
Collaborative research and development initiative by ten cultural organizations in Plymouth
City-­‐wide ‘what’s on’ smartphone app listing of arts and culture events
Participation is voluntary but users are incentivized with ‘Art Miles’ for checking in at different venues and providing feedback data
These can then be redeemed at participating venues for a range of offers, from coffee or tea to discounted tickets and special ‘VIP’ access
Artory
To avoid over-­‐taxing app users, three different levels of feedback are used
Audience members invited to opt into further levels of depth in providing feedback in order to unlock greater incentives
The most basic level includes a matching pre-­‐ and post-­‐visit survey item evaluating expectations and outcomes:
Advantages of specifically app-­‐
based evaluation systems:
ü Allows for the same phrasing and survey structure across multiple data collection institutions
ü Allows measurement over time and across different cultural experiences in the same space (a city, a campus, a festival, etc.)
ü Respondents / users only have to enter profile information once instead of re-­‐entering their demographic data repeatedly over time and across different formats.
Advantages of specifically app-­‐
based evaluation systems (cont.):
ü Enables tracking of unique visitors – new level of knowledge about how specific cultural experiences fit within a wider range of experiences for an individual
ü Results can come in real-­‐time statistical data analysis, so the impact of science communication and any potential problems are seen and can be altered immediately if necessary by the institution
ü Responding to questions is simple, interactive and does not require researcher supervision
E-­‐mail / Web-­‐based systems
Case Example: Parque das Aves
Bird Park in Brazil Tablet to Collect Emails | Enrolment
Visitors complete registration survey on iPad. This short survey designed primarily to collect visitor e-­‐mails. Data collection process entirely in language of participant.
Short On-Site Form (Survey 1)
Example of how the initial on-­‐site survey page looks to visitors. The 'Email' field is important, but It also has room for error. For this reason, it was the only required field. This is validated by the system.
Emails for visitors are queued by the system once this form is complete.
Emails to Visitors | Invitations to Survey
Visitors receive an email from the Branded emails are sent organisation automatically. from organisation’s email Email invites them to participate in address. the main survey.
Emails are sent out at end of business each day, 2 hours after last entry.
Based on language selection, visitors are sent communications in one of three languages: Brazilian Portuguese, Spanish and English)
Survey 2 | Device Accessible Visitors able to access the e-­‐mail invitation and complete the main survey from any internet enabled device (responsive design).
Tablet
Personal computer
Smartphone
Laptop
Survey 2 | Responsive design
The main survey is fully responsive to any screen size.
Likert-­‐Scale on a Large Screen; Shown as ‘multiple choice’ options. Likert Scale on a Small Screen; Shown as ‘drop down’ options.
Dashboard
Access your data at any time through the dashboard
Automatic Analysis (structured in advance)
Analytics that have been decided in advance are automatically generated and available on the dashboard under ‘Survey Analytics’ in the menu at any time
Access your Data | Dashboard
Overview
Analytics visible on Dashboard display for available surveys.
Sidebar contains active surveys and data analytics.
Questions located in appropriate sections. Comparative analysis and benchmarking available here.
Responses are
visualised into
the most
appropriate
charts based on
the questiontype and type of
data.
Pie Charts
Pie Charts
Hovering the mouse over areas of the pie chart shows the number of participants who gave a particular response
Bar Charts
Histogram
Likert scale results are colour coded red to green to make it clear when feedback is bad/good, with reverse-­‐
coded questions colour coded accordingly)
Crosstabs
Graphs are built from cross-­‐tabular analysis set up in advance. Clicking ‘see details’ will then show a chi squared table
Comparisons
Access to Data | Data Range and Download
Raw data file accessible at any time with Export options. You simply select date range and an automated e-­‐mail is sent with an Excel file attached. How can technology help with public engagement evaluation?
Example: Events
Feedback system for family events
Creative problem solving for data collection
2 models for logistics of data collection
Enrolment survey completed by visitor services during walk-­‐‑up sign up (when visitors given stickers) process
Enrollment survey completed by family assistant while circulating within event, gathering up email addresses from participants.
ALTERNATIVES – e.g. integration with booking form.
Sampling approach (ENROLMENT SURVEY)
Instructions for staff using this form
If you are collecting sign-­‐ups for an event, please complete this form for each person signing up (unless things get very busy, in which case you should follow a set rule e.g. every 5th or 10th person for the busy period)
If you are circulating at an event, please systematically select respondents based on a set rule (e.g. every 5th
person; or one person from each row for a seated event) depending on how much time you have and the goal for number of respondents.
Develop feedback questions
Please indicate your level of agreement with the following statements:
– The National Gallery family activity I participated in was sometimes confusing. – The presenter for the National Gallery family activity made me feel welcome.
– I talked about the paintings with my family after the workshop was over. – Some parts of the workshop were boring. – Overall, the family activity failed to meet my expectations.
– I was disappointed by the National Gallery family activity.
– I am a frequent visitor at the National Gallery
OVER TO YOU!
How can technology help with public engagement evaluation in your situation?
Limitations of Technology-­‐
Enhanced Evaluation for Public Engagement in General?
Limitations of technology-­‐
enhanced evaluation tools
Each digital technology brings its own patterns of participation/exclusion. Norris (2001) highlights three distinctive forms of digital divide: v ‘Global’ (between rich and poor nations)
v ‘Social’ (inequality within a nation)
v ‘Democratic’ (between those who use digital technology for civic, or ‘public’ purposes and those who do not)
Overcoming the limitations of automated evaluation systems
Digital divide is partly mitigated with an e-­‐
mail/online system (rather than smartphone), as these are more ubiquitous digital technologies (along with cellular text messaging). Minority language speakers have long had their voices excluded from audience research, therefore a means of routinely including this audience group certainly contributes to an inclusion agenda. Overcoming the limitations of automated evaluation systems
Low-­‐technology methods can also be exclusionary. Exclusion risk needs to be carefully weighed each time a new evaluation system is set up. Key issues to consider include: v The current audience profile v Alternative methods of gathering feedback or evaluating impact for those without the access to technology Over to You!
Limitations of Technology-­‐
Enhanced Evaluation for Public Engagement in your Context?
Conclusions
Automated evaluation systems can provide a financially sustainable means of bringing public engagement practice and evaluation findings closer together.
Robust understanding of audience experiences can enable public engagement practice to be better attuned to, anticipate and predict the changing needs and interests of audiences/publics.
Automating routine survey data collection and analysis could free up resources for more in-­‐depth qualitative research that it is not feasible to automate
Promise, quality problems and new solutions in public engagement evaluation: Towards evidence-­‐based practice
Dr Eric Jensen (e.jensen@warwick.ac.uk)
Further reading
vDawson, E. & Jensen, E. (2011). Towards a ‘contextual turn’ in visitor research: Evaluating visitor segmentation and identity-­‐related motivations. Visitor Studies, 14(2): 127-­‐140.
vGaskell, G., & Bauer, M. W. (2000). Towards public accountability: Beyond sampling, reliability and validity. In M. W. Bauer & G. Gaskell (Eds.), Qualitative researching with text, image and sound (pp. 336-­‐350). London: Sage.
vJensen, E. (2014). ‘ The problems with science communication evaluation’. JCOM: Journal of Science Communication, 1(2014)C04. Last accessed 20 August 2015 at http://jcom.sissa.it/archive/13/01/JCOM_1301_2014_C04/JCOM_1301_2014_C04.pdf.
vJensen, E. (2015). Enabling city-­‐wide audience evaluation: Plymouth Artory metrics development. Report published online. Last accessed 20 August 2015 at: https://www.researchgate.net/publication/281109028_Enabling_City-­‐
wide_Audience_Evaluation_Plymouth_Artory_Metrics_Development
Further reading
vJensen, E. (2015). Highlighting the value of impact evaluation: Enhancing informal science learning and public engagement theory and practice. JCOM: Journal of Science Communication.
vJensen, E., Dawson, E. & Falk, J. (2011). Dialogue and synthesis: Developing consensus in visitor research methodology. Visitor Studies, 14(2): 158-­‐161.
vNorris, P. (2001). Digital Divide: Civic Engagement, Information Poverty, and the Internet Worldwide. Cambridge: Cambridge University Press. vRowe, G. and Frewer, L. J. (2004). Evaluating public participation exercises: A research agenda. Science, Technology & Human Values, 29(4): 512-­‐556.
vThorne, S. (1997). The art (and science) of critiquing qualitative research. In J. M. Morse (Ed.), Completing a Qualitative Project: Details and Dialogue (pp. 117-­‐132). Thousand Oaks: SAGE.