EVALUATION OF THE
INTERNATIONAL WORKSHOP ON
NEW COMPUTATIONALLY-ENABLED
THEORETICAL MODELS
TO SUPPORT HEALTH BEHAVIOR CHANGE
AND MAINTENANCE
OCTOBER 16-17, 2012, BRUSSELS, BELGIUM
Prepared by Donna Spruijt-Metz, Ilkka Korhonen, Stephen
Intille, Wendy Nilsen, Niilo Saranummi and Heidi Lehtonen
Workshop background



In October of 2012, 30 invited researchers from the United
States and Europe, 4 representatives from NSF, NIH and the
European Commission, and 5 junior faculty/postdoctoral and
pre-doctoral students met for 2 intensive days in Brussels.
Our task was to forge a path for major discoveries and
innovations that would enable the use of existing and
emerging technology to its full potential to understand
human health behavior, change that behavior and maintain
behavior change. The full mission statement is
here:http://behaviorchange.be/index.php/mission/
The workshop was funded by NSF, NIH and the European
Union, and led by Donna Spruijt-Metz, Stephen Intille, Niilo
Saranummi and Ilkka Korhonen.
Evaulation




This evaluation was conducted via SurveyMonkey
There were 11 questions total.
27 participants were kind enough to complete the
survey.
The following slides summarize the responses.
1. What is your main area of
expertise/training?
(27 responses)
2. Prior to the workshop, what were your major
goals or expectations for the workshop?
(27 responses)







New, innovative solutions, strategies, ideas, future
directions – thinking outside the box (8 people)
Multidisciplinary and multicultural views, meeting experts in
different fields (8 people)
Learning more about state of the art, where we are now,
key problems currently (7 people)
Meeting people, networking, new connections (7 people)
Modeling behavior change; link between computer models
and behavior (6 people)
Evaluation methods, new methods for testing theories (2
people)
Policy development; White paper; Intellectual stimulation,
challenge to existing ideas; Learn about technology; Learn
about behavior change; Big data; Change the world
(Other responses)
3. Were your goals met?
(27 responses)
The workshop did not meet my goals or
expectations
The workshop met some of my goals or
expectations
The workshop met most of my goals or
expectations
The workshop exceeded my goals or
expectations
0%
14,8 %
40,7 %
44,4 %
Distilled comments: We spent much time on synchronizing our dictionaries and ways of thinking
It takes time for the group to learn to communicate across disciplines
I don't think we made as much progress on thinking about how to evaluate such models and
technologies based on them
Very hands-on, very productive!
We maybe need after this workshop a more pragmatic approach
“the pace of the work was refreshingly aggressive” “amazing workshop, very well organized”
“A great group of people!”
Full comments Question 2: page 1of 2





Eagerly awaiting the white paper!
I think that we spent most of our time on synchronizing our dictionaries and ways of
thinking, because it was a multi-disciplinary workshop. I think we need to obtain
some sort of common language before we can actually reach the goal that I
described above to satisfaction.
For the most part, the workshop more than met my expectations. By the end of the
workshop, there seem to be a general consensus that we need better real-time
individualized models of behavior change, and that control systems theory and
agent-based modeling can help move us in that direction. I don't think we made as
much progress on thinking about how to evaluate such models and technologies
based on them, however.
It was just very hands-on, very productive!
I think we had great discussions, but we need to continue to have them. It takes
time for the group to learn to communicate across disciplines. We made great
progress.
Full comments Question 2: page 2 of 2









Overall, I was very satisfied with the workshop.
I think it was excellent. It had been many years since I was in a workshop so
enriching
The caliber of participant and the pace of the work were refreshingly
aggressive
Great learning experience
It was an amazing workshop, very well organized, that really exceeded my
expectations.
The workshop was an excellent start
We maybe need after this workshop a more pragmatic approach, trying to
provoke small changes, make them public (best practices, social networks,...) and
after that, change the world :-)
I learned a lot and met some great people. I'm not sure I came out of it with a
full understanding of what is the state of the art in behavior change, but I left the
meeting with lots of next steps and reading to do.
A great group of people!
4. What were your most positive experiences (scientific,
social, cultural or personal) at the workshop?
(23 responses)






Multi/transdisciplinary experts, diversity of skills and
expertise, many perspectives (13 people)
Meeting people, networking, community, new
collaborations (7 people)
Arrangements (online pre-cooperation, provocations,
2nd day breakout groups, concrete goals, good sized
group) (5 people)
Informal interaction, side conversations (3 people)
New, provocative ideas (2 people)
Firstbeat user testing ; Great primer on the state of the
science; That everyone was so responsive (others)
5. On a scale from 1 (Poor) to 10 (Excellent), how
would you rate each of the parts of the workshop?
(27 responses)
The Idea Set and exchange of ideas beforehand
38.5 %
23.1 %
11.5 %
0.0 % 0.0 %
1
2
3.8 % 0.0 %
3
4
3.8 %
5
6
11.5 %
7.7 %
7
8
34.6 %
Provocations
0.0 % 0.0 %
1
2
9
0.0 % 0.0 %
3
4
3.8 %
5
7.7 %
6
10
30.8 %
11.5 % 11.5 %
7
8
9
10
5. On a scale from 1 (Poor) to 10 (Excellent), how
would you rate each of the parts of the workshop?
(27 responses)
22.2 % 22.2 %
Breakout group 1 (Day 1)
3.7 %
0.0 %
0.0 %
0.0 %
1
2
3
4
22.2 %
14.8 %
7.4 % 7.4 %
5
6
7
8
9
10
Breakout group 2 (Project development, Day 2 am)
25.9 % 25.9 %
29.6 %
11.1 %
0.0 %
0.0 %
1
2
0.0 % 0.0 %
3
4
3.7 % 3.7 %
5
6
7
8
9
10
5. On a scale from 1 (Poor) to 10 (Excellent), how
would you rate each of the parts of the workshop?
(27 responses)
Breakout group 3 (White paper topic outline, Day 2 pm)
34.6 %
19.2 % 19.2 %
0.0 %
0.0 %
0.0 % 0.0 %
3.8 %
23.1 %
0.0 %
8 comments: A lot of time was used for developing a shared understanding of the terms
used (2 people)
“I missed the knowledge of professionals from the field of Communication, marketing and
game industry”
”Groups seemed to run out of energy for final Breakout”
”We may have fallen short of concrete next steps”
”the breakout groups may have been too big to ensure everyone's voices were heard”
”especially pleased with the idea set responses and provocations”
6. There were three main goals for the
workshop. Do you think we achieved them?
(27 responses)
Goal 1. Identify strides that need to be made in innovation and
scientific discovery related to understanding of human health
behavior, behavior change, and behavior maintenance.
92.6 %
3.7 %
Missed Goal
3.7 %
Achieved Goal
Exceeded goal
6. There were three main goals for the
workshop. Do you think we achieved them?
(27 responses)
Goal 2. Think of new ways to overcome barriers that may hinder
this innovation that go beyond individual (siloed) research areas.
74.1 %
14.8 %
Missed Goal
11.1 %
Achieved Goal
Exceeded Goal
6. There were three main goals for the
workshop. Do you think we achieved them?
(27 responses)
Goal 3. Identify major content areas that need to be addressed to
truly move the science of human behavior change forward.
81.5 %
11.1 %
7.4 %
Missed Goal
Achieved Goal
Exceeded Goal
Comments from Question 6
“Useful but not always very clear, expect a
consolidated view in white paper”
 ”Not sure we had enough discussion on ways
to overcome barriers”
 ”We made strides but need more
communication to keep things going”
 ”Our work will depend upon generating robust
sets of co-occurrence data preferably with
little effort and little cost.”

7. Was the appropriate expertise in
the room to achieve our goals?
No
15,4 %
Yes
84,6 %
(26 responses)
If you answered ‘no’ – what were
we missing?
 More systems modelers (2
people)
 Communication, marketing
 More engineers/techies
 Folks from the smaller companies
and startups that are working in
the mobile health/sensing space
 More sensing-oriented folks,
 Data visualization and analytics,
 Architecture, urban planning,
 Neuroscience
 Behavioral geneticist
8. Did your experience at the workshop
impact your thinking or your (future) work?
(27 responses)
No
3,7 %
Yes
96,3 %
Comments
 New partners and
collaborations (7 people)
 New areas to think about,
new perspectives, new ideas,
new tools (6 people)
 Experimentation and
evaluation methods, study
designs (4 people)
 More understanding of other
disciplines (4 people)


Reinforcement to current
beliefs, validation of
current work (2 people)
More focus on systems and
modeling (2 people)
9. What would you like to see come
out of this workshop?
(26 responses)









White paper, journal papers, report (16 people)
Encouraging cross-disciplinary collaborations, understanding
of which players/disciplines needed (5 people)
New Projects (4 people)
A forum/community for connecting with others (3 people)
New RFA (s) (3 people)
Another conference/workshop (2 people)
A working group/task force (2 people)
Common data collection/repository (2 people)
Metrics of success for academics (beyond publications) ;
Recommendations to EU-US policy makers; Crossnational comparisons; Educational incentives
(fellowships, traineeships); Change in the way we all
see and create systems for behavior change (others)
10. Any ideas you would like to share for moving
these ideas/this group forward in the future?
(16 responses)






We need a place where projects could be proposed,
commented on, refined; where people could work on
projects together; a way to communicate (5 people)
Important to include the ”funding” side (policy-makers) (4
people)
Important to include the ”funding” side (policy-makers) (3
people)
A follow-up workshop, let’s meet again! (3 people)
Set of specific projects, project ideas (3 people)
How to bridge the theory, model and business, Work on a
common language across disciplines, write a paper
together, a wish-list of resources and desired distal and
proximal outcomes, What is being done by the organizers is
very good (others)
11. Is there anything else you would like to share
about your experience at the workshop?
(15 responses)



More time to get to know each other, to meet
with people and interact (2 people)
What is the state-of-the art in computational
modeling, more key note speeches would have
guided the thinking
“Great people!” “Organizers did an amazing
job!” “Organization of the workshop was
perfect” “Such a wonderful experience” “Very
good, very focused, very productive” “A
wonderful and diverse group of individuals”
“Networking sessions were amazing”
Thank you all for sharing your thoughts!