NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Veteran Metrics
Initiative:
Learning What Works for Veterans and their Families
Ms. Cynthia L. Gilman, JD
Vice President, Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc.
Center for Public-Private Partnership
Ms. Katy Hussey-Sloniker, MBA
Program Coordinator, The Veteran Metrics Initiative: Learning What Works for Veterans and their Families
Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc.
Center for Public-Private Partnership
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
1
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Henry M. Jackson Foundation
for the Advancement of Military Medicine, Inc.
Mission: Serves as a vital link among the military medical community and its federal and
private partners to advance military medicine. The Foundation’s support and
administrative capabilities allow DoD, VA and civilian researchers and clinicians to
maintain their scientific focus and accomplish their research goals effectively and
efficiently.
•Authorized by Congress in 1983
•Foundation manages more than 1,000 research grants and projects around the world
•Core function of HJF is to support research and education at the Uniformed Services
University of the Health Sciences
•Not a traditional grant-making Foundation
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
2
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Center for Public-Private Partnerships
Mission: Facilitates diverse partnerships to advance medical research and
clinical care for service members and veterans, regardless of whether the
resources reside within the military, VA or civilian sectors.
•Heroes of Military Medicine Awards & Dinner
•HJF Military Medicine Symposium
•Veteran Metrics Initiative
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
3
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
THE NATIONAL CHALLENGE
The Need To Capture and Analyze Data that Can Assess Impact
•
Veteran success is affected by many variables that exist in external communities
•
DoD and VA provide numerous transition support programs and services
•
More than 45,000+ community based organizations and 10,000+ websites exist to
help veterans and their families heal and successfully reintegrate
•
Veteran reintegration supports and interventions are currently based on conjecture
and anecdotes, and not on evidence based data
•
Lack of a standardized vocabulary
•
No universally agreed upon standard on WHAT to measure and the methodology of
HOW best to evaluate
•
No veteran reintegration metrics coordination among DoD, VA and communities
© 2012 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
4
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Veteran Metrics Initiative Goal
Improve the long-term health and wellness of veterans and their
families by developing metric tools to guide the use of effective
interventions along the continuum of transition, separation and
community reintegration.
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
5
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Large Group Session Discussions
•Impact should be assessed at the community level
•Outcome impact should be assessed from the perspective of all veterans
without regard to era of service, prior combat experience or health status
•A common taxonomy must be developed
•Analytical scanning has the potential to improve quality of life across all
dimensions of well-being
•Gathering or tapping into lots of “data” for triangulation and meaning
•Core data elements should be identified across all dimensions of well-being
for quality of life
•Common indicators of value should be identified
•Show impact over time (longitudinal), not as a snapshot
© 2012 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
6
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Small Group Metrics Session
•Data analyzed in most well-being and health risk surveys/studies is only
cross-sectional, therefore long term impact cannot be derived
•Longitudinal data with multiple assessment points is required to evaluate
impact
•Set studies up with parameters to assess long-term outcomes
•Begin assessment prior to transition from military service – then follow
through to community reintegration
•“Crosswalk” resulting data with other known Global Assessment Tools that
DoD, VA and others currently utilize
© 2012 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
7
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
THE STRATEGIC RESPONSE
CP3 Will Leverage Expertise by Building Links Across Systems
•
Create a shared vision of learning what works for veterans and their families
through evidence-based research
•
Build teams of strategic, philanthropic, industry, federal, community-based
organization, and veteran advisors
•
Connect and link with like-minded counterparts to amplify vision
•
Work in partnership with DoD, VA and civilian subject matter experts &
stakeholders to construct research projects with translation potential
•
Work in an open and transparent environment
“The more complex of an issue,
the more connectivity across-systems is needed to tackle the issue.”
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
8
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
The Veterans Metrics Initiative
Learning What Works for Veterans and Their Families
Research Objectives
•Learn how to define and measure the effectiveness of a broad range of
interventions offered by the DoD, VA and local communities on the long-term
health and wellness of veterans and their families.
•Determine the methodologies that most reliably measure long-term health and
wellness impacts of interventions utilized by veterans and their families along the
continuum of transition, separation from service, and community reintegration.
•Discover the core elements of the interventions that are causally associated with
improved veteran and family health and wellness, and determine the common
factors of the most effective interventions.
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
9
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
10
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
11
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
The Veterans Metrics Initiative
Learning What Works for Veterans and Their Families
Research Projects
1. Crowd-sourcing & Data-visualization to Understand Unstructured Data:
Exploring Social Media to Improve Transitioning Veteran Outcomes
• Privately sponsored research agreement
• Collaborative research with the Uniformed Services University of
the Health Sciences
• Qualitative research that will help lay a foundation for future
projects
2. Common Factors of Success
• Formative stages
• Excited to share and hear your thoughts
2. Crosswalk
• Concept: “Common Factors” Project Results mixed with other GATs
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
12
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Veteran Metrics Initiative:
Learning What Works for Veterans and their Families
Common Factors of Success Research Project
Data Collection
Service Member (SM) self-reports contact & interventions in communities
SM Identifies Community Interventions Utilized
(Non-Governmental Organizations)
Community
Relocation
2
4
3
1
SM Identifies Government Interventions Utilized
(Local, State, and Federal)
Individual
Post 9/11 Veteran
N=TBD
1
2
Point-in-time of
Separation from
military service
© 2013 Henry M. Jackson Foundation
for the Advancement of Military
Medicine, Inc.
5
3
4
5
Objective: Data collection of interventions
utilized and assessment of pre- and post
transition health & wellness of SM
population over established time
Identify Service Specific
Transition Assistance
Interventions
Identify Service Members (SM) transitioning out of military service prior to departure D-t days
Prior to SM transition administer baseline health and wellness measuring tool at D day, obtain DoD records of
study indentified military interventions, SM self-reports community interventions utilized in personal transition
process, and identifies intended post-transition community location.
Administer health & wellness tool D+x, D+y, D+z days in new community location; SM identifies federal, state and
community interventions utilized.
12
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Veteran Metrics Initiative:
Learning What Works for Veterans and their Families
Common Factors of Success Research Project
Data Analysis and Interpretation
Identified and Quantified
Service Specific
Transition Assistance
Interventions Received
Prior to Service
Departure
Identified and
Quantified SM
Reported Utilized
Community &
Government
Interventions
Outcome: Learn how to measure what preand post transition interventions impact
individual veteran and family health &
wellness over time
Data analysis and correlation of causal
effects among specific Government
and community interventions and
improved long-term veteran and
family health & wellness.
Discover core elements of most
effective interventions and learn the
common factors among them.
Assessed Health &
Wellness (Individual &
Complete Participant
Population)
© 2013 Henry M. Jackson Foundation
for the Advancement of Military
Medicine, Inc.
Assessed Health &
Wellness (Individual &
Complete Communitybased Participant
Population)
Point-in-time of
Separation from
military service
Impact: Enhance and improve the lives of
veterans and their families by providing a metric
resource communities can use for the
development of effective, evidence-based
interventions
13
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Questions and Discussion
to follow LTC Ken Jones, PhD, USA
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
15
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Using Qualitative Data Visualization
Techniques to Drive Quantitative Analysis
Kenneth D. Jones II, PhD, FACHE
US Army Lieutenant Colonel, Medical
Services Corps
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
16
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
The Problem
•In Quantitative methodology, “Theory” drives the
research
•In the real world, we may not have a well informed
“Theory”
•We may not have a rich understanding of the
phenomena we wish to study
•We have cognitive bias when developing, deploying, and
analyzing survey type data
•So, what can we do?
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
17
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Getting at a Solution
•Qualitative, data visualization tools allow us to make
sense and develop a rich understanding of dynamic
phenomena
•Many unstructured data exist in social media spaces and
elsewhere on the Internet (i.e. Blogs, RSS Feeds, New
Feeds, Twitter)
•Capturing and making sense of these data will allow us
to form better research questions and give us tool with
which to explore
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
18
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
An Exercise in Working with
Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
19
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Twitter “Weather Tweets”
Across the U.S.
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
20
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
21
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Unstructured Data
Once there was a mother pig who had three little pigs. She did not have enough to keep them, so she sent them out to seek their fortunes. The first little pig had not gone far
when he met a man with a bundle of straw. The little pig said to him, "Please, man, give me that straw to build me a house." This the man did, and soon the little pig had built a
house with it. Just after the house was built, along came a wolf. He knocked at the door of the little pig's house and called, "Little pig, little pig, let me come in!" But the little
pig answered, "No, no! Not by the hair of my chinny chin chin!" Then the wolf said, "I'll huff and I'll puff, and I'll blow your house in!" So he huffed and he puffed until he blew
the house in, and ate up that little pig. The second little pig had not gone far when he met a man carrying a bundle of sticks on his shoulders. The little pig said to him, "Please,
man, give me those sticks to build me a house." This the man did, and soon the little pig had built a house with them.
Just after the house was built, along came the wolf. He knocked at the door of the little pig's house and said, "Little pig, little pig, let me come in!" But the little pig answered,
"No, no! Not by the hair of my chinny chin chin!" Then the wolf said, "I'll huff and I'll puff, and I'll blow your house in!" So he huffed and he puffed and he blew the house in,
and ate up that little pig.
The third little pig met a man with a load of bricks. The little pig said to him, "Please, man, give me those bricks to build me a house." This the man did, and soon the little pig
had built a house with them. Just after the house was built, along came the wolf. He knocked at the door of the little pig's house and said, "Little pig, little pig, let me come in!"
But the little pig answered, "No, no! Not by the hair of my chinny chin chin!" Then the wolf said, "I'll huff and I'll puff, and I'll blow your house in!" So the wolf huffed and he
puffed and he puffed and he huffed, and he huffed and he puffed, but he couldn't blow this third little pig's house down. When he found that with all his huffing and puffing
he could not blow this little pig's house down, he said, "Little pig, I know where there is a field of fine turnips. "Where?" the little pig asked eagerly. "Over in Mr. Smith's home
field. And if you will be ready tomorrow morning I will call you and we will go together and get some for our dinner." "Thank you," replied the little pig. "I will be ready when
you come for me. What time do you want to go?" "Oh, I'll come for you at six o'clock."
Now the little pig rose at five o'clock and was back home with his turnips when about six o'clock the wolf came and said, "Little pig, are you ready?" "Ready?" exclaimed the
little pig. "Why, I have been there and back home again, and I have a fine pot of turnips already cooked for my dinner!" The wolf was very angry, but thinking that he would be
equal to the little pig, he said, "Little pig, I know where there is a very nice apple tree." "Where?" the little pig asked eagerly. "Down at Merry Garden," replied the wolf. "And if
you will not deceive me this time, I will come for you at five o'clock tomorrow morning and we will go down there together and get some nice apples." "I will be ready," replied
the little pig. The little pig got up early the next morning, and was on his way by four o'clock. But this time he had to go much farther, and besides, he had to climb the tree to
get the apples. Just as he was ready to jump down, he spied the wolf. "What, little pig!" said the wolf. "You here before me? Are they nice apples?" "Oh, yes," replied the little
pig. "Here, I will throw one down for you." Now the little pig threw that apple so far that while the wolf was going after it, he jumped down to the ground and ran home with
his basket of apples as fast as he was able. He dashed into the house, slammed the door, and locked it. Then he sat down to rest. Of course the wolf was again very angry, but
the next day he came to the little pig's house once more and said, "Little pig, there's a fair over at Shanklin this afternoon. Will you go there with me?”
"Oh, yes," replied the little pig. "What time shall I expect you?" "At three," answered the wolf. That afternoon the little pig went off before three o'clock, just as usual, got to
the fair, bought a butter churn, and was going home with it when he spied the wolf coming. This time the little pig was terribly frightened. He didn't know what to do, so he got
into the churn to hide. But as he was climbing in, the churn started to roll round and round. Down the hill it rolled, faster and faster, with the little pig still in it. This frightened
the wolf so much that he ran home, forgetting all about going to the fair at Shanklin that afternoon. The next day he went to the little pig's house again and told him how
frightened he had been while going to the fair. The little pig laughed, and said, "Ha, ha! I frightened you that time! I had gone to the fair and had bought a butter churn there;
and when I saw you coming I climbed inside the churn and rolled down the hill." Then the wolf was very angry indeed. He vowed that he would eat up that little pig -- that he
would climb up on the roof of the little pig's house and go down the chimney after him. When the little pig heard the wolf on the roof of his house and saw what he was about,
he made a blazing fire in his fireplace, filled a big pot with water, and hung it over the fire. Just as the wolf was coming down the chimney, the little pig lifted the lid off the big
pot of boiling water, and in fell the wolf. And then the little pig quickly popped on the cover again, and had the wolf for supper. And that is how it came about that this little pig
lived happily in his snug little brick house ever after.
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
22
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Cloud of Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
23
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Phrase Net of Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
24
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Tree “Pig” of Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
25
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Tree “House” of Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
26
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Tree “Wolf” of Unstructured Data
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
27
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Developing Good Questions
•How many Pigs and Wolves are involved here?
•Are there anger management issues in the Wolf
community?
•Is there an epidemic of Wolf on Pig aggression?
•Is there a housing problem in the Pig
community?
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
28
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Is there a housing problem in the
Pig community?
Search Data
for “Housing
Issues”…
Building
Houses
Houses Being
Destroyed
House Built
with Straw
House
Destroyed
House Built
with Sticks
House
Destroyed
House Built
with Bricks
House Stable
Every time a house was built with straw or sticks (perhaps sub-standard materials), it
was destroyed.
Build houses with bricks (perhaps a better material) to avoid house destruction.
29
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
What is our Space?
•We want to know what is working for
transitioning veterans
•We want to know what communities are doing
•We want to know what success means
•Who can tell us?
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
30
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
A Veteran – “Alex Horton”
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
31
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
How Do We Reach Him?
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
32
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
His Blog – “Army of Dude”
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
33
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Change in our process
Develop
Constructs
Define
Variables
Develop
Questions
Deploy
Surveys
Gather
Surveys
Analyze
Listen to
Conversation
Gather Data
Back Out
Constructs
Develop
Surveys
Report Results
Analyze
Report Results
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
34
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
How We Do it Now…
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
35
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Another Way……MetaLayer
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
36
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Notional Free-Text Data from
Transitioning Veterans
•The best part about WarriorAdventures was that is was something different.
• I really liked WarriorAdventures because it allowed me to spend time with other folks just like me.
• The best part about WarriorAdventures was the people.
• I’m so proud I just finished my bachelor’s degree! I put those GI bill benefits to good use and now I’m on my way to a new career.
• Still not feeling well. I have more physical therapy until I am used to these new prosthetics.
• My new service dog Pixly is awesome! I totally love him!
• The best part about WarriorAdventures was there were tons of other wounded vets there. No one looked at me like I was different.
• I’m so excited about the fun run this weekend. We will reach hundreds of people in the community and inform them about the Wounded
Warrior Project.
• I wonder how much longer until I get my new prostetics? I’m looking forward to being able to run again.
• I just finished a 3-day training program on resume writing and interviewing hosted by CitiBank. It was really well done and I learned a lot.
• The partnership program I just joined is just what I needed. I can feel the depression lifting and I have new things to look forward to.
• The best part about WarriorAdventures was fellowship with other wounded veterans.
• The best part about WarriorAdventures was having a place to fit in.
• North Carolina really cares about veterans!!! The state hosted 5 K run this past weekend was awesome. Met some cool people and had a
great time.
• I learned a ton about my VA benefits at the Wounded Warrior get together in Vienna this weekend.
• The best part of the program was the people. They really reached out to me.
• I spent the weekend fishing with my fellow veterans. Caught a nice trout!
• The best part about WarriorAdventures was meeting new people.
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
41
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Tag Cloud
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
42
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Phrase Net
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
43
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Tree – “WarriorAdventures”
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
44
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Word Tree – “Proud”
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
45
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Project Timeline
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
46
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Endstate:
Taxonomy of Programs and Outcomes
Intervention
Type 1
Intervention
Type 2
Intervention
Type 3
Outcome
Type 1
Outcome
Type 2
(Recreation)
(Outreach)
(Navigation)
(Increased Health
Index Score)
(Increased
Education Score)
Element A
Element A
(peer-to-peer
mentoring)
(peer-to-peer
mentoring)
X
Element B
Element B
(peer-to-peer
counseling)
(peer-to-peer
counseling)
Element C
Element C
(caregiver respite)
(caregiver respite)
Element D
Element D
Element D
(benefits counseling)
(benefits counseling)
(benefits counseling)
Element E
Element E
(safe/affordable
housing)
(safe/affordable
housing)
X
X
X
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
47
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Conclusion
Using Qualitative Data Visualization Techniques to Drive
Quantitative Analysis
Kenneth D. Jones II, PhD, FACHE
US Army Lieutenant Colonel, Medical
Services Corps
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
48
NC Governor’s Focus on Service Members, Veterans and their Families
28 February 2013
Questions for either:
•The Center for Public-Private Partnerships
•Ms. Cynthia Gilman/Ms. Katy Hussey-Sloniker
•LTC Ken Jones, PhD
© 2013 Henry M. Jackson Foundation for the
Advancement of Military Medicine, Inc.
49