Running Head: FAMILY UP SURVEY
“Family Up”: Development of a Family-centered Early
Intervention Family Survey Program Utilizing an Electronically
Administered Data Collection Method
Melanie McKinney
Boise, Idaho
Michelle Norton
Brigham Young University
Rita Quigley
University of Montana
Heidi Wengreen
Utah State University
Running Head: FAMILY UP SURVEY
Introduction
The Individuals with Disabilities Education Act (IDEA) is a federal law ensuring
services to children with disabilities regardless of the nature or severity of their disability
(American Physical Therapy Association (APTA), (2010); “Building the Legacy,” n.d.).
IDEA governs the provision of early intervention services, special education, and related
services by state and public agencies (“Building the Legacy,” n.d.). IDEA has two
distinctive parts: Part B for children and youth ages 3 to 21 years receiving special education
and related services (“Building the Legacy,” n.d.), and Part C that supports early intervention
for infants and toddlers from birth to 3 years of age (APTA, 2010). Early intervention
services provide opportunities for infants and toddlers with disabilities or delays to improve
key skills, and to remediate developmental delays (“Overview of Early Intervention,” 2012).
“Up to 3” is an award-winning IDEA Part C provider within Utah Department of Health’s
Baby Watch Program. Up to 3 is at the Center for Persons with Disabilities at Utah State
University serving families in the Cache, Rich, and Box Elder counties (Up to 3 early
intervention, 2012), and is one of 15 Early Intervention programs in the state of Utah. In
Federal Fiscal Year (FYY) 2012-13, Up to 3 served 683 infants and toddlers (Utah
Department of Health Bureau of Child Development, 2013).
Providing families with the knowledge and tools to enhance their child’s growth and
development is the mission of the Up to 3 program (Up to 3 early intervention, 2012). The
philosophy that a family’s concerns, values, priorities, and resources establish the framework
for service is supported through family-centered practices (Up to 3 early intervention, 2012).
To aid in quality improvement and program development and/or evaluation, the aim of this
project is to develop and implement an evaluation strategy for the Up to 3 program utilizing
Running Head: FAMILY UP SURVEY
electronically formatted family satisfaction surveys along the continuum of the early
intervention service provided by Up to 3. Currently, Up to 3 only collects federally
mandated outcomes data that is reported to the Utah’s Baby Watch program. However,
organizations are encouraged to collect additional data to aid in program development and
improvement, including satisfaction-related data pertaining to effectiveness and efficiency of
services.
Concepts of survey validity and reliability will be utilized during the survey
development process to ensure collection of useful data for Up to 3 ongoing program
development and/or improvement.
Rationale
Beyond a funding requirement, program evaluation is an opportunity to assess service
quality, effectiveness, and efficiency. Information collected through program evaluation may
serve as a foundation for the Up to 3 program’s quality improvement initiatives, and
ultimately for improvement in care provided to participating children and families.
Distinction between family outcomes and family satisfaction. All state agencies
responsible for oversight and provision of IDEA Part C services are mandated by the Office
of Special Education Programs (OSEP) to collect data on family outcome indicators (Bailey,
Bruder & Hebbeler, 2006). As one of 15 agencies providing IDEA Part C early intervention
services in the Utah, the mandated family outcome data collected by Up to 3 is reported to
the state’s Baby Watch Early Intervention program. The family outcome indicators required
by OSEP include the effect early intervention services have on families’: 1) knowing their
rights with regard to early intervention services, 2) effectively communicating their child’s
needs, and 3) helping their child develop and learn (Bailey, Bruder & Hebbeler, 2006).
Running Head: FAMILY UP SURVEY
However, Up to 3 is not currently collecting data specific to parent’s perception of quality,
effectiveness, and efficiency of services.
While closely linked, family outcomes are distinguished from family satisfaction.
Specifically, an outcome is defined as a benefit of services and supports provided to families
(Bailey, Bruder & Hebbeler, 2006). Conversely, satisfaction is defined as a feeling
associated with the act of providing what is needed or desired (Satisfaction, n.d.). For
example, beyond knowing whether a parent communicated his/her child’s needs effectively
as an outcome, inquiry into parent satisfaction may include details regarding the quality,
effectiveness, and efficiency of communication with the early intervention provider/staff.
Satisfaction-related information may specifically aid Up to 3 in program/process
improvement and/or development.
States are strongly encouraged to collect additional data on the services provided by
early intervention programs to examine the effectiveness with which programs are supporting
families in caring for their children, and to help identify areas in which additional assistance
is needed to strengthen the program’s capabilities in serving children and their families
(Bailey, Bruder & Hebbeler, 2006). For Up to 3, additional information is desired specific to
the families’ perception of quality, effectiveness, and efficiency of services to aid in program
evaluation, and development and monitoring of quality improvement initiatives.
Family-centered care and program performance. The mission of the Up to 3
program includes providing family-centered practices whereby the framework for service is
established by the concerns, values, priorities, and resources of the family (Up to 3 early
intervention, 2012).
Running Head: FAMILY UP SURVEY
According to Can Child Center for Childhood Disability Research (2013), family-centered
services are:
...services made up of a set of values, attitudes, and approaches to services for
children with special needs and their families. Family-centered service recognizes
that each family is unique; that the family is the constant in the child’s life; and that
they are the experts on the child’s abilities and needs. The family works with service
providers to make informed decisions about the services and supports the child and
family receive. In family-centered service, the strengths and needs of all family
members are considered.
Ritzema, Saracino and Sladeczek (2010) report a positive correlation between parent rating
of family-centered care with overall satisfaction of early intervention services, satisfaction
with child outcomes, program adequacy, parent’s use of effective coping strategies, and the
extent to which parents actively sought help.
Up to 3 specifically aims to measure the extent to which staff and service providers
effectively and efficiently communicate and educate families regarding the rationale, and
strategies to follow through with recommended interventions following provider visits. An
additional aim is to measure the effect of early intervention services on the social, economic
and physical areas of the child’s and family’s life.
Electronic survey administration. Using a written format, Up to 3 has previously
attempted to collect additional data regarding parent satisfaction with services. Efforts by Up
to 3 to collect parent feedback about their experiences and satisfaction with the
interdisciplinary services provided by the program has had low rate of return (estimated to be
33%) when administered by telephone or in written format (disseminated by mail via postal
Running Head: FAMILY UP SURVEY
service). These methods have not provided sufficient data from which to make program and
policy decisions.
An alternate mechanism for survey administration is necessary to improve rate of
participation and to generate useful data. Due to the growing availability of the internet on
handheld devices, electronic surveys have become increasingly popular (Fan & Yan, 2010;
Sauermann & Roach, 2012). Likewise, electronic surveys have demonstrated many
advantages. Research consistently reports the benefits of electronic surveys in cost
efficiency, time in analyzing and entering data, and shorter time moving data between the
point of collection and the point of analyzation and use (Cobanoglu, Warae, & Morec, 2001).
Electronic surveys have been found especially useful in conducting research for populations
that regularly use the Internet (Couper, Traugott, & Lamias, 2001). Olsen, Wygant, and
Brown (2004) found many practical benefits to using electronic surveys in a college
population. In comparison to mail surveys the number of hours analyzing data (4 versus 107),
project cost ($84 versus $942), and return rate (50 percent versus 32 percent) all showed
some advantages to using electronic surveys. Results also found a 40 percent increase in the
number of words used to respond to open-ended questions in the electronic format.
Electronic surveys provide an attractive alternative to written format, but
limitations/disadvantages to this method have been identified. One is the concern about the
generalization of the results and the use of a convenience sample. Using the Internet to
distribute surveys limits the availability of that survey to those with Internet. In addition, it
favors selection in a group that is comfortable with technology (Cook, Heath, & Thompson,
2000; Sauerman & Roach, 2012).
Running Head: FAMILY UP SURVEY
Beyond cost, the use of electronic surveys to collect data for the Up to 3 program is a
logical choice for a number of reasons including: confidentiality, and ease of data
collection/tracking by program administrators. First, the program will provide each
participant access to the survey electronically to ensure confidentiality. In addition, the use
of an electronic survey mechanism will allow quick data collection, collected in “real time”
while maintaining confidentiality of responses, and reducing bias that may occur with paper
and pencil survey administration. This will allow immediate response by the family as well
as easy access by program administrators for data archiving and analysis. Specifically, the
Qualtrics electronic survey system has been selected for this project.
Qualtrics provides advanced survey logic and analysis targeted at research with a
focus on academic institutions. The Qualtrics Research Suite provides data analysis support
for cross-tabulation, conjoint analysis, subgroup analysis, and time series analysis. Data
generated using the Qualtrics system can be exported directly to a statistical analysis system
such as SPSS. Adding to the capability and benefits of “real time” data collection and
analysis, the Qualtrics system has the capability of generating a report with an associated
website which updates automatically as new data is generated (Qualtrics, 2013). Other
publicly available online survey tools do not provide these features or capabilities.
Up to 3 will provide access to the electronic surveys via handheld devices with
internet connection for ease of electronic survey administration.
Method
A set of (6) surveys was developed to be administered electronically using the
Qualtrics system. A demographic survey is included in the set of surveys along with (5)
Running Head: FAMILY UP SURVEY
additional surveys focusing on effectiveness and efficiency of services provided along the
continuum of participation in the Up to 3 program from intake to transition.
Procedures
Survey development. At the outset of the survey development process, Up to 3
administrators and selected staff provided guidance regarding the desired content of the
family surveys. Specifically, Up to 3 aimed to understand the families’ perception of quality,
effectiveness, and efficiency of services to aid in program evaluation, and development and
monitoring of quality improvement initiatives. Brevity was an important consideration
aiming to minimize respondent burden as a possible barrier to participation. A set of
electronic surveys was developed to administer using the Qualtrics electronic survey system.
The set consists of a demographic survey along with five different surveys for each phase of
service provided by the Up to 3 program: 1) Intake, 2) Evaluation, 3) Individualized Family
Service Plan development, 4) Service provider visits, and 5) Transition out of the program.
The basis for survey development was provided by content guidelines from Up to 3 program
administrators and service providers.
Each survey is between 9 and15 questions in length, and includes a combination of
both quantitative (Likert scale), and qualitative, open-ended questions. Examples of Likert
scale questions from the “Provider Visit” survey include, “During today’s visit, the service
provider worked directly with my child “hands-on”, and “During today’s visit, information
was provided to help our family continue the activity we worked on today on our own after
the visit.” An example of qualitative question includes, “Please comment on what you liked
about today’s activity.”
Running Head: FAMILY UP SURVEY
Qualtrics. The Qualtrics electronic survey system has been used to set up the
electronic surveys. Survey questions were input into the Qualtrics system, and a URL was
generated to access the survey at the time it is administered.
Survey validity. Methods to assess content validity of the surveys included expert
review by Up to 3 program administrators, service providers, as well as by university faculty
advisors. Additionally, parents who have past and present experience with services provided
by early intervention programs (including participation with the Up to 3 program) provided
review and feedback of 29 surveys. Families were asked to provide feedback about the
understandability of survey questions. The surveys were translated into Spanish, and
reviewed by Spanish-speaking families currently participating in the Up to 3 program. Face
validity was determined by a question at the end of the survey asking “What do you think
this survey intended to measure?” Appendix A includes the set of six surveys including:
Demographics Survey, Intake Survey, Evaluation Survey, Individual Family Service Plan
Survey, Service Provider Survey, and Transition Survey.
Discussion
The aim of this project was to develop a set of surveys in order to collect data from
families receiving services from Up to 3. The goal of these surveys was to gather data in real
time, or as quickly as possible following a service encounter. As such, electronic survey
administration was chosen in place of paper and pencil survey administration that had not
been previously successful. Six different surveys were developed and reviewed by experts as
well as parents who have received services from Up to 3. Based on the feedback provided by
the reviewers, revisions were made. The surveys were developed using the Qualtrics
Running Head: FAMILY UP SURVEY
electronic survey system that allows data analysis and archiving capability. Appendix A
includes the URL for each of the six developed surveys.
These surveys will provide Up to 3 with data based on family feedback that may be
used to improve services. Using Qualtrics, Up to 3 can track data in real time. In addition,
electronic survey administration provides each participant the opportunity to confidentially
participate in the survey program. Confidentiality using an electronic format is important for
reducing bias, while maintaining the capability of tracking and analyzing the collected data.
Limitations
This study was primarily limited by the small sample size that reviewed the survey.
Participants were chosen by Up to 3 to review the survey. Ideally, participants would be
randomly selected and from a variety of backgrounds. An earlier completion time of the
surveys would have allowed more time to have individuals review the surveys and increase
the reliability of the surveys. In addition, an earlier completion time may have allowed for a
piloting of the surveys and collection of validity and reliability results.
Future survey assessment
Future development of the Up to 3 family survey program may include pilot testing
along with test-retest reliability assessment.
Piloting. Systematic pilot testing using a random selection of Up to 3 participants is
recommended at the initiation of the survey program. Pilot testing will allow Up to 3 to test
the system for administering the survey program as a whole, as well as testing the feasibility
and logistics of collecting data in real time utilizing internet connections via handheld
devices. The program provides services to families in Cache, Rich, and Box Elder counties
with infants or toddlers under 3 years of age with developmental delays, disabilities or
Running Head: FAMILY UP SURVEY
diagnosed conditions with a high probability of resultant developmental delays (Up to 3 early
intervention, 2012). This pilot project is within the existing Up to 3 program’s Institutional
Review Board approval (protocol #5146).
Survey participants may be randomly selected from points along the continuum of
service (from intake to transition). The only qualification for participation in the survey will
be current enrollment in the Up to 3 program. The survey will be accessed using a URL link
via a tablet provided by the Up to 3 program (such as an iPAD). Identification numbers will
be assigned to each family to conceal identify for confidential participation in the survey
program. Assigning an confidential identification number will allow the program to track a
single family through the program, allowing analysis of specific variables related to that
family’s service such as specific demographic attributes, number of sessions, specific
services in the IFSP, and provider-specific variables. Appendix B (Table 1) describes the
proposed piloting program as well as the timeline for survey administration.
Test-retest reliability. Survey reliability using test-retest design may be assessed
during the piloting process. In a test-retest design, the survey may be administered as
scheduled following the particular service provided, then readministered a second time the
following day. This time frame minimizes time between the provided service and
readministration of the survey. Minimizing this timeframe between survey administration
points is important to capture the responders perception of the encounter as close to “real
time” as possible. Ultimately, Up to 3 aims to collect data in real time. A correlationcoefficient test may be used to assess test-retest reliability (the relationship between the first
administration and second administration of the survey). This measures the correlation on a
scale of -1.0 to 1.0. The closer to 1.0 the better the positive relationship between the
Running Head: FAMILY UP SURVEY
variables, the closer to -1.0 represents a negative relationship, while 0 represents no
relationship (Easton & McColl, 2013).
Implications
The completion of this project will provide a reliable and valid measure of family
satisfaction for use by early intervention organizations (including Up to 3). Previous research
has not been done on early intervention family satisfaction surveys. In addition, the data
collected will provide Up to 3 capability to evaluate the program and to develop and monitor
the progress of performance improvement initiatives.
References
American Physical Therapy Association (APTA) Practice Committee. (2010). Early
Intervention Physical Therapy: IDEA Part C.
Bailey, D., Bruder, M. B., Hebbeler, K. (2006). Guidance for states in documenting family
outcomes for early intervention and early childhood special education.
Can Child Centre for Childhood Disability Research. (2013). About family-centered
services. McMaster University. Retrieved (September 26, 2013), from
http://www.canchild.ca/en/childrenfamilies/about_fcs.asp
Up to 3 early intervention (2012). Center for Persons with Disabilities. Retrieved
(September 27, 2013), from http://www.cpdusu.org/projects/upto3/
Cobanoglu, Cihan, Warde, & Moreo (2001). A comparison of mail, fax, and web survey
methods. International Journal of Market Research, 43, 441-452.
Cook, C., Heath, F., Thompson, R. L. (2000). A meta-analysis of response rates in web- or
internet-based surveys. Educational and Psychological Measurement, 60, 821-836. Doi:
10.1177/00131640021970934
Couper, Mick, P., Traugott, M. W., Lamias, M. J. (2001). Web survey design and
administration. Public Opinion Quartlerly, 65, 230-253.
Easton, V. L. & McColl, J. H. (2013). Statistics glossary: Correlation coefficient.
Retrieved from http://www.stats.gla.ac.uk/steps/glossary/paired_data.html
Running Head: FAMILY UP SURVEY
Family Outcomes Survey--Revised Version (2010). Early Childhood Outcomes Center.
Retrieved September 27, 2013, from
http://projects.fpg.unc.edu/~eco/assets/pdfs/FOS-Revised.pdf
Fan, W., Yan, Z. (2010). Factors affecting response rates of the web survey: A systematic
review. Computers in Human Behavior, 26, 132-139.
Olsen, D. R., Wygant, S. A., & Brown, B. L. (2004). Electronic survey administration:
Assessment in the twenty-first century. Assessment update: Progress, Trends, and
Practices in Higher Education, 16(3).
Overview of Early Intervention (2012). National Dissemination Center for Children with
Disabilities. Retrieved (February 8, 2014) from http://nichcy.org/babies/overview
Qualtrics (2013). Analysis & reporting. Retrieved December 6, 2013 from
http://qualtrics.com/research-suite/survey-analysis-reporting/
Ritzema A, Saracino J, & Sladeczek I. (2010). Parents and early intervention:
satisfaction, coping, and family-centered care. Retrieved September 27, 2013 from
http://www.nasponline.org/conventions/handouts2010/papers/Parents%20and%20EI
%20-%20Satisfaction,%20Coping,%20and%20FamilyCentered%20Care%20(Ritzema%20et%20al.,%202010).pdf
Satisfaction. (n.d.) In Merriam-Webster online dictionary. Retrieved from
http://www.merriam-webster.com/dictionary/satisfaction
Sauermann, H., Roach, M. (2012). Increasing web survey response rates in innovation
research: An experimental study of static and dynamic contact design features.
Research Policy, 42, 273-286.
Thayer-Hart N, Dykema J, Elver K, Schaeffer NC, Stevenson J. (2010). Survey
fundamentals: a guide to designing and implementing surveys. Retrieved
September 27, 2013 from:
http://oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf
Building the Legacy: IDEA 2004. United States Department of Education. Retrieved
(February 8, 2014), from http://idea.ed.gov/
Running Head: FAMILY UP SURVEY
Utah Department of Health Bureau of Child Development (2013). Federal fiscal year
2012-2013 annual program profile. Baby Watch: Early intervention Program Birth to
Three Development. Retrieved (April 4, 2014), from
http://www.utahbabywatch.org/docs/localprograms/profiles/Up%20to%203%20%28FFY%2
012-13%20Profile%29.pdf
Running Head: FAMILY UP SURVEY
Appendix A: Set of (6) family surveys:
Demographics: https://usu.co1.qualtrics.com/SE/?SID=SV_3FbwIcpILvGflEF
IFSP: https://usu.co1.qualtrics.com/SE/?SID=SV_3pXqD7vS9YGyHC5
Service Provider: https://usu.co1.qualtrics.com/SE/?SID=SV_cCRZ2pnDee2m1uZ
Intake Survey:https://usu.co1.qualtrics.com/SE/?SID=SV_bdOLiKZ2aaYfWbr
Transition Survey: https://usu.co1.qualtrics.com/SE/?SID=SV_8AlZwTk4KWqX1K5
Evaluation Survey: https://usu.co1.qualtrics.com/SE/?SID=SV_a67sraclhe5QIwl
Running Head: FAMILY UP SURVEY
Appendix B:
Table 1. Proposed Pilot Program for the Up to 3 Family Survey
GOAL:
SURVEY PROGRAM PREPARATION:
ADMINISTRATION OF:
INTAKE SURVEY
ACTION:
1. Up to 3 will designate personnel to oversee the
survey program:
 Responsibilities include:
o Assigning and providing a
confidential identification number
to each participating family
o Developing a tracking system for
the survey program
o Communicating with service
providers regarding the survey
program, including
troubleshooting
o Developing an alternative plan for
administering the survey in real
time in the event of technology
failure (such as providing a “pen
and paper” survey that may be
completed in real time, sealed for
confidentiality then returned to
the survey administrator for future
input into the electronic system
for data collection)
2. Up to 3 will purchase tablets for providers to
use to administer the survey(s) in real time
(with the exception of the Intake Survey)
3. Up to 3 will provide mobile internet access for
the tablets
4. Up to 3 will train personnel on the survey
program including step-by-step instructions
regarding communication with survey
administrator (tracking procedures), use of
technology, administering the surveys
(including standardized instructions to provide
to the family at the time of administering the
survey), and troubleshooting
5. Up to 3 will develop a COVER LETTER
describing the survey program, including IRB
approval information
1. Individual completing the INTAKE interview
will alert the survey administrator to assign a
confidential identification number to the
family.
2. Survey administrator will send the family (via
email) the survey program cover letter with the
family’s CONFIDENTIAL IDENTIFICATION
NUMBER
3. Survey administrator will send the URL link for
the INTAKE SURVEY via email to the family
email
Running Head: FAMILY UP SURVEY
ADMINISTRATION OF:
EVALUATION SURVEY
INDIVIDUAL FAMILY SERVICE PLAN (IFSP)
SURVEY
SERVICE PROVIDER SURVEY TRANSITION
SURVEY
1.
2.
After the VISIT is completed, the service
provider will access the appropriate SURVEY
URL link using the tablet provided by Up to 3
(ie, Evaluation/IFSP/Service
Provider/Transition Survey)
The family member will complete the
SURVEY in real time which will be
electronically confidentially submitted prior to
the service provider departing
NOTE: The family member must enter the
confidential family identification number each
time he/she completes a survey.