See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/233899378
An Integrated Telehealth System for Remote Administration of an Adult Autism
Assessment
Article in Telemedicine and e-Health · December 2012
DOI: 10.1089/tmj.2012.0104 · Source: PubMed
CITATIONS
READS
43
3,905
5 authors, including:
Bambang Parmanto
I Wayan Pulantara
University of Pittsburgh
University of Pittsburgh
135 PUBLICATIONS 3,132 CITATIONS
17 PUBLICATIONS 217 CITATIONS
SEE PROFILE
SEE PROFILE
Jamie Lynn Kulzer
Michael Mccue
University of Pittsburgh
University of Pittsburgh
11 PUBLICATIONS 159 CITATIONS
76 PUBLICATIONS 1,395 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Telerehabilitation View project
Just-in-time Adaptive Intervention (JITAI) View project
All content following this page was uploaded by Bambang Parmanto on 12 July 2017.
The user has requested enhancement of the downloaded file.
SEE PROFILE
An Integrated Telehealth System for Remote Administration
of an Adult Autism Assessment
Bambang Parmanto, PhD,1,2 I. Wayan Pulantara, MS,1,2
Jamie L. Schutte, MS,2,3 Andi Saptono, PhD,1,2
and Michael P. McCue, PhD 2,3
Departments of 1Health Information Management
and 3Rehabilitation Science and Technology
and 2Rehabilitation Engineering Research Center
on Telerehabilitation, University of Pittsburgh, Pittsburgh,
Pennsylvania.
Abstract
We developed a telehealth system to administer an autism assessment remotely. The remote assessment system integrates videoconferencing, stimuli presentation, recording, image and video
presentation, and electronic assessment scoring into an intuitive
software platform. This is an advancement over existing technologies
used in telemental health, which currently require several devices.
The number of children, adolescents, and adults with autism spectrum disorders (ASDs) has increased dramatically over the past 20
years and is expected to continue to increase in coming years. In
general, there are not many clinicians trained in either the diagnosis
or treatment of adults with ASD. Given the number of adults with
autism in need, a remote assessment system can potentially provide
a solution to the lack of trained clinicians. The goal is to make the
remote assessment system as close to face-to-face assessment as
possible, yet versatile enough to support deployment in underserved
areas. The primary challenge to achieving this goal is that the assessment requires social interaction that appears natural and fluid,
so the remote system needs to be able to support fluid natural interaction. For this study we developed components to support this
type of interaction and integrated these components into a system
capable of supporting the entire autistic assessment protocol. We
then implemented the system and evaluated the system on real patients. The results suggest that we have achieved our goal in developing a system with high-quality interaction that is easy to use.
Key words: telemental health, telepsychiatry, telerehabilitation,
telehealth, telemedicine, telecare, autism, assessment
Introduction
V
ideoteleconferencing is the most widely used technology
for telemental health applications.1,2 Many telemental
health applications require additional functions beyond
interactive videoconferencing, such as stimuli presenta-
88 TELEMEDICINE and e-HEALTH F E B R U A R Y 2 0 1 3
tion and recording capabilities. A typical videoteleconferencing suite
in telemental health consists of a videoconferencing system such as
Polycom (San Jose, CA) and Tandberg (a Cisco Systems Company,
Oslo, Norway) with a TV display at each location, a document reader
for capturing hardcopy stimuli, a VHS videotape player for recording,
and a computer system for transmitting still images and full-motion
video.3 We have developed an integrated remote assessment system
that integrates all these functions (videoconferencing, stimuli presentation, recording, image and video presentation) and electronic
assessment scoring into an intuitive software. The integrated system
is used to support remote assessment of adolescents and adults with
possible autism spectrum disorder (ASD).
Adolescents and adults with ASDs primarily experience impairments in social interaction, impairments in communication, and
restricted and repetitive patterns of behavior and interests.4 Accurate
diagnosis is important because it has implications for treatment (e.g.,
access to appropriate supports and appropriate educational and vocational planning and accommodations). Many adolescents and
adults with high-functioning ASDs are currently undiagnosed or
misdiagnosed with emotional or psychiatric disorders. With the
number of children identified with ASDs increasing dramatically
over the past 20 years, the number of adolescents and adults with
ASDs is also expected to increase in the coming years. In fact, the U.S.
Centers for Disease Control and Prevention considers ASDs an urgent
public health concern.5
The Autism Diagnostic Observation Schedule (ADOS) has recently
become part of the gold standard in ASD diagnosis.6,7 The protocol
consists of a series of structured and semistructured tasks that involve
communication, social interaction, and play or imaginative use of
materials between the examiner (clinician) and the subject (patient).
There are four ADOS modules, appropriate for different ages and
expressive communication levels. ADOS Module 4 is used with adolescents and adults who are verbally fluent.8 Because ASD onset is
in childhood, many ADOS administrators regularly see only young
children (Modules 1 and 2). In general, there are not many clinicians
trained in either the diagnosis or treatment of adults with ASD. Given
the number of adults with autism in need and the lack of trained
clinicians, assessment services are not available at the level required.
We have developed an integrated remote assessment system in order
to address the aforementioned problems. The goal of the development is
to make the remote assessment system as close to face-to-face assessment as possible, yet versatile enough to support deployment in lowresource underserved areas. The primary challenge to achieving this goal
is that the assessment requires social interaction that appears natural and
fluid, while existing telemedicine technologies are often a barrier to fluid
DOI: 10.1089/tmj.2012.0104
REMOTE ADMINISTRATION OF ADULT AUTISM
natural interaction. An example of technology limitation that breaks the
natural flow of interaction is a remote clinician might ask a technician to
adjust the video or to present stimuli to the patient. We have developed
an innovative solution that allows the clinician to have more control of
the remote side in order to facilitate fluid natural interactions. The innovation includes the capability for the clinician to control the display
and layout of the display on the remote side, the capability to present and
control stimuli remotely, and the capability to control cameras on the
patient side. This innovation, combined with the high-resolution video
and low-latency interaction that often characterize high-end videoconferencing systems, is intended to achieve natural interaction between
clinician and patients. We conducted formative usability studies on
clinicians and summative usability studies on patients to evaluate if the
goal of the development of an integrated system for remote administration of an autistic assessment has been achieved.
Materials and Methods
The remote assessment system is intended to serve remote areas
with limited access to clinicians trained in ADOS, especially rural
areas. Therefore, the system needs to make use of off-the-shelf
technologies and should work with the broadband Internet bandwidth commonly available in rural clinics or schools, which is de-
fined by the U.S. Federal Communications Commission has having
data transmission speeds of at least 4 megabits per second (Mbps)
downstream and 1 Mbps upstream.9 The system was developed based
on the versatile and integrated system for telerehabilitation (VISYTER) platform,10 which provides versatility to adapt to the bandwidth in rural areas and requires no expensive devices but also has
the capability to support integrated services. Typical remote sites will
include small rural clinics, primary care doctor’s offices, or schools.
The remote sites will be connected to the ADOS-certified clinicians
located in major medical centers. The system needs to be designed to
support all phases of the assessment that go beyond conventional
videoconferencing, including stimuli presentation and assessment
documentation. In addition to being integrated and extensible, the
system should require minimal equipment beyond standard commodity computers to minimize the initial investment cost. Second,
the system has to be easy to install and to operate. This is not only to
minimize maintenance costs but also to address the fact that the
remote facilities usually have no information technology support
staff. The system should be able to be operated by a technician, an
administrative staff, or clinical assistant at the remote location.
ADOS was developed and validated by Dr. Catherine Lord and
colleagues and was first published in 1999. The ADOS is a
Table 1. Autism Diagnostic Observation Schedule Activities and Technology Development Input/Output Demands
ACTIVITY
SOCIAL AND COMMUNICATION DOMAINS
(NOT SPECIFIC TO ACTIVITY)
TECHNOLOGY DEMANDS
Construction Taska
Remote translation impossible
Telling a Story from a Book
Presentation of stimuli on a tablet that is controlled by both the
administrator and the patient
Description of a Picturea
Presentation of stimuli on a tablet that is controlled by the
administrator
Conversation and Reporting
Face-to-face videoconferencing; observational camera
Current Work or Schoola
Face-to-face videoconferencing; observational camera
Social Difficulties and Annoyance
Face-to-face videoconferencing; observational camera
Emotions
Face-to-face videoconferencing; observational camera
Demonstration Task
Face-to-face videoconferencing; observational camera
Cartoonsa
Presentation of stimuli on a tablet that is controlled by the
administrator; Face-to-face videoconferencing; observational camera
Break
Face-to-face videoconferencing; observational camera
a
Daily Living
Face-to-face videoconferencing; observational camera
Friends and Marriage
Face-to-face videoconferencing; observational camera
Loneliness
Face-to-face videoconferencing; observational camera
Plans and Hopes
Face-to-face videoconferencing; observational camera
Creating a Story
Stereotyped/idiosyncratic use of words or phrases
Descriptive, conventional, instrumental gestures
Unusual eye contact
Facial expressions directed towards others
Quality of social overtures
Amount of reciprocal social communication
Quality of social response
Conversation
Emphatic or emotional gestures
Empathy/comment on others’ emotions
Responsibility
Immediate echoing
Speech abnormalities
Imagination
Mannerisms
Unusual sensory behaviors
Excessive, specific interests
Rituals and compulsive behaviors
Overactivity
Negative behavior
Anxiety
Face-to-face videoconferencing; observational camera
13
Adapted from Lord et al.,
pp. 207 and 209.
a
Optional activity.
ª M A R Y A N N L I E B E R T , I N C . VOL. 19
NO. 2 FEBRUARY 2013
TELEMEDICINE and e-HEALTH 89
PARMANTO ET AL.
recording/archiving. Figure 1 shows a user
interface on the clinician’s station with the
components of the integrated remote assessment system.
Videoconferencing. The
videoconferencing system is the main component of the
system. It supports real-time interactivity
between a clinician and a remote patient
that resembles face-to-face communication. The system requires high-quality
videoconferencing that should be as close
to face-to-face as possible. The remote assessment system is designed to have the
following capabilities:
Low-latency and high-resolution audio
and video. The system should have a
latency of less than 250 milliseconds
Fig. 1. User interface on the clinician’s station. It shows two videostreams from the
with synchronized video-audio on both
(a) face-to-face and (b) observational cameras from the patient’s side, (c) image capture,
ends and be capable of supporting high(d) pan-tilt-zoom on-video remote camera control, (e) electronic scoring system, (f) stimuli
presentation, (g) stopwatch, and (h) a quick note.
definition (1280 · 720) or high-quality
(800 · 600) video resolutions.
. Able to support more than one camera. Two cameras are needed
semistructured, standardized assessment of communication, social
on the patient’s side: face-to-face and observational cameras.
interaction, and play or imaginative use of materials for individuals
The first camera is primarily to support interviews, and the
who have been referred because of possible autism or other pervasive
second camera is to observe nonverbal behaviors such as gesdevelopmental disorders.12 The ADOS Module 4 was designed for
ture or body language that cannot be observed by using just a
adolescents and adults who are verbally fluent (i.e., producing a
face-to-face camera.
range of flexible sentence types, providing language beyond the
. Remote camera control. We built the capability for the clinician to
immediate context, and describing logical connections within a
remotely control the cameras on the patient’s side. The unique
sentence). Module 4 consists of 10 standard activities and 5 optional
feature of camera control that we developed allows the clinician to
activities. The activities focus on social, communicative, and lancontrol the cameras (zoom, pan, or tilt) directly on the video
guage behaviors important in the diagnosis of ASD. They combine
windows using a mouse. This capability is especially important for
unstructured conversation with a variety of presses for particular
the observational camera in activities such as the demonstration
kinds of social and communicative behavior. The general format of
task, which requires the clinician adjust the camera or focus on a
the test administration is meant to create an interaction that appears
certain part of the patient’s activities. The observational camera
natural, during which preplanned occasions for certain behaviors
can be controlled by the clinician and is able to capture the client’s
arise.12,13 The activities, their in-person requirements, and the remote
hands and fingers to view complex mannerisms, gestures, and use
translation of the requirements are detailed in Table 1.
of presented objects and materials.
After the 45–60-minute assessment, during which detailed ob. Image capture from camera. We developed an image capture
servations are recorded, specific behaviors are coded based on the
capability that allows the clinician to take an image snapshot
participant’s behavior throughout the entire evaluation. The diagfrom videostreams that can then be used for clinical reports.
nostic algorithm classifies patients as having autism, ASD (including
pervasive developmental disorder and atypical autism), or nonThe videoconferencing system is the anchor for other components
spectrum. We designed the remote assessment system to include an
in the remote assessment system.
electronic scoring form and algorithm to calculate the diagnostic
score as well as the group area (domain) scores.
Layout control. Having multiple streams of video and stimuli on a
screen monitor can be very confusing for the patient. The patient may
DEVELOPMENT OF THE REMOTE ASSESSMENT SYSTEM
also be uncomfortable and intimidated by the multiple videos of himWe developed an integrated system to address the requirements for
or herself, especially those from the observational camera. Therefore,
a remote assessment discussed in the previous section. The system
the presentation on the patient’s side needs to be as simple as possible,
consists of the following components: videoconferencing, layout
yet able to be changed to follow the progression of the assessment
control, stimuli presentation, electronic scoring system, and session
protocol. To accommodate this need, we developed a layout control
.
90 TELEMEDICINE and e-HEALTH F E B R U A R Y 2 0 1 3
REMOTE ADMINISTRATION OF ADULT AUTISM
can also control on which display the
stimuli should be presented, on the tablet or
at a specific location on the patient’s display
monitor. Because the clinician has full
control of the stimuli, the clinician can show
a particular stimulus (e.g., a page from a
book) without having to ask for the patient
to flip pages as in the face-to-face assessment. Thus, the interactivity and fluidity of
the remote assessment system can be as
good as or better than face-to-face assessment. This system will be capable of providing much broader and richer stimuli than
the current document camera system used
in telemental health. Any type of stimuli
(picture, movie clip, animation, etc.) can be
presented and is not limited to a document
or still images as in the document camera
system.
Fig. 2. Remote control of the layout. In this illustration, the clinician is using (left) remote
layout control to change the appearance on the patient’s station (upper right) from three
parallel videostreams (lower right) to one clinician video as the focus.
system that allows the clinician to control the screen layout on the
patient’s and clinician’s stations (local and remote layout control). One
of the innovations of this remote assessment system is the capabilities
for the clinician to control screen layout, stimuli, and cameras on the
patient’s station. Controlling what appears on the patient’s screen is
important because of the number of windows and streams that can
appear on the patient’s station that can include the patient’s own videostreams, the clinician’s videostreams, and stimuli. This includes
deciding which videostream is presented on the screen and if more
than one videostream will be presented. The clinician can control the
layout of the streams using predefined layout settings that include
choosing which video is the focus and is presented larger on the screen
(e.g., one video large at the center with other videos on the left and
right, or one large video on the left with other videos on the right, etc.).
The layout control is available for controlling both local and remote stations, with ‘‘R’’ icons on the menu indicating controls for
remote layout. On the clinician’s station, the local layout control is
important because the clinician’s screen is also crowded with many
windows, including three or four videostreams (two from the patient’s side, one or two from the clinician’s side), a stimuli screen, and
the electronic scoring system. Figure 2 illustrates how the clinician
can change a layout on the patient’s station remotely by clicking one
of the predefined layouts.
Stimuli presentation. We developed a stimuli presentation component that allows the clinician to upload stimuli from the clinician’s
station and present it on the patient’s station. If the stimuli consist of
a series of stimuli, the clinician will be able to control which stimulus
is presented one at a time, the same way a clinician would present a
stimulus from a book page or from a deck of cartoons. The clinician
Electronic scoring system. In a face-toface assessment, a clinician conducts scoring after the assessment is finished, writing
quick notes to record any observations during the assessment. We
developed a scoring system that mimics this process. The scoring
instrument is implemented as a Web-based system that is integrated
into the remote assessment system. The goal of the remote assessment
system is to implement all the assessment and scoring activity electronically, with all processes conducted paperlessly. The assessment
begins after the session is ended, as in face-to-face sessions. A Webbased evaluation form was developed, and the clinician assigns a
score for each of the 31 evaluation items by selecting one of the
scores in the combo box. The system automatically calculates the
final score, as well as the scores of five areas (language, imagination,
etc.) using the algorithms prescribed in the ADOS scoring assessment.
Within the remote assessment system, the clinician can write quick
notes during the assessment process. Notes are taken during administration of each ADOS module. Overall ratings are completed
immediately after administration, even if the session is recorded.
Session archive database. The entire assessment session needs to
be archived, including the videostreams and stimuli presentations. A
secured session archived database was developed to allow clinicians
to record entire sessions in a secure archive database server. In addition to serving as a record, the archive is very useful for scoring and
for educational purposes as well. Any session can be replayed by the
clinician during the scoring after the assessment to increase the accuracy of the assessment. Interesting cases from the assessment,
segments of the session, or the entire session can be used to train new
clinicians on how to do assessment or how to spot diagnostic events.
Other components. The clinician often uses paper notes to write
observations during assessment that can later be used during scoring.
ª M A R Y A N N L I E B E R T , I N C . VOL. 19
NO. 2 FEBRUARY 2013
TELEMEDICINE and e-HEALTH 91
PARMANTO ET AL.
We developed an electronic quick note for the clinician to type observations during assessment to replace the paper note. One of the
requirements for the remote assessment system is to be as little intimidating and as similar to face-to-face sessions as possible. Having
eye contact is important for this patient population, and this is difficult to achieve with a desktop conferencing environment. We use
an inexpensive teleprompter that allows the clinician to see the patient using a periscope or to read verbatim protocol. A stopwatch with
noticeable display was also developed for the clinician to measure
how long the patient conducts an activity.
USABILITY EVALUATION
We conducted two types of usability studies on the system: formative and summative usability studies. A protocol for usability
studies was reviewed and approved by the Institutional Review Board
of the University of Pittsburgh, Pittsburgh, PA. Participants in the
formative usability studies were experienced ADOS administrators
(at least 1 year of experience administering ADOS). Five clinicians
were involved in the studies and were recruited using convenience
sampling from the Autism Service, Education, Research, and Training
network at the Western Psychiatric Institute and Clinic in Pittsburgh.
Participants in the summative studies were students currently attending the Hiram G. Andrews Center, a state-operated vocational
facility located in Johnstown, PA. The Hiram G. Andrews Center
primarily serves consumers of services from Pennsylvania’s Office of
Vocational Rehabilitation. Students at the Center are individuals 17
years of age and older with a disability. Ten patients participated in
the summative usability studies.
Results
DEPLOYMENT OF THE REMOTE ASSESSMENT SYSTEM
Figure 3 illustrates the deployment of the remote assessment
system consisting of clinician’s and patient’s stations connected over
the Internet. The patient’s station consists of a computer with a 24inch LCD monitor, two cameras (face-to-face camera on top of the
monitor and observational PTZ camera on the table), a tablet for
stimuli presentation, an echo-canceling microphone, and ADOS
materials for the ‘‘Creating a Story’’ activity. The clinician’s station
does not have an observational camera and tablet. The remote assessment system was used to connect patients at a state-operated
vocational facility located in Johnstown with the clinician at the
University of Pittsburgh. Johnstown is a small town in the Laurel
Highlands (Appalachian) region of Western Pennsylvania, a 2-hour
drive from Pittsburgh. The patient’s station is connected to the Internet via DSL cable with a speed of 768 kilobits per second up/
2 Mbps down, whereas the clinician’s station is connected to the
Internet using fiber optic connection with a speed of above 5 Mbps.
The assessment process was initiated by the clinician and the remote site operator/technician by launching the VISYTER program,
authenticating their identities to the server, entering the clinic room,
having the patient in a front of the computer, and then beginning the
assessment. After being connected, the clinician is able to control the
entire session remotely. Throughout the session, the layout on the
patient’s side was mostly displaying only the face-to-face videostream (Fig. 3, left), and hiding other streams such as the patient’s
own videos was intended to reduce distractions. It is important to
note that the clinician was observing gestures throughout the assessment using the observational camera. In the first activity
(‘‘Telling a Story from a Book’’), the clinician opened visual stimuli
and sent it to the tablet on the patient’s station (Fig. 3, left). The
clinician and the patient are able to work together to view the visual
stimuli, both able to move forward or backward as desired. The next
activity also required presentation of visual stimuli. The following
five activities consisted of interviews, and the clinician used the faceto-face camera and the observational camera. The stimuli presentation was used again to support the ‘‘Cartoon’’ activity before the
Break. The four activities following the Break were interviews using
the cameras. The last activity (‘‘Creating a Story’’) required the clinician to move his or her camera so the patient could see his or her
desk and then his or her face again. The clinician recorded the session,
and VISYTER archived the entire session, including all the videoconferencing streams and stimuli presentations. The archives were
available to the clinician for later review. After the session ended, the
clinician used the electronic scoring system on the right side of the
system and could access the archived session for a review.
EVALUATION OF THE REMOTE
ASSESSMENT SYSTEM
Fig. 3. (Left) Patient’s station and (right) clinician’s station.
92 TELEMEDICINE and e-HEALTH F E B R U A R Y 2 0 1 3
The goal of a formative usability study
is to identify usability problems and concerns and improve the usability of systems
by addressing their problems. For this
study, another goal of the formative study
is to identify which aspects of the system
can be adjusted to better match the standardized face-to-face administration. The
methodology used in this study is the
‘‘cognitive walkthrough’’ usability inspection.14 In its original form, cognitive
walkthroughs involve one or a group of
REMOTE ADMINISTRATION OF ADULT AUTISM
Table 2. Results of the Usability Studies Using
the Telehealth Usability Questionnaire
QUESTION
MEAN SCORE
1
It was simple to use this system.
6.14
2
It was easy to learn to use the system.
6.14
3
The way I interact with this system (the computer
screen and the tablet) is pleasant.
6.29
4
I like using the computer screen and tablet.
6.57
5
The computer screen and tablet are simple and
intuitive.
6.57
6
This system is able to do everything I would want it
to be able to do during the assessment.
6.57
7
I can easily talk to the clinician in Pittsburgh.
6.57
8
I can hear the clinician in Pittsburgh clearly.
6.50
9
I felt I was able to express myself effectively.
6.14
10
I can see the clinician in Pittsburgh as if we met in
person.
6.00
11
I think the assessment provided over the telehealth
system (using videoconferencing) is the same as the
in-person assessment.
5.86
12
I feel comfortable communicating with the clinician
in Pittsburgh.
6.50
13
Telehealth is an acceptable way to receive healthcare
services.
5.83
14
I would use telehealth services again.
6.17
15
Overall, I am satisfied with this telehealth system.
6.50
Total average
6.29
evaluators inspecting a user interface by viewing a set of tasks and
evaluating understandability and ease of learning. In this project, experienced clinicians participated in a tele-assessment to evaluate if the
remote administration system replicates face-to-face administration
conditions. The clinician used the system to perform a remote ADOS
assessment on a mock client (i.e., a member of the development team). In
this formative evaluation, no formal measurement was used. Instead,
any issues encountered by the clinician during the walkthrough were
identified, and subsequently the system was revised to address the issues. The results of this formative usability studies through walkthrough
inspection were an operational telehealth system for administration of
ADOS assessment.
Subsequent to the formative usability study and the resulting
improved system, the system was used in a research protocol to remotely assess adults with an ASD diagnosis. The summative usability
study was administered to the participants after they completed the
remote ADOS administration. Ten patients were involved in the
summative usability studies. We used a draft subset of a Telehealth
Usability Questionnaire (TUQ) that we are currently developing,
primarily to evaluate ease of use and learnability of the system, interface quality, interaction quality, reliability, and satisfaction and
future use. Our TUQ adapts and combines questions from three
questionnaires relevant to telehealth systems: the Technology Acceptance Model’s Perceived Usefulness/Ease of Use,15 the Computer
System Usability Questionnaire,16 and the Telemedicine Satisfaction
Questionnaire.17 Specifically, the TUQ adapts questions on ease of use
aspects from the Technology Acceptance Model’s Perceived Usefulness/Ease of Use, adapts questions from the Computer System Usability Questionnaire on learnability, interface quality, reliability,
and satisfaction, and adapts questions from the Telemedicine Satisfaction Questionnaire on interaction quality, and future use. This
TUQ uses a 7-point Likert scale with 1 representing disagreement and
7 representing agreement. The results are shown in Table 2.
We used the same computers, peripherals, and network settings for
the clinician’s station in Pittsburgh and for the patient’s station in
Johnstown for all sessions. We did not encounter technical problems
during the sessions. In our previous experience with deploying the
telehealth system, the problem was usually caused by setting of the
devices, peripherals, and Internet connections. Learning from this
experience, we tested all the settings (video resolution, network
speed, bit rate, compression, etc.) and peripherals (camera, speakerphone, and tablet) and kept the same optimal settings for all the
sessions.
Discussion
Overall, the patients were satisfied with the telehealth system we
developed, which received an average overall score of 6.5 out of 7.
The system received high marks on ease of use and learnability
(average of 6.14 for questions 1 and 2), interface quality (average of
6.5 for questions 3–6), and interaction quality (average of 6.3 for
questions 7–10). Patients expressed that they were very comfortable
with the use of the telehealth system (6.5 for question 12) and would
use the system again (6.17 for question 14). However, the scores for
the two questions related to general acceptability of telehealth as a
replacement for in-person service were slightly lower (5.86 for
question 10 and 5.83 for question 13). It seems that participants were
marginally less satisfied (5.85 compared with 6.29 overall) when it
came to the questions comparing healthcare received in-person and
through telehealth and about whether telehealth is an acceptable way
to receive telehealth service. The primary goals of the development of
the remote assessment system was to have a system enabling highquality interaction close to that in face-to-face assessment, a system
that is fluid and easy to use, and one with overall good usability. The
excellent scores the system received for ease of use and learnability
(6.14 out of 7), interface quality (6.5 out of 7), and interaction quality
(6.3 out of 7) suggest that these goals have been achieved.
An important contribution of the integrated assessment system is
that it can be used for other types of assessments that require fluid
interactions between clinicians and patients, such as neuropsychology tests. This type of assessment is very challenging and difficult to
implement using existing technologies. To further evaluate the
ª M A R Y A N N L I E B E R T , I N C . VOL. 19
NO. 2 FEBRUARY 2013
TELEMEDICINE and e-HEALTH 93
PARMANTO ET AL.
efficacy of the autistic remote assessment system, one of the authors
( J.L.S.) is currently conducting research on the reliability and validity
of the remote assessment system for autism. The reliability study will
compare the remote assessment system with the face-to-face system,
while the validity study will estimate the diagnostic accuracy of the
remote administration of autistic assessment.
Acknowledgments
We gratefully acknowledge the help of Ashlee Filippone as well as
the participants in this research. This project is funded in part by the
National Institute on Disability and Rehabilitation Research’s Rehabilitation Engineering Research Center on Telerehabilitation
(projects number H133E040012 and number H133E980025) and by
the Pennsylvania Department of Public Welfare’s Autism Service,
Education, Research, and Training project.
Disclosure Statement
Authors B.P., I.W.P., and A.S. hold patent pending for the VISYTER
system.
developmental delay: Evaluating diagnostic validity. J Autism Dev Disord
2008;38:657–667.
8. Bastiaansen JA, Meffert H, Hein S, Huizinga P, Ketelaars C, Pijnenborg M, Bartels
A, Minderaa R, Keysers C, de Bildt A. Diagnosing autism spectrum disorders in
adults: The use of Autism Diagnostic Observation Schedule (ADOS) module 4. J
Autism Dev Disord 2011;41:1256–1266.
9. Federal Communications Commission. Sixth Broadband Deployment Report.
Available at http://transition.fcc.gov/Daily_Releases/Daily_Business/2010/
db0720/FCC-10-129A1.pdf (last accessed January 23, 2012).
10. Parmanto B, Saptono A, Pramana G, Pulantara W, Schein R, Schmeler M, McCue
M, Brienza D. VISYTER: Versatile and integrated system for telerehabilitation.
Telemed J E Health 2010;16:1–6.
11. Lord C., Rutter M, DiLavore PC, Risi S. Autism Diagnostic Observation Schedule—
WPS (ADOS-WPS). Los Angeles: Western Psychological Services, 1999.
12. Lord C, Rutter M, DiLavore P, Risi S. Autism Diagnostic Observation Schedule:
ADOS manual. Los Angeles: Western Psychological Services, 2008.
13. Lord C, Risi S, Lambrecht L, Cook EH Jr, Leventhal BL, DiLavore PC, Pickles A,
Rutter M. The Autism Diagnostic Observation Schedule-Generic: A standard
measure of social and communication deficits associated with the spectrum of
autism. J Autism Dev Disord 2000;30:205–223.
14. Nielsen J, Mack RL (eds). Usability inspection methods. New York: John Wiley &
Sons, 1994.
REFERENCES
15. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of
information technology. MIS Q 1989;13:319–340.
1. Antonacci DJ, Bloch RM, Saeed SA, Yildirim Y, Talley J. Empirical evidence on the
use and effectiveness of telepsychiatry via videoconferencing: Implications for
forensic and correctional psychiatry. Behav Sci Law 2008;26:253–269.
16. Lewis JR. IBM computer usability satisfaction questionnaires: Pyschometric
evaluation and instructions for use. Int J Hum Comput Interact 1995;7:57–78.
2. Yellowlees P, Shore J, Roberts L. Practice guidelines for videoconferencing-based
telemental health—October 2009. Telemed J E Health 2010;16:1074–1089.
17. Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the Telemedicine
Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: A
preliminary study. J Telemed Telecare 2003;9:46–50.
3. Grady B, Singleton M. Telepsychiatry ‘‘coverage’’ to a rural inpatient psychiatric
unit. Telemed J E Health 2011;17:603–608.
4. American Psychiatric Association. Diagnostic and statistical manual of mental
disorders (4th ed.) text revision. Washington, DC: American Psychiatric
Association, 2000.
5. Autism and Developmental Disabilities Monitoring Network Surveillance Year
2006 Principal Investigators; Centers for Disease Control and Prevention (CDC).
Prevalence of autism spectrum disorders—Autism and Developmental
Disabilities Monitoring Network, United States, 2006. MMWR Surveill Summ
2009;58(10):1–20. Erratum in MMWR Surveill Summ 2010;59(30):956.
6. Le Couter A, Handen G, Hammal D, McConachie H. Diagnosing autism spectrum
disorders in pre-school children using two standardized assessment
instruments: The ADI-R and the ADOS. J Autism Dev Disord 2008;38:362–372.
7. Gray KM, Tonge BJ, Sweeney DJ. Using the Autism Diagnostic Interview-Revised
and the Autism Diagnostic Observation Schedule with young children with
94 TELEMEDICINE and e-HEALTH F E B R U A R Y 2 0 1 3
View publication stats
Address correspondence to:
Bambang Parmanto, PhD
Department of Health Information Management
6026 Forbes Tower
University of Pittsburgh
Pittsburgh, PA 15260
E-mail: parmanto@pitt.edu
Received: April 26, 2012
Revised: May 25, 2012
Accepted: May 29, 2012