RESNA Annual Conference - 2019

Investigation Of Video Visual Scene Display Technology

Salena Babb1, David McNaughton, PhD1, Janice Light, PhD2, Jessica Caron, PhD2, Kirk Wydner2

1Department of Educational Psychology, Counseling, and Special Education,

The Pennsylvania State University, University Park, PA

2Department of Communication Sciences and Disorders, The Pennsylvania State University


In order to maximize the positive outcomes for individuals with disabilities interventions designed to enhance communication, independence, and meaningful participation in natural environments are desperately needed. The need may be greater for individuals with disabilities who as present with complex communication needs,  meaning that speech alone does not meet their daily communication needs [1]. In regards to community activities such as employment and volunteering, tt has been reported that less than 5% of individuals with complex communication needs are employed [2] and although 13% of Americans have a disability, only 5% of volunteers report having a disability, and only 1% report having a developmental disability [3]. The low proportion of participation in these activities among individuals with complex communication needs is likely related to the lack of effective communication supports to meet the functional demands of participation within real-world settings. Visual scene displays capture meaningful events within an individual’s life in an integrated scene (i.e., photograph), with language concepts embedded as hotspots within the scene [4]. However, most communication technologies support the integration of only static photo visual scene displays, which do not capture the dynamic routines that require communication within real-world activities. In 2014, researchers proposed the use of videos with integrated visual scene displays to facilitate participation and communication within daily activities [5]. Videos with integrated visual scene displays capture dynamic routines that support communication in real settings.

Video visual scene display (Video VSDs) approaches were first examined in a series of research studies as part of the RERC on AAC, a federally funded research and development center in augmentative and alternative communication, funded by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR). Video VSD technology was developed in an Android app called EasyVSD by Erik Jakobs, a software engineer at Invotek, Inc, and a partner in the RERC on AAC.

To date, video VSD interventions have only been examined in case study research projects with a single participant, and have examined only a limited number of skills and interaction opportunities [6, 7].The purpose of this research study, therefore, was to investigate the effect of video VSDs on the number of steps completed by four adolescents with severe disabilities and complex communication needs in a small group activity which included a variety of target skills and communication opportunities. The following questions were addressed: (a) What is the effect of video VSDs on the percent of steps completed (including communication opportunities) during a volunteer vocational activity for four adolescents with complex communication needs? (b) Are the skills maintained overtime? And (c) Is the intervention deemed effective, efficient, and socially valid by key stakeholders?



This research study makes use of a single case multiple-baseline across participants experimental design [8] to evaluate the effects of video VSDs on the completion of the steps in a vocational task. The independent variable is the video VSD app (i.e., videos with integrated VSDs and embedded hotspots), along with brief instructional sessions. The dependent variable is the percentage of steps in each task completed independently (including communication opportunities). The study involves three phases: baseline, intervention, maintenance.

Approval for this study was obtained from the university Human Research Ethics committee, participating school district, and the participants’ families prior to initiation of study procedures. All participants attended a rural school district in the northeastern United States.


Four adolescents with complex communication needs participated in the study. The participants were between 14 and 20-years-old. The participants were recommended for this study by their classroom teachers based on the participant’s reported dependence on prompts to complete tasks within the classroom and community, and a need for additional training on vocational skills. Two of the participants had autism and limited communication (e.g., typically communicated with spoken 2-4 word phrases when prompted by a staff member). The remaining two participants had Down syndrome and typically communicated with speech approximations for words and brief phrases; however their speech was only intelligible to familiar partners. All students typically received their instruction in special education classrooms, except for one or two elective classes with their general education peers each semester.



The study used a 12-inch Samsung Galaxy Note Pro 7®  tablet that contained the video VSD application (i.e., EasyVSD). Figure 1 provides a screenshot of a VSD created in the EasyVSD app for the activity ‘Packing the Backpacks.’To create a video VSD, the researcher used previously uploaded videos of the target activity (these videos were stored on the photo library on the tablet) and uploaded them into the app.Then, the researcher paused the video at the end of each step in the task, which automatically created VSDs (i.e., photos) at these junctures. Lastly, the VSDs were programmed with hotspots containing the relevant vocabulary (e.g., “Can you let me in the storage room?’). When viewed by the participants, the selected video filled most of the screen of the tablet, aside from the navigation icons and a play/pause button positioned vertically on the left-hand side of the screen (as illustrated in Figure 1).


Each participant participated in three different phases of investigation during this study: baseline, intervention, and maintenance. In each of the three phases, the session for the activity began with a probe. A minimum of five baseline sessions were conducted separately for each participant prior to intervention [6]. Once a participant demonstrated a stable baseline for the activity (i.e., a minimum of five data points if baseline was stable, additional data points if the baseline was not stable), the researcher introduced the intervention for the activity while other participants remained in baseline. Intervention sessions were conducted approximately three times per week with sessions lasting 1-1.5 hr. All of the sessions were conducted by the researcher (i.e., first author). The participant continued in intervention for the activity until an intervention effect was observed. An intervention effect was defined as the participant’s completion of at least three steps in the task analysis above the highest baseline probe (e.g., completion of at least five steps if the highest baseline probe was two steps) for three consecutive sessions.

Probes were completed to measure the participants’ performance during the volunteer task. During the probes, the participants were brought to the volunteer setting. The researcher provided an initial cue at the start of the activity (e.g., “It’s time to pack the backpacks”), and then provided wait time. If the participant completed the step in the task analysis for the activity successfully, the researcher provided wait time for the participant to complete the next step in the activity. If, at any point, the participant did not initiate action within 5 s, completed the step incorrectly, or completed a step out of sequence, the researcher blocked the view of the participant and completed the targeted step.

In baseline, only probe sessions (as described above) were completed. The participants did not have access to the video VSD app (as was characteristic in their school program) and no instruction was provided. Sessions in the intervention phase included both a probe and instructional sessions. Each participant had access to the video VSD app during both probe and instructional activities. The probe sessions in the intervention phase were identical to the sessions in baseline with the exception of the provision of the tablet (with the video VSD app), which was given to each participant at the start of the probe session. Immediately after the completion of the probe session, the researcher provided an instructional session.   Instructional sessions included a review of the video VSD app for the activity and guided practice (using a least to most prompting hierarchy) in using the app to complete the activity.

Each participant learned a five-step procedure to operate the video VSD app by the researcher during the modeling session: (1) press the play button (the arrow located at the top left); (2) watch the video segment depicting the step from the task analysis (shown in the large area on the right side of the screen); (3) complete the step by performing the motor act and/or fulfill the communication act portrayed in the segment by selecting the hot spot from the VSD (i.e., touching the hotspot or circle around the model to retrieve the spoken message, “I am going to put the backpacks in the storage room”); (4) press the play button again to watch the video model of the next step or select the thumbnail of the video of the next step from the menu on the left hand side; and (5) repeat steps 1-4 for each step to complete the entire task. For example, when a participant finished the step of zipping up the backpacks, the participant then watched the video model of the next step in the task (i.e., telling the supervisor that they are going to put the backpacks away). The app paused automatically at the conclusion of the video model for this step (i.e., putting the backpacks away), and included a VSD with hot spots of appropriate vocabulary to support the participants’ communication (i.e., the phrase “I am going to put the backpacks in the storage room”).

Mastery was defined as three consecutive probes above 90% task completion of independent steps complete.

When mastery was observed for a task, the task entered the maintenance phase. During this phase, no instructional sessions were provided for a task; the participants were simply given an opportunity to participate in a probe that followed the same conditions as an intervention probe (i.e., the participant had access to the table with video VSD). Each participant demonstrated the mastery criterion for the activity, and therefore maintenance data were collected for all participants at two and four weeks.


The participants completion of target steps of the volunteer task during baseline phase (i.e., without access to the video VSD app) was compared to the steps completed correctly during the intervention phase (i.e., with access to the video VSD app). Each session was video-recorded and a graduate student blind to the goals and conditions of the study coded the video recordings post hoc. Percent of target steps completed (i.e., the dependent variable) was calculated by dividing the number of steps completed independently by the total number of steps and multiplying by 100. The data were summarized for each session and graphed separately for each participant. The data were analyzed visually for changes in trend, slope, and variability to explore the effects of the video VSD app on independent task completion.


This image displays a screenshot view of the EasyVSD app from the view of the participant. The majority of the image shows the still frame of a videoVSD for the task of packing backpacks. Aside from navigation icons and a play/pause button positioned vertically on the left-hand side of the screen, the videos filled the majority of the screen of the tablet. The image is of an individual standing and holding two backpacks and a tablet. The individual is standing close to a work supervisor. The thumbnails on the left hand side of the screen show the individual steps in the task.
Figure 1. EasyVSD app screenshot
Results indicated changes in performance immediately upon introduction of the app for each participant. Participant 1 increased from an average of 6% of steps completed during baseline to an average of 96% during the final intervention probes. Participant 2 averaged 2% at baseline and increased to 100% by only the fourth intervention probe. Participant 3 had an average performance of 9% in baseline and increased to an average of 96% during intervention. The final participant, increased from an average of 18% at baseline to 100% at just the second intervention probe. Participants 1 and 2 maintained their performance at both 2 and 4-week maintenance probes. Participants 3 and 4 maintained their performance at a 2-week maintenance probe. Social validation measures by the teachers, special education director, and the speech-language pathologist reported that each professional found the intervention effective, efficient, and important.


The intervention using video VSDs was successful in teaching four adolescents with complex communication needs to independently complete a volunteer activity by following a sequence of steps that included opportunities for communication. Each participant reached mastery for the task and each participant made use of communication supports within the app to communicate with multiple communication partners within the task. The rapid acquisition of both motor and communication skills in such a short period of time offers preliminary evidence that the integration of communication and activity supports, as provided by a video VSD approach, is a promising approach to supporting participation for adolescents with complex communication needs in real-world settings.

This project also provides evidence of the benefits of providing communication supports for meaningful participation in socially valued activities –such as volunteer activities. The volunteer activity provided over 160 students with needed food backpacks each week, and the work of the study participants was clearly valued by community members.


This study provides evidence that the use of video VSDs by adolescents with complex communication needs can result in both increased independence for task completion, and successful interaction with a variety of communication partners. Meaningful participation for persons with complex communication needs, whether in volunteer or employment settings, necessitates support for both vocational and communication skills. Video VSD technology appears to be a promising approach to providing effective, efficient, and socially valid supports for persons with complex communication needs to increase participation and communication in valued societal activities.


[1] Beukelman, D., & Mirenda, P. (2013). Augmentative and alternative communication: Supporting children and adults with complex communication needs. Baltimore MD: Brookes Publishing.

[2] McNaughton, D., & Bryen, D. N. (2002). Enhancing participation in employment through AAC

technologies. Assistive Technology14, 58-70.

[3] Miller, K., Schleien, S., Rider, C., Hall, C., Roche, M., & Worsley, J. (2002). Inclusive volunteering: Benefits to

participants and community. Therapeutic Recreation Journal, 36, 247-259.

[4] Light, J., & McNaughton, D. (2012). Supporting the communication, language, and literacy development of children with complex communication needs: State of the science and future research priorities. Assistive Technology, 24:34-44.

[5] Light, J., McNaughton, D., & Jakobs, T. (2014). Developing AAC technology to support interactive video visual scene displays. RERC on AAC: Rehabilitation Engineering Research Center on Augmentative and Alternative Communication. Retrieved from

 [6] Babb, S., Gormley, J., McNaughton, D., & Light, J. (2018). Enhancing Independent Participation Within Vocational Activities for an Adolescent With ASD Using AAC Video Visual Scene Displays. Journal of Special Education Technology, 0162643418795842.

[7] O’Neill, T., Light, J., & McNaughton, D. (2017). Videos with integrated AAC visual scene displays to enhance participation in com- munity and vocational activities: Pilot case study with an adolescent with autism spectrum disorder. Perspectives SIG 12: Augmentative and Alternative Communication, 12, 55–69. doi: 10.1044/persp2.SIG12.55

 [8] Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26–38. doi:10.1177/0741932512452794