Applications Of Video Visual Scene Display Technology In A Vocational Setting

Salena Babb1, Jessica Gormley M.A., CCC-SLP1, Janice Light, PhD1, David McNaughton, PhD1

1 The Pennsylvania State University


Only 25-50% of adults with autism spectrum disorders (ASD) are employed [6] a disturbingly low rate of employment both in comparison to other individuals with disabilities [2] and to the population at large. Characteristics typically associated with ASD, such as difficulties in learning new skills (e.g., following spoken directions), and working independently (e.g., completing tasks without prompting) can make it difficult for individuals with ASD to participate in the educational and workplace training activities needed to obtain employment (Hendricks, 2010). In addition to the challenge of learning to perform new skills independently, persons with ASD often struggle with the communication skills that are identified by employers as key to success in the workplace [1]. These communication challenges are frequently observed for the 20-30% of individuals with ASD who do not develop the ability to make use of speech to communicate, and who are described as having complex communication needs [7]. Without appropriate intervention, difficulty with speech can act as a severe barrier to communication and participation, and it has been estimated that the employment rates for individuals with complex communication needs are even lower (less than 5%) than those for individuals with ASD [5].

One method to display vocabulary on an AAC system is through use of visual scene displays (VSDs) to capture meaningful events within an integrated scene (e.g., photograph) with language concepts embedded as hotspots within the scene [2]. Current technology supports the integration of only static VSDs within an AAC system; however, Light, McNaughton, and Jakobs [4] proposed the use of videos with integrated VSDs (Video-VSDs) to facilitate participation and communication within daily activities. In real-world settings, such as a vocational environment, video-VSDs may more effectively support communication than static VSDs because Video-VSDs capture the spatial and temporal contexts of communication opportunities, preserving the dynamic relationships and engagement cues found in real-world interactions. Furthermore, automatic pausing of videos at key segues mark an opportunity for communication and provide vocabulary for an individual to fulfill communication demands. Video-VSDs capture dynamic routines within the learner’s life (e.g., vocational activities) and sequentially lead the learner through the activities, promoting independence.

The purpose of this study was to evaluate the effects of Video-VSDs using a tablet-based app (i.e., EasyVSD) on the percentage of steps completed (including fulfilling communicative opportunities) by a high school student with autism spectrum disorder (ASD) within vocational activities.



A multiple-probe across behaviors design was used to determine the effects of the Easy VSD app, a tablet-based app that incorporates use of Video-VSD, on participation and communication within vocational activities in a library setting.


Approval for this study was obtained from the university Human Research Ethics committee, participating school district, and the participant’s family prior to initiation of study procedures. The participant, James (pseudonym), was an 18-year-old male with a diagnosis of autism spectrum disorder (ASD). He attended a rural school district in the north-eastern United States. James received his instruction in a self-contained special education classroom and attended one elective class (e.g., cooking) with his general education peers each semester. James typically communicated through ‘yes/no’ responses (e.g., signaling ‘yes’ with a thumbs up) to simple spoken questions from communication partners (e.g., “Do you want pizza?”). He would point to items in the environment on request when provided a choice among preferred items (e.g., pointing to the pizza, rather than a hot dog). He did not use any form of assistive technology to support communication.

James typically required multiple prompts (e.g., the repeating of the question) in order to respond to others, and was less likely to respond with unfamiliar partners. He also required multiple prompts (e.g., repetition of verbal instructions, modeling of expected behavior, physical guidance) in order to complete basic tasks and skills outside of his normal routine. James’ hearing and vision skills were both reported to be within the normal limits. James was able to identify the most letters and letter sounds and would accurately identify approximately 15-20 sight words; however, he demonstrated no functional use of reading or writing. James was recommended for this study by his teacher and transition coordinator based on his challenges with dependence on prompts to complete tasks within the community.


A 12-inch Samsung Galaxy Note Pro 7® tablet that contained the EasyVSD application was used in the intervention phase of the current study. Easy VSD is an application that allows for the creation of Video-VSDs. To create each Video-VSD, videos of each step of the task were captured by the researcher and uploaded to the tablet. VSDs were then integrated into the video using the EasyVSD app at the end of each step. When communication opportunities were warranted, the researchers programmed a hotspot by circling the key referent depicted on the touch screen and by audio recording the communicative turn (e.g., “Can I look for more books?’).

When the EasyVSD app was viewed by the participant, the videos filled the majority of the tablet screen. Navigation icons and a play/pause button was positioned vertically on the left-hand side of the screen with the play/pause button was located at the top of the page. When a video was viewed, the play button became a pause button. Below the play/pause button, the app contained two menu bars positioned vertically. The menu bar on the left contained a single still VSD (i.e., thumbnail representations) displaying the current task (e.g., putting books away) that the participant was completing. The thumbnails on the right side displayed each of the steps in the task, represented by still VSDs within the task which depicted the start of each step or video. These thumbnails were used to navigate between tasks and the steps within tasks and were highlighted with a blue border to indicate the video currently playing. When the video segments automatically stopped (i.e., to indicate step completion or a communication opportunity) the still VSDs provided the participant with the opportunity to complete the subsequent step.


Activity probes were completed to measure James’ performance during each vocational task. During the probes, the student was brought to the library work setting). The researcher then provided an initial cue (i.e., ‘It’s time to _____’) at the beginning of the task. The researcher then waited 5 seconds and if there was no response from the participant or an error occurred, the researcher blocked the view of the participant to the greatest extent possible, completed the step, and said, “What’s next?”. An error was defined as incorrectly completing a step or completing a step out of sequence.

A minimum of five baseline sessions were collected for each task. During each baseline session, James completed the activity probe with no instructional feedback and no access to the Easy VSD app. Once a stable baseline was achieved for Task 1, the intervention (i.e., EasyVSD app and a brief instructional session) was introduced. To operate the app, James was taught to follow these steps: (a) press the play button, (b) watch the video segment portraying one step from the task analysis, (c) perform the step or fulfill the communication opportunity depicted in the segment (d) select the thumbnail of the next video from the left menu or press the play button again, (e) repeat steps 1-5 for each video segment to complete the entire task. Instructional sessions were conducted approximately once per week with sessions lasting 1-1.5 hours and involved James completing the target task with the researcher providing a least to most prompting hierarchy. Intervention probes were collected the following work session after the instructional sessions were provided. Instructional sessions were conducted immediately following the completion of the intervention probe. The probes conducted during intervention were identical to the probes conducted during baseline, except that the tablet with the video VSD app was available to the participant during the intervention probe. Mastery was defined as three consecutive probes above 80% task completion of independent steps complete.

When mastery was observed for a task, the task entered the maintenance phase. During this phase, no instructional sessions were provided for a task; the participant was simply given an opportunity to participate in a probe that followed the same conditions as an intervention probe (i.e., the participant had access to the table with video VSD). Maintenance probes were collected for activities one and two. Maintenance data for activity three were not collected due to the end of the school year. Maintenance probes were conducted at one, three, and five weeks after intervention for tasks one and two. Procedures were available for booster sessions if the participant demonstrated performance below 80% after instruction was withdrawn, however this was not observed.

In order to assess generalization, data was collected for the participant on a task for which no instruction was provided: a paper shredding task. As with all tasks, five data points were collected in the baseline phase for this task. In contrast to tasks one, two and three, no review of the video VSD or guided practice was provided for the generalization task. The tablet with the interactive VSD app (programmed for the paper shredding activity) was given to the participant, and the researcher then provided an initial cue (i.e., ‘It’s time to _____’).  The probe procedures used for these generalization sessions followed the procedures used for the intervention and maintenance session probes.


James’ completion of target steps of each work task during baseline phase (i.e., without access to the EasyVSD app) was compared to his steps completed correctly during the intervention phase (i.e., with access to the EasyVSD app). Each session was video-recorded with the first and second researcher coding video recordings post hoc.  Percent of target steps completed (i.e., the dependent variable) was calculated by dividing the number of steps completed independently by the total number of steps and multiplying by 100. The data were summarized for each session and graphed separately for each task in the order in which they were collected. The data were analyzed visually for changes in trend, slope, and variability to explore the effects of the video VSD app on independent task completion [3].


This image displays a screenshot view of the EasyVSD app from the view of the participant. The majority of the image shows the still frame of a videoVSD for the task of putting books away. Aside from navigation icons and a play/pause button positioned vertically on the left-hand side of the screen, the videos filled the majority of the screen of the tablet. The image is of an individual standing at a table in a library. There are several books in two organized piles in front of the individual as well as the empty box that was used to transport the books to the table. At the bottle of the image, a text label reads the steps for the task, “Tell a staff member you are going to put the books away.” This text is accompanied by a circle with the number 4 inside. Additionally, a dark blue outline of a circle accompanied by a circle with the number 5 inside represents the hot spot for the communicative turn within the skill. Aside from navigation icons and a play/pause button positioned vertically on the left-hand side of the screen, the videos filled the majority of the screen of the tablet. To the left of the image, there are navigating tools for the participant to use the app. In the top left corner is a play/pause button, labeled by a circle with a number 2 in it. When a video is playing, the play button becomes a pause button. Below the play/pause button, the app contains two menu bars positioned vertically. The menu bar on the left contained a single still VSD (i.e., thumbnail representations) displaying the current task the participant was completing. This single VSD is labeled by a circle with the number 1 inside and is an image in a square shape outlined in purple. The thumbnails on the right side, labeled by a circle with a number 3 inside, displayed each of the still VSDs within the task and depicted the start of each step or video. These thumbnails were used to navigate between video segments and were highlighted with a blue border to indicate the video currently playing. At the bottom of this image, is a text box that relates each labeled part of the image to the corresponding number in each circle. The text reads, “Note: (1) topics for selection, (2) play button, which becomes a pause button when the video plays, (3) specific steps within each task, (4) text on screen that describes the current step of the task analysis, (5) hot spot for communicative turn”.
Figure 1. EasyVSD app screenshot
Results indicated rapid performance changes upon introduction of the app. During baseline for task one (checking in books), James completed an average of 8% of the steps correctly. James increased to 42% correct after the first intervention session, and 100% after the second intervention session. For task two (sorting books), James averaged 5% of steps completed correctly during baseline, and increased to 90% correct on each of the first three probes after intervention began. For task 3 (dye cut), James averaged 15% on the percentage of steps completed correctly for the 8 baseline sessions. James to 50% steps correct in the first intervention probe. Mastery was achieved for task 1 at the end of 6 intervention sessions, for task 2 at the end of 3 intervention sessions, and for task 3 at the end of 4 intervention sessions. Maintenance probes were conducted for tasks 1 and 2: Task completion scores were 90% or higher on the maintenance probes for both tasks.

Task 4, paper shredding, was used as a generalization activity. James had a mean performance of 9% of steps completed independently on the five baseline probes. In the generalization probe, James had access to the tablet with the app, but no modeling or instruction was provided at any time. Again, due to the end of the school year, it was only possible to collect a single generalization probe, at which time James completed 63% of the steps correctly.


The intervention using video VSDs (as available in the EasyVSD app) was successful in teaching an adolescent with ASD to independently complete three work tasks by following a sequence of steps that included opportunities for communication. James reached mastery for three tasks within his vocational setting of the library in a small amount of time (i.e., 3-6 intervention sessions). He also made use of communication supports within the app to ask a supervisor to start a task, let a supervisor know he was finished with a task, and seek support while conducting tasks (i.e., “Can you check my work?”; “Can I check for extra books?”).

The rapid acquisition of both vocational and communication skills in such a short period of time provides preliminary evidence that the integration of communication and activity supports, as supported by a video VSD approach, is a promising approach to supporting participation for adolescents with ASD in real world vocational settings. Further, this study extends the video modeling and video prompting research to incorporate AAC and opportunities for communication within the video prompting technology.


Results provide preliminary evidence that videos with integrated VSDs may serve as an effective means to maximize independent participation and communication for individuals with CCN and ASD in real world contexts. Ultimately, this assistive technology could reduce dependence on aides (e.g., job coaches, paraprofessionals) and create increased opportunities for employment and independent participation in meaningful community activities.


[1] Bryen, D. N., Potts, B. B., & Carey, A. C. (2007). So you want to work? What employers say about job skills, recruitment and hiring employees who rely on AAC. Augmentative and Alternative Communication, 23, 126-139.

[2] Hendricks, D. (2010). Employment and adults with autism spectrum disorders: Challenges and strategies for success. Journal of Vocational Rehabilitation, 32, 125-134.

[3] Kazdin, A. E. (2013). Single-case research designs (2nd). New York, NY: Oxford.

[4] Light, J., McNaughton, D., & Jakobs, T. (2014). Developing AAC technology to support interactive video visual scene displays. RERC on AAC: Rehabilitation Engineering Research Center on Augmentative and Alternative Communication. Retrieved from https://rerc-aac.psu.edu/development/d2-developing-aac-technology-to-support-interactive-video-visual-scene-displays/

[5] McNaughton, D., & Bryen, D. N. (2002). Enhancing participation in employment through AAC technologies. Assistive Technology, 14, 58–70.            

[6] Wehman, P., Schall, C., Carr, S., Targett, P., West, M., & Cifu, G. (2014). Transition from school to adulthood

     for youth with autism spectrum disorder: What we know and what we need to know. Journal of Disability Policy Studies, 25, 30-40.

[7] Wodka, E. L., Mathy, P., & Kalb, L. (2013). Predictors of phrase and fluent speech in children with autism and severe language delay. Pediatrics, 131, 1128-1134.