Motion Recognition System for Remote Vocational Support

Michael McCue, PhD, CRC1 , Edmund Lopresti, PhD2 , Jessica Hodgins, PhD3 , Adam Bargteil, PhD3 & Andrea D. Fairman, MOT, OTR/L, CPRP1

1. University of Pittsburgh School of Health and Rehabilitation Sciences Department of Rehabilitation Science and Technology
2. AT Sciences, LLC
3. Carnegie Mellon University The Robotics Institute: School of Computer Science, Pittsburgh, PA 15260


A system is in development to provide remote guidance and assessment in vocational task performance for persons with cognitive disabilities. The system, based upon motion technology, is trained to identify individual, discrete movements that make up the actions of a job-related activity. The remote system recognizes whether the actions are accurate and performed in the correct sequence. A pilot study is being conducted to determine if the remote observation of activity performance using motion based sensors (accelerometers) provides comparable information to direct observation as is commonly provided as part traditional job coaching found in vocational rehabilitation programs.


job coaching, telerehabilitation, vocational rehabilitation, motion recognition


Persons with cognitive disabilities, resulting from conditions such as brain injury, often require rehabilitation services and long-term support for successful community reintegration, including employment.  Research has demonstrated that some of the more effective approaches for persons with cognitive disabilities involve the application of specific strategies, including accommodations and supports, occurring in the natural environment where obstacles are encountered that impact performance. 

In vocational rehabilitation one such intervention that has been effective with this population is supported employment (SE).   SE provides individualized services to an individual with a disability in the workplace through a job coach who assists the client in maintaining satisfactory job performance (1, 2).  Job coaches often address cognitive problems that can impact vocational performance by monitoring the client as he or she performs job related tasks and providing cuing responses to address problems with initiation, attention, memory, learning and sequencing. In order for this SE approach to be effective, job coaching services are provided in person in the natural employment setting and often require full-time, one-to-one intervention for an indefinite period of time. While there is support of job coaching as an effective intervention for persons with traumatic brain injury (TBI), commitment of staff and financial resources can be very challenging for vocational rehabilitation providers (3).  As a result, consistent provision of job coaching services is often not readily available to rehabilitation consumers to be provided on the “as needed” basis that is recommended. Funding can tend to steer programming affecting the intensity, duration and availability of job coaching services (4, 5, 6).Another problem that has been  identified in placing a job coaches the work environment is that it can draw unwanted attention to the consumer (and the disability) and may not be readily accepted by other persons in the environment (for example, an employer or co-worker).

Image shows model in which the remote job coach is able use data from accelerometers to automatically recognize components of the task (or errors) and provide feedback to a client as appropriate
Figure1: Remote Job Coaching Model (Click for larger view)

A telerehabilitation-based “in vivo” intervention is being developed to support consumers who have cognitive disabilities.We are implementing activity recognition using accelerometers to monitor specific pre-determined work behaviors and, ultimately, to deliver cues and instruction in response to problem behaviors. The system is trained using video labeling synchronized with data regarding motion from accelerometers worn while performing a given vocational task being performed correctly, as well as for common errors.  A model is then developed to automatically recognize components of the task (or errors) and provide feedback to a client as appropriate (e.g., a task sequence error triggers a task instruction delivered via headphone to repeat the step in the proper sequence). A diagram of the system is shown in Figure 1.

The goals of the project are as follows:

  1. To use motion technology to ascertain task specific, qualitative behavioral data regarding individuals with cognitive/behavioral disabilities performing job-related tasks.
  2. To determine the reliability and validity of this technology for “in vivo” behavioral assessment.
  3. To use the capacity of the technology to deliver specific task instructions (job coaching) based upon the behavioral assessment described above.
  4. To determine the utility (validity, cost-effectiveness) of the technology for enhancing performance of specific work tasks in persons with significant cognitive disabilities.
  5. To identify the specific environmental and client characteristics that indicate that such technology is likely to be effective.      


A series of focus groups and interviews with employers and vocational rehabilitation providers, and observations at industrial and food service work sites were conducted. Through this collaboration we identified food preparation as a domain in which many people requiring job coaching are employed and which has tasks for which this technology might be appropriate.  More specifically, we identified the task of grilling hamburgers in a fast food setting as an initial application. In institutional settings this task requires a long series of actions which must be performed in proper sequence and pace to create a high quality product.

This hamburger grilling task was initially replicated in a motion capture lab at Carnegie Mellon University.  Video was captured for an able-bodied subject performing the task.  A machine learning technique (AdaBoost) was then used to automatically identify features in the person’s movement patterns by which the system could automatically recognize components of the task (e.g. flipping burgers, salting burgers, placing burgers on or taking burgers off the grill).

Performance of the model is shown in Table 1, analyzing the testing data for a model based on the training data.

Table 1: Performance for the AdaBoost Model


Total Actions

False Positives

False Negatives

Merged Actions

Separated Actions



1 (0.15%)


18 (2.6%)

2 (0.29%)






3 (1.3%)

















38 (8.5%)

1 (0.22%)

None of the actions were mislabeled.  There was one erroneously labeled flip, but it appears this was a flip which was aborted when the subject realized he'd forgotten to press the burgers. The most common errors are actions which are merged or separated.  Merged actions occur when actions are performed with very little time between them and actions are split when there the subject appears to stop mid-action.

Image shows an investigator engaged in the simulated burger flipping task as data is being collected regarding performance via accelerometers and an overhead camera to further refine and train the system.
Photo 1: Simulated Burger Flipping - Lab at the University of Pittsburgh (Click for larger view)

The set-up has since been relocated to a lab at the University of Pittsburgh as we further train and refine the system with additional able-bodied subjects and begin to apply this approach with a small population of persons with TBI in a pilot study.

The performance of the model has been tested on additional able-bodied subjects and compared to traditional job coaching observations. IRB approval has been obtained and subjects are currently being recruited to determine the efficacy of the system as applied to a clinical population of persons with traumatic brain injury.


 Initial work has yielded a model which is able to correctly identify component tasks for multiple subjects without disabilities.  Next steps include (1) testing the model’s ability to recognize movement patterns for additional subjects; (2) testing the model’s ability to recognize movement patterns for individuals with cognitive impairments; (3) processing the activity recognition output to determining when the task is being performed correctly and when errors are being made (e.g. pressing should follow salting, otherwise there’s an error); (4) develop a user interface to prompt the client when errors are made (or to provide occasional praise); and (6) develop an interface to share a summary of the data (e.g. success rate, types of errors) with a job coach or other support person.

Once these tasks are accomplished, the complete system will be tested for a larger sample of people with cognitive impairments, in which the system will observe their performance of the task and provide cues as needed.  Data will include the match between the system’s identification of task errors and the true incidence of task errors, as well as the number of uncorrected errors with and without the intervention.  This evaluation will be used to determine whether the system is effective in assisting people with disabilities, as well as the settings in which it is likely to be effective; for example, whether the system has promise as a cueing system on the final job site or is more likely to be effective as a training tool.


  1. Johnstown, R. (1998). How people get back to work after severe head injury? A ten year follow up study. Neuropsychological Rehabilitation, 8(1), 61-79.
  2. Preston, B., Ulincy, G. & Evans, R. (1992). Vocational placement outcomes using a transitional job coaching model with severe acquired brain injury. Rehabilitation Counseling Bulletin, 35(4), 230-239.
  3.  Wehmen, P., Kregal, P., West, M.& Cifu, D. (1994). Return to work for patients with traumatic brain injury: analysis of costs. American Journal of Physical Medicine and Rehabilitation, 73(4), 280-282.
  4. Hart, T., Dijkers, M., Fraser, R., Cicerone, K., Bogner, J.A., Whyte, J., Malec, J. & Waldron, B. (2006). Vocational services for traumatic brain injury: treatment definition and diversity within model systems of care. Journal of Head Trauma Rehabilitation, 21(6), 467-482.
  5.  Malec, J.F., Buffington, A.L. Moessner, A.M. & Degiorgio, L. (2000). A medical/vocational case coordination system for persons with brain injury: an evaluation of employment outcomes. Archives of Physical Medicine and Rehabilitation; 81(8), 1007-1015.
  6. Gamble, D. & Moore, C.L. (2003). Supported employment: disparities in vocational rehabilitation outcomes, expenditures and service time for persons with traumatic brain injury. Journal of Vocational Rehabilitation, 19, 47-57. 


This study was delivered through the Rehabilitation Engineering Research Center (RERC) on Telerehabilitation and funded by NIDRR Department of Education, Washington DC, Grant #H133E040012.

Author Contact Information:

Andrea D. Fairman, MOT, OTR/L, CPRP, Graduate Student Researcher, University of Pittsburgh, School of Health and Rehabilitation Sciences, Department of Rehabilitation Science and Technology, Forbes Tower, Suite 5051, Pittsburgh, PA 15260, Office Phone (412) 383-6884 (123) 456-7891 EMAIL: