Myoelectric Modeling of Joystick Control for Adaptive Smart Wheelchairs

Tim D. Yang1-3, Laura A. Rice, PhD, MPT, ATP2, Ann David1, Seth A. Hutchinson, PhD4-6, Yih-Kuen Jan, PT, PhD1-3

1Rehabilitation Engineering Lab, 2Kinesiology & Community Health, 3Computational Science & Engineering, 4Coordinated Science Lab, 5Electrical & Computer Engineering, 6Beckman Institute,

University of Illinois, Urbana-Champaign, IL

ABSTRACT

Power wheelchairs (PWCs) potentially effect increased independence for people with severe mobility disabilities. However, many users report difficulty in performing activities of daily living or even functional steering. Smart wheelchairs aim to accommodate this population by augmenting power wheelchairs with computational assistance. To maximize the driver’s independence, computational intervention should be invoked only when necessitated by contextual and human factors. In this study, we focused on detecting human factors that entail computational assistance. Our specific aim was to computationally model the myoelectric behavior of arm muscles during PWC driving. Four driving tasks by two participants were fitted with vector autoregression models, which were cross-validated using a leave-one-out procedure. Individualized models of each participant yielded at least 90% accuracy, and a combined model of both participants yielded 80% accuracy. While more participants and training sequences are needed, the myoelectric models showed promise at characterizing arm activities during PWC driving. These results can inform the development of smart wheelchairs that adapt the level of shared control in response to the state of the driver.

BACKGROUND

Power wheelchairs (PWCs) empower people with severe mobility disabilities to live more independently. In 2007, there were 173,000 PWC users in the Medicare system alone (Levinson, 2009), and subsequent totals have likely risen as a result of increased accessibility, accommodations, and aging. However, PWC users can present with secondary impairments that hinder or prevent functional PWC usage. A survey of 200 clinicians indicated that using a PWC made it “extremely difficult or impossible” to perform activities of daily living in approximately 9–10% of clients and that secondary impairments prevented functional steering in up to 40% of clients (Fehr, Langbein, & Skaar, 2000). Smart wheelchairs aim to accommodate this population by augmenting PWCs with computational assistance (Simpson, 2004).

However, smart technologies require a careful balance between human control and computational assistance. While overly intrusive automation will generally result in successful task completion, the Human Activity Assistive Technology (HAAT) model encourages assistive technology professionals to consider human and contextual factors beyond the activity itself, including why and how activities are performed (Cook & Polgar, 2015). For many clients, the ability to perform activities independently embodies a personal meaning beyond merely whether the task was completed. Furthermore, excessive computational assistance may induce over-reliance on automation and degradation of skills (Cooper, Ohnabe, & Hobson, 2006). To minimize these effects, smart technologies should relinquish control to the driver as much and as often as possible.

By detecting contextual and human factors in real time, smart technologies can adaptively modulate the degree of computational intervention based on the detected states of the context and driver (Figure 1). Previously, we introduced a contextual risk index for PWC driving by using a Kinect sensor to map the environment and a Markov model to compute driving costs (Yang, Patil, & Jan, 2014). In the current paper, we focus on the state of the driver by modeling the myoelectric behavior of the driver’s arm during PWC driving. The long-term goal is to predict the onset of motor difficulties, which can then be combined with the contextual feedback from our Kinect-Markov framework to assign dynamic levels of shared control.

PURPOSE

The purpose of this study was to computationally model myoelectric behavior in the arm during joystick control of basic PWC driving tasks. Results will be used to inform the development of adaptive smart wheelchairs.

METHODS

Participants

The conceptual plot depicts how shared control should be adapted in response to contextual risk (vertical axis) and human capacity (horizontal axis). When human capacity is high and contextual risk is low, the system is mostly under human control. When human capacity is moderately high and contextual risk is moderate, the system is less under human control. When human capacity and contextual risk are both moderate, the system is roughly under 50/50 control. When human capacity is low and contextual risk is high, the system is mostly under computer control.
Figure 1: The level of shared control in smart technologies can be adaptively modulated if the systems have real-time awareness of contextual and human factors.
Two members of the Rehabilitation Engineering Laboratory participated in this study. Neither participant presented with a disability.

Instrumentation

All driving sequences were performed in a front-wheel-drive PWC (model C400; Permobil, Inc.; Lebanon, TN). Three-lead myoelectric amplifiers (model EMG100C; Biopac Systems, Inc.; Goleta, CA) were used to sample upper limb myoelectric data via bipolar surface electrodes (model EL507; Biopac System, Inc.; Goleta, CA). The sampling rate was set to 2,000 Hz with a 60 Hz notch option to mitigate power line interference.

Procedure

Electrodes were placed on five arm muscles: flexor carpi radialis, extensor carpi ulnaris, extensor carpi radialis, pronator teres, and triceps brachii. Electrode placements were verified using mild electrical stimulation. Participants were asked to perform four driving activities,

  • Forward (FW): roll forward for 10 m
  • Rightward (RW): turn 90° rightward while moving forward
  • Backward (BW): roll backward for 5 m
  • Leftward (LW): turn 90° leftward while moving forward

 

which were based on tasks #8, #10, and #11 of the Wheelchair  Skills Test for PWCs (Kirby, Swuste, Dupuis, MacLeod, & Monroe, 2002). Each activity was repeated five times with brief resting periods in between.

Data Reduction

The data were centered to remove the DC offset and decimated to 1,000 Hz to reduce the computational modeling cost. A bandpass filter was applied using a Butterworth filter with low- and high-cutoff frequencies of 15 Hz and 500 Hz, respectively. Signals within each myoelectric channel were normalized to the peak amplitude observed throughout each activity (Halaki & Gi, 2012).

Data Analysis

Vector autoregression (VAR) was introduced by Sims (1980) as a method to model the joint dynamical behavior of econometric variables, but has since been applied to other time-series domains. In this study, VAR was utilized to model the behavior of five myoelectric channels during PWC joystick control.

A pth-order VAR process is a linear dynamical system that models current observations using past observations,

C = A P + b + e

(1)

where P contains the past p observations (i.e., p “lags”) of each channel, C contains the current observations of each channel, A is the coefficient matrix, b is the bias vector, and e is the error vector.

In a condensed form, the bias vector b is concatenated to A as its last column, and an all-ones vector is appended to P as its last row,

A [ A   b ]

(2)

P [ P 1 ] (3)
C = A P + e (4)

where the coefficient matrix A can be estimated with the least squares method. The VAR models were custom coded in MATLAB (version 2013a; The MathWorks, Inc.; Natick, MA).

Model Validation

A VAR coefficient matrix was modeled for each class (i.e., A i , where ι FW ,   RW ,   LW ,   BW ) using training sequences. Each A i  model was then applied to validation sequences. For each sequence, the class  with the minimum mean squared error was selected as the prediction,

i = arg   min   E i   vec C   C i ˆ T vec C   C i ˆ

(5)

where C was the true sequence and C i ˆ  was the sequence reconstructed using Ai.

Evaluation sets were constructed using leave-one-out cross-validation, in which each sequence was successively excluded from the training set to serve as a validation sequence. Under this scheme, the total number of evaluation sets equaled the total number of observation sequences. Three VAR models were validated: an individualized model for the first participant, an individualized model for the second participant, and a combined model for both.

RESULTS

Mean squared error was computed as a function of the number of the past p observations for p=1,2,...,50 (Figure 2). Based on the observed performance plateau, a lag order of p=30 was selected for the coefficient matrices of each VAR model (Figures 3, 5, and 7).

Individualized Model (First Participant)

Mean squared error (MSE) is shown as a function of the number of lags (p=1,2,…,50). All four vector autoregression models are shown simultaneously and exhibit the same pattern: from p=1 through p=9, there is a sharp decline in MSE, and from p=10 through p=50, the decline in MSE plateaus. The rightward model begins at approximately 0.016 MSE and plateaus to approximately 0.01; the leftward and forward models begin at approximately 0.014 MSE and plateau to approximately 0.008; and the backward model begins at approximately 0.01 and plateaus to approximately 0.006.Figure 2: Mean squared error as a function of lag.
Using the training models to test the training set resulted in 100% overall accuracy (Figure 4a). Using the training models to test the validation set resulted in 95.0% overall accuracy (Figure 4b). The sole error resulted from a LW sequence being confused for a BW sequence.

Individualized Model (Second Participant)

Using the training models to test the training set resulted in 100% overall accuracy (Figure 6a). Using the training models to test the validation set resulted in 90.0% overall accuracy (Figure 6b). A RW sequence was confused for a BW sequence, and a LW sequence was confused for a FW sequence.

Combined Model

Using the training models to test the training set resulted in 90.3% overall accuracy (Figure 8a). Using the training models to test the validation set resulted in 80.0% overall accuracy (Figure 8b). The most common misclassification was the LW class being confused with the FW class.

DISCUSSION

Coefficient matrices are shown for the forward, backward, leftward, and rightward classes. Each matrix is relatively sparse. Most activity is contained in the lefthand side of the matrix in repeated diagonal bands with magnitudes between -1 and 1.Figure 3: Coefficient matrices of the first individualized model for each activity class: (a) forward, (b) rightward, (c) backward, and (d) leftward.
We used VAR processes to model the myoelectric behavior of the arm during four basic PWC driving tasks. Based on preliminary findings, VAR models showed promise in discriminating between normal and abnormal myoelectric patterns. In particular, the individualized VAR models performed well. Training sequences were classified perfectly, and validation sequences were classified with at least 90% accuracy.

Two 4x4 confusion matrices are shown. Each row represents the true instances of the forward, rightward, backward, and leftward classes; and each column represents the predicted instances of those classes. Correctly predicted instances lie along the diagonal. In the first confusion matrix, the diagonal elements contain values of 20, 20, 20, and 20; and each remaining element contains a value of 0. In the second confusion matrix, the diagonal elements contain 5, 5, 5, and 4; the element in the fourth row (true leftward class) and third column (predicted backward class) contains a value of 1; and each remaining element contains a value of 0. Figure 4: Confusion matrices from testing the first individualized model on: (a) the training set and (b) the validation set.
However, the combined VAR model performed more poorly. Unlike the individualized models—in which no training sequences were misidentified—the combined model yielded 35 misclassified training sequences. That is, sequences that contributed to the construction of the combined VAR model were then misclassified by the very same model. This was likely due to the increased variation from the inclusion of a second participant. Performance further declined in the validation set, in which more than half of all LW sequences were confused with FW sequences. This phenomenon was not observed in the individualized models. Accordingly, the difficulty of the combined model to characterize LW sequences may suggest that inter-participant variation is higher for LW joystick movements compared to other classes. Furthermore, LW class confusion may be related to driver handedness. Since both participants were right-handed, LW joystick movements were adductive; for left-handed participants, the adductive task would correspond to the RW class. Thus, for left-handed participants, the VAR model may produce higher confusion for RW sequences.

Figure 5: Coefficient matrices are shown for the forward, backward, leftward, and rightward classes. Each matrix is relatively sparse. Most activity is contained in the lefthand side of the matrix with magnitudes between -1 and 1. The activity is less structured than in Figure 3. Figure 5: Coefficient matrices of the second individualized model for each activity class: (a) forward, (b) rightward, (c) backward, and (d) leftward.
While our results are preliminary, they demonstrate the potential of myoelectric models for adaptive shared control. The models can be used to inform our smart wheelchair on the degree of shared control needed at any given point in time. For example, consider a smart navigation system that computes a heading angle of ° while the driver executes a heading angle of °. Depending on how closely the driver’s myoelectric signals were to match the VAR model, the smart system could adapt the ratio of computational and human input; i.e., if the VAR model were to indicate a high likelihood that the driver had attempted a desired action, the computational weight could be diminished, whereas if the VAR model were to indicate abnormal myoelectric activity, the computational weight could be augmented.

Two 4x4 confusion matrices are shown. Each row represents the true instances of the forward, rightward, backward, and leftward classes; and each column represents the predicted instances of those classes. Correctly predicted instances lie along the diagonal. In the first confusion matrix, the diagonal elements contain values of 20, 20, 20, and 20; and each remaining element contains a value of 0. In the second confusion matrix, the diagonal elements contain 5, 4, 5, and 4; the element in the second row (true rightward class) and third column (predicted backward class) contains a value of 1; the element in the fourth row (true leftward class) and first column (predicted forward class) contains a value of 1; and each remaining element contains a value of 0.Figure 6: Confusion matrices after testing the second individualized model on: (a) the training set and (b) the validation set.
This study had limitations. First, our sample size was low. Second, our participant sample did not include people with upper limb impairments. Third, our method assumed signal stationarity, and there is no clear consensus on the pure-, quasi-, or non-stationarity of myoelectric signals (Raez, Hussain, & Mohd-Yasin, 2006). Nevertheless, the method performed well in this applied setting.

Coefficient matrices are shown for the forward, backward, leftward, and rightward classes. Each matrix is relatively sparse. Most activity is contained in the lefthand side of the matrix with magnitudes between -1 and 1. The activity is more structure than Figure 5 and resembles the structured bands of Figure 3.
Figure 7: Coefficient matrices of the combined model for each activity class: (a) forward, (b) rightward, (c) backward, and (d) leftward.
In future work, more participants will be recruited, particularly people with upper limb impairments. More joystick actions will be included to accommodate a wider range of movements common to PWC driving, and more observation sequences per action will be collected to increase the model’s robustness. Lastly, feature selection and extraction will be explored to reduce channels and dimensionality, respectively.

CONCLUSION

Two 4x4 confusion matrices are shown. Each row represents the true instances of the forward, rightward, backward, and leftward classes; and each column represents the predicted instances of those classes. Correctly predicted instances lie along the diagonal. In the first confusion matrix, the diagonal elements contain values of 86, 90, 83, 66; the element in the first row (true forward class) and fourth column (predicted leftward class) contains a value of 4; the element in the third row (true backward class) and first column (predicted forward class) contains a value of 7; the element in the fourth row (true leftward class) and first column (predicted forward class) contains a value of 24; and each remaining element contains a value of 0. In the second confusion matrix, the diagonal elements contain 9, 9, 9, and 4; the element in the first row (true forward class) and fourth column (predicted leftward class) contains a value of 1; the element in the second row (true rightward class) and third column (predicted backward class) contains a value of 1; the element in the third row (true backward class) and first column (predicted forward class) contains a value of 1; the element in the fourth row (true leftward class) and first column (predicted forward class) contains a value of 6; and each remaining element contains a value of 0.
Figure 8: Confusion matrices from testing the combined model on: (a) the training set and (b) the validation set.
Vector autoregression showed promise for modeling myoelectric behavior in the arm during PWC driving. Given more participants and training data, these models can be exploited by adaptive smart wheelchairs to dynamically adjust the ratio of human-computer shared control.

REFERENCES

Cook, A. M., & Polgar, J. M. (2015). Assistive Technologies: Principles and Practice (4 ed.). St. Louis, MO: Elsevier.

Cooper, R. A., Ohnabe, H., & Hobson, D. A. (2006). An Introduction to Rehabilitation Engineering. Boca Raton, FL: CRC Press.

Fehr, L., Langbein, W. E., & Skaar, S. B. (2000). Adequacy of power wheelchair control interfaces for persons with severe disabilities: A clinical survey. Journal of Rehabilitation Research and Development, 37(3), 353–360.

Halaki, M., & Gi, K. (2012). Normalization of EMG signals: To normalize or not to normalize and what to normalize to? In G. R. Naik (Ed.), Computational Intelligence in Electromyography Analysis - A Perspective on Current Applications and Future Challenges: InTech.

Kirby, R. L., Swuste, J., Dupuis, D. J., MacLeod, D. A., & Monroe, R. (2002). The Wheelchair Skills Test: A pilot study of a new outcome measure. Archives of Physical Medicine and Rehabilitation, 83(1), 10–18. doi: 10.1053/apmr.2002.26823

Levinson, D. R. (2009). Power Wheelchairs in the Medicare Program: Supplier Acquisition Costs and Services. (OEI-04-07-00400). Washington: Government Printing Office.

Raez, M. B., Hussain, M. S., & Mohd-Yasin, F. (2006). Techniques of EMG signal analysis: Detection, processing, classification and applications. Biological Procedures Online, 8, 11–35. doi: 10.1251/bpo115

Simpson, R. (2004). Smart wheelchairs: A literature review. Journal of Rehabilitation Research and Development, 42(4), 423–436. doi: 10.1682/JRRD.2004.08.0101

Sims, C. A. (1980). Macroeconomics and reality. Econometrica, 48(1), 1–48.

Yang, T. D., Patil, A., & Jan, Y. K. (2014). Individualized performance quantification of power wheelchair driving. Paper presented at the Proceedings of the RESNA 2014 Annual Conference, Indianopolis, IN.

ACKNOWLEDGMENTS

We thank Jeannette Elliott for helpful discussions and Permobil, Inc. for providing the PWC. We thank Yu-Xuan Huang, Ling-Yi Wang, and Yu-Chen Fa for their assistance as summer interns.

Audio Version PDF Version