RESNA Annual Conference - 2019

Upper Limb Robot Rehabilitation Of Post-Stroke Patients Using Mirror Therapy

Binyun Chen


This research describes the initial testing of the ARWED, which is a virtual reality system for physical rehabilitation of patients with reduced upper extremity mobility resulting from a stroke. The purpose of the ARWED is to increase limb Active Range of Motion. The system performs a symmetric reflection of the patients’ healthy limb into a virtual 3D photorealistic model and maps it in real time on to their most affected limb, tapping into the mirror neuron system and facilitating the initial learning phase. Using the developed system, pilot experiments tested the extension of the action-observation priming effect linked to the mirror-neuron system on healthy subjects and one post-stroke patient. The initial assessment of the developed virtual photorealistic 3D hand models with healthy subjects imply that the developed models prime the human motor system in a manner consistent with the human model.

The pilot tests with a post-stroke patient suggest that the virtual reality mirror therapy could trigger muscle activation in patients that are more than 3 years post-stroke. This can further serve as an evidence that the time needed for recovery from stroke is not limited to one year and that additional practice can improve mobility in both the subacute and chronic phases following a stroke. The future work includes increased frequency and duration of training with the device to assess for any changes in the assessment score.


Mirror visual feedback (MVF) was initially utilized by Ramachandran and Rogers-Ramachandran in 996 to alleviate pain and paralysis in amputees [1] and was specifically designed to trick the patient’s brain while transforming their mind [2]. When patients with chronic pain issues anticipate movements to be painful, mirrors help deceive them into thinking that they are not experiencing pain via dynamic feedback to their brains [3]. McCabe, Haigh and Blake have stated that "mirrors and vision are inextricably linked, and the reflected image appears strikingly believable even if deliberately distorted” [4]. Using observation of the uninvolved limb helps to "drive proprioception" in the involved limb, thereby normalizing the "movement process” [5]. Thus, the use of the mirror gives the patient the "impression of having two normal limbs” [6]. The concept behind this "visual input" modality is that it helps patients re-educate, or re-introduce to their altered higher processing neural networkings, a normal relationship between a physical movement and the sensory feedback it provides [2].

Several studies have been conducted looking at the effectiveness of MVF therapy with Chronic Regional Pain Syndrome (CRPS) patients. McCabe et al. studied eight subjects, hypothesizing that a disturbance between motor and sensory cortices was the cause of their CRPS [4]. The investigators found that MVF was helpful for pain reduction in the early stage of CRPS and for stiffness in the intermediate stage; no changes were seen with late stage CRPS [7]. Tichekaar and colleagues reported that mirror therapy, in conjunction with cognitive behavioral therapy, "plays a positive role" in patients with CRPS [8].

A pilot study conducted by Sato and colleagues that employed a virtual reality MVF system to observe pain intensity in patients with CRPS, showed MVF to be a "promising alternative treatment" for this population, though further investigation into such technologies is critical [9]. Altschuler & Jeong described a case report of a patient following a post-operative distal radius fracture who initially was only able to extend her wrist with electrical stimulation; following approximately two months of MVF both in and outside of the clinic, the patient was able to regain thirty-five degrees of active wrist extension.

In order to successfully incorporate MVF into a treatment plan, a mirror and a box are needed. These can easily be constructed from materials found at home, acquired at hardware stores, or purchased as a complete set specifically for MVF from therapy catalogs. Therapy is initiated by asking the patient to describe, with vision occluded, his or her perception of the painful limb. The patient is then asked to sit at a table and position him or herself with the involved extremity behind the mirror (or inside the box) and hidden from view. The uninvolved extremity should be placed in front of the mirror so as to make the reflection look like the contra-lateral limb. Before any movement or exercises, the patient should simply look at the limb in the mirror and focus on engaging in the belief that the mirrored image is in fact his or her contra-lateral limb. The patient is then instructed to perform gentle movements with the uninvolved extremity in front of the mirror while continuing to focus on the mirror This creates the illusion that the movements are occurring bilaterally. At this point, the therapist can watch to see what the hidden involved extremity is doing. McCabe and colleagues have suggested that the way the involved limb is moving is of little importance, as long as it is "bilateral and synchronized” [6].


Robotic Based Stroke Therapy (RBST) has been proven to be beneficial for patients to learn the required perception-action skill. Success of using RBST to help patients relearn other task necessary for Activities of Daily Living (ADL) has been limited [10, 11]. Observational learning of motor skills, however, has been shown to produce transfer across limbs and generalization across muscle groups in the same limb [12], as well as transfer to perceptual tasks [13,14]. Therefore, observational learning may offer a greater benefit regarding transfer to ADLs in comparison to RBST. 

Research over the past ten years suggests that action-observation training improves upper-limb function in children with unilateral cerebral palsy [15]. Human imaging work (PET, fMRI) has revealed a mirror-neuron network (pre-motor cortex, parietal lobe, temporal lobe) that supports our ability to learn through action imitation and action observation [16-18]. This direct link between human visual perception and human action execution is diminished [19] or disappears when non-anthropomorphic motion is observed [20-22]. Recent research started to investigate how action-observation protocols linked to the mirror neuron system may benefit recovery of function after stroke and enhance clinical training protocols to produce transfer of recovered function from the clinic to ADLs [23-25]. Some promises regarding the use of action-observation as a means to tap into the mirror neuron system in the clinic have come from training protocols that use video to help patients mimic ADL [26] and virtual reality systems that transfer the motion of the patient’s real arm to a set of virtual arms in real time [27], [28].

Recent research shows that Virtual Reality Based Therapy (VRBT) is feasible and a cost-effective means of bringing therapy to stroke patients. Furthermore, VRBT has a “wow factor” not experienced through conventional therapies and is considerably safer than RBSTs [29]. Use of remotely monitored virtual reality videogames, regular use of tele-rehabilitation appears to produce improved hand function and forearm bone health in adolescents with chronic disability. Improved hand function appears to be reflected in functional brain changes [30]. Additionally, virtual/augmented reality exercise programs have been shown to improve the ambulation ability of subjects with cerebral palsy [31]. The use of virtual or augmented reality in physical therapy has produced some encouraging results which demonstrate the potential of using these techniques for all types of physical therapy, including stroke therapy [31-34].

For stroke, motor recovery of the upper extremity plateaus in the first year after the initial incident according to clinical and biomechanical measures. However, there is some evidence that the time needed for recovery is not limited to one year and that additional practice can improve mobility in both the sub-acute and chronic phases following a stroke [35]. The latter leads naturally to the concept of “functional potential”. The additional recovery is statistically significant and provides a baseline effect with which to work. However, patients often only achieve small levels of improvement to their mobility.

Given the limitations of recovery, it is necessary to find novel tools and methods for retraining the motor system. The current paper explores combining action observation and virtual reality into virtual reality mirror therapy for improving upper limb mobility for post-stroke patients.


Figure 1. The ARWED maps the healthy onto the impaired limb
Virtual-reality systems aim to integrate additional, computer-generated information within the human perception of the physical world [31]. In our case, human vision is targeted and the aim is to modify visual input of some object within the visual field of the subject. If the original object is an upper limb with a limited range of motion, then the modified object is a realistic 3D model of the subject’s upper limb. In order to perform the virtual/modified perception, the object needs to be identified and tracked within the visual field. Its kinematic structure needs to be determined to calculate its motion, all in real-time. For this research, mapping of the active motion of the unaffected limb onto the affected one was used. In order to do so, the subject’s unaffected, healthy hand and forearm was tracked using a Leap Motion sensor, and the resulting tracking data was mapped onto the affected arm and forearm in the virtual environment (see Figure 1) [36, 37].

The ARWED system can be used as a standard desktop system or as a head-mounted virtual reality system using Oculus Rift. The goal was to have fast tracking and rendering so that the virtual limb tracks seamlessly the movement of a real limb. Some challenges to this are: quick rendering of a photorealistic representation of the subject’s affected limb; motion smoothness; shadow casting; calculation or estimation of appropriate translation, orientation and scaling parameters of the virtual limb to smoothly align with the object(s) in the real environment; discontinued rendering of the virtual limb in the presence of occlusion or sporadic loss of tracking data to reduce/eliminate the loss of persistence for the user, among others. Some of these problems are detailed in [36, 37].  


Experiment on the 3D model to assess the performance of ARWED

One key point in the development of the ARWED device is to assess whether a 3D model of a limb can create the desired priming effect in the subject, and if this effect is any different depending on the hand representation and accuracy presented to the user. In order to answer this question, a series of experiments were developed to examine automatic elicitation of hand actions in individuals without any known loss of neurological function. The experiments progressed from static priming conditions (Experiment 1 described below) to dynamic priming conditions (Experiment 2 described below) [36, 37].

Figure 2. The pinch, hand open, hand close and pointing gestures.
Experiment 1 required participants to rapidly identify gestures generated by a human model or generated by a 3D model of the human arm. The gestures were presented as static pictures. Since the ARWED device blocks vision of the hand and arm, participants’ hands and arms were blocked from view when producing the action. Reaction time was measured, based on surface EMG and a key pressing action provided a measure of response priming. The three different actions, a pinch, hand open/hand closed, and pointing, shown in Fig. 6, were presented as test stimuli. The task required participants to correctly identify a specific response over a series of 40 trials in a block. Each block consisted of 80% compatible actions and 20% incompatible actions, with participants only responding to compatible actions by pressing a response key. For example, if the block consists of the hand open gesture, catch trials might be two fingers raised or the thumb up. The results from Experiment 1 showed that the reaction times (both EMG and key press) and percent correct responses were similar for all four gestures for both the 3D model and human model. This implies that the 3D model limb primes the human motor system in a manner consistent with the human model.

Experiment 2 utilized the same hand gestures as Experiment 1 but presented as dynamic hand gestures; i.e., participants saw a hand pinch and then were asked to replicate the action. The same four gestures were used and presented as a video. Reaction time was measured with surface EMG (first dorsal interossei muscle), and movement time was measured from hand kinematics (see Figure 2). Hand kinematics was recorded with a 3D motion capture system, and movement time was based on motion of the?

Figure 3. Experimental set up, allowing the therapist to follow the muscle activation of the sEMG attached to the impaired left arm
Each action was presented in a block of 40 trials with 10% the trials consisting of catch trials. Within each block of trials, the actions were completed with four different movement times. Participants were required to both imitate the action and the rate of the action. The results from Experiment 2 showed that the reaction time and movement time values between the 3D model limb and the human model are similar between the two conditions. This implies that the dynamic motions of the developed 3D model limbs mimic those of a human model.

Experiment to assess the significance of ARWED as a tool to improve upper-limb mobility for post-stroke patients

Figure 4 . The sEMG are located on the subject’s impaired left arm at the shoulder center, biceps, triceps, top and bottom of the palm.
After initial testing outside the virtual reality environment, one post-stroke volunteer was recruited for four training sessions (once a week with a duration of 30-40 min.). The patient was 55 years of age and was diagnosed with a right basal ganglia hemorrhagic stroke more than three years ago, causing residual left-sided hemiparesis without residual cognitive deficits. A series of protocols for the application of the ARWED system and experiment on its significance as a tool to improve the upper-limb mobility were developed. The stroke volunteer was required to wear virtual reality goggles and sEMG sensors were attached to the hand and arm (see Figure 3 and Figure 4).

Figure 5A. Shoulder muscle activation during weeks 2 (blue), 3 (green), and 4 (red).
The participant was asked to start with their unaffected hand in a neutral position, palm down resting on a table and aligned with the body midline. This experimental task extends the experiments described in the previous sub-section, by having the patient use their unaffected upper extremity for reaching, grasping and moving objects, as well as pushing away flying objects while wearing the Oculus Rift goggles. The experimental setup allowed for priming to be examined in terms of reaction time, which was measured with surface EMG. This task allowed for a more detailed analysis of the reaction time to determine if the 3D model presentation of the limb in virtual reality can generate similar priming processes in a more complex movements. Within the Virtual Reality environment, the participant was able to see the reflection of his unaffected limb superimposed onto his affected upper extremity performing the required task. The results from the sEMG in Figure 5A and 5B clearly show muscle activation/peaks in the biceps and shoulder muscles during weeks 3 and 4. 

The post-stroke patient was also tested and evaluated on Box and Blocks and Fugl Meyer Motor Scale (FMMS) standardized tests prior to training and at the end of the four training session. All four tests measure arm motor function changes and have been thoroughly investigated and reported in literature on post-stroke rehabilitation. Overall, no major changes were noticed in the assessment score on the preliminary testing with the four sessions of device training. However, the Box and Blocks test showed a slight improvement in the right upper extremity, so a possible success indicator could be the beneficial transfer of training from the unaffected to affected limb or vice-versa.


Figure 5B. Bicep muscle activation during weeks 2 (blue), 3 (green), and 4 (red).
This paper describes initial testing of the ARWED, which is a virtual reality system for physical rehabilitation of patients with reduced upper extremity mobility resulting from a stroke. The purpose of this system is to increase limb Active Range of Motion (AROM). To do this, the ARWED maps, in real-time, the patient’s unaffected limb to a virtual representation of the affected limb with the intent of tapping into the mirror neuron which will facilitate the initial learning phase.

The experiments with healthy subjects show that people with reduced joint motions can react to computer animations, link those animations onto joint motions, and learn to move successfully with a constraint. The results on the assessment of the developed virtual photorealistic 3D hand models with healthy subjects imply that the dynamic motions of the 3D models mimic those of a human limb model, as well as that the developed 3D models prime the human motor system in a manner consistent with the human model. The pilot tests with a post-stroke patient suggest that the virtual reality mirror therapy could trigger muscle activation in patients that are more than 3 years post-stroke. This can further serve as an evidence that the time needed for recovery from stroke is not limited to one year and that additional practice can improve mobility in both the sub-acute and chronic phases following a stroke. The future work includes increased frequency and duration of training with the device to assess for any changes in the assessment score.


[1] Ramachandran VS, Rogers-Ramachandran D. Synaesthesia in phantom limbs induced with mirrors. Proc Biol Sci. 1996 263(1369):377-386.

[2] Altman E. Selected and summarized: graded motor imagery. J Hand Ther. 2011 24(1):10-13.

[3] Priganc VW, Stralka SW. Graded motor imagery. J Hand Ther. 2011 24(2):164-8 quiz 9.

[4] McCabe CS, Haigh RC, Blake DR, Reports H. Mirror visual feedback for the treatment of complex regional pain syndrome (type 1). Curr Pain Headche R. 2008 12(2):103-107.

[5] Altschuler EL, Hu JJSJoP, Surgery R, Surgery H. Mirror therapy in a patient with a fractured wrist and no active wrist extension. Scand J Plast Reconstr Surg Hand Surg. 2008 42(2):110-111.

[6] McCabe C. Mirror visual feedback therapy. A practical approach. J Hand Ther.2011 24(2):170-179.

[7] McCabe C, Haigh R, Ring E, Halligan P, Wall P, Blake DJR. A controlled pilot study of the utility of mirror visual feedback in the treatment of complex regional pain syndrome (type 1). Rheumatology. 2003 42(1):97-101.

[8] Tichelaar YV, Geertzen JH, Keizer D, Van Wilgen CP. Mirror box therapy added to cognitive behavioural therapy in three chronic complex regional pain syndrome type I patients: a pilot study. Int J Rehabil Res. 2007 30(2):181-188.

[9] Sato K, Fukumori S, Matsusaki T, Maruo T, Ishikawa S, Nishie H, et al. Nonimmersive virtual reality mirror visual feedback therapy and its application for the treatment of complex regional pain syndrome: an open-label pilot study. Pain Med. 2010 11(4):622-629.

[10] Hogan N, Krebs HI. Interactive robots for neuro-rehabilitation. Restor. Neurol. Neurosci. 2004 22(3-5):349-358.

[11] Krakauer JW. Motor learning: its relevance to stroke recovery and neurorehabilitation. Curr Opin Neurol. 2006 19(1):84-90.

[12] Buchanan JJ, Wright DL. Generalization of action knowledge following observational  learning. Acta Psychol. 2011 136(1):167-178.

[13] Maslovat D, Hodges NJ, Krigolson OE, Handy TC. Observational practice benefits are limited to perceptual improvements in the acquisition of a novel coordination skill. Exp Brain Res. 2010 204(1):119-130.

[14] Moon NRHS, Langari R, Buchanan J, et al. How Does the Central Nervous System Govern Motions with Reduced Arm Mobility. ASME/IMECE. Int Mech Eng Congress Expo. 2012.

[15] Sgandurra G, Ferrari A, Cossu G, Guzzetta A, Fogassi L, Cioni G, et al. Randomized trial of observation and execution of upper extremity actions versus action alone in children with unilateral cerebral palsy. Neurorehabil Neural Repair. 2013 27(9):808-815.

[16] Calvo-Merino B, Glaser DE, Grèzes J, Passingham RE, Haggard PJCc. Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb. Cortex. 2004 15(8):1243-1249.

[17] Casile A, Caggiano V, Ferrari PF. The mirror neuron system: a fresh view. Neuroscientist. 2011 17(5):524-538.

[18] Fogassi L, Ferrari PF, Gesierich B, Rozzi S, Chersi F, Rizzolatti G. Parietal lobe: from action organization to intention understanding. Science. 2005 308(5722):662-667.

[19] Press C, Bird G, Flach R, Heyes C. Robotic movement elicits automatic imitation. Brain Res Cogn Brain Res. 2005 25(3):632-640.

[20] Oberman LM, McCleery JP, Ramachandran VS, Pineda JA. EEG evidence for mirror neuron activity during the observation of human and robot actions: Toward an analysis of the human qualities of interactive robots. Neurocomputing. 2007 70(13-15):2194-2203.

[21] Tsai CC, Brass M. Does the human motor system simulate Pinocchio's actions? Coacting with a human hand versus a wooden hand in a dyadic interaction. Psychol Sci. 2007 18(12):1058-1062.

[22] Urgesi C, Moro V, Candidi M, Aglioti SM. Mapping implied body actions in the human motor system. J Neurosci. 2006 26(30):7942-7949.

[23] Buccino G, Solodkin A, Small SL. Functions of the mirror neuron system: implications for neurorehabilitation. Cogn Behav Neurol. 2006 19(1):55-63.

[24] Franceschini S, Gori S, Ruffino M, Pedrolli K, Facoetti A. A causal link between visual spatial attention and reading acquisition. Curr. Biol. 2012 22(9):814-819.

[25] Pomeroy VM, Clark CA, Miller JS, Baron JC, Markus HS, Tallis RC. The potential for utilizing the “mirror neurone system” to enhance recovery of the severely affected upper limb early after stroke: a review and hypothesis. Neurorehabil Neural Repair. 2005 19(1):4-13.

[26] Ertelt D, Small S, Solodkin A, Dettmers C, McNamara A, Binkofski F, Buccino G. Action observation has a positive impact on rehabilitation of motor deficits after stroke. NeuroImage. 2007 36:T164-T173.

[27] Eng K, Siekierka E, Pyk P, Chevrier E, Hauser Y, Cameirao M, Holper L, Hägni K, Zimmerli L, Duff A, Schuster C. Interactive visuo-motor therapy system for stroke rehabilitation. Med Biol Eng Comput. 2007 45(9):901-907.

[28] Pastor I, Hayes HA, Bamberg SJ, editors. A feasibility study of an upper limb rehabilitation system using kinect and computer games. Engineering in Medicine and Biology Society (EMBC), Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE (1286-1289). IEEE; 2012.

[29] Walter M, Ringleb S, Walter R, Crouch J, Van Lunen B, Morrison S. Virtual reality–enhanced partial body weight–supported treadmill training poststroke: feasibility and effectiveness in 6 subjects. Arch Phys Med Rehabil. 2010 91(1):115-122.

[30] Golomb MR, McDonald BC, Warden SJ, Yonkman J, Saykin AJ, Shirley B, et al. In-home virtual reality videogame telerehabilitation in adolescents with hemiplegic cerebral palsy. Arch Phys Med Rehabil. 2010 91(1):1-8.

[31] Lee BH. Clinical usefulness of augmented reality using infrared camera based real-time feedback on gait function in cerebral palsy: a case study. J Phys Ther Sci. J Phys Ther Sci. 2016 28(4):1387-1391.

[32]Kalron A, Fonkatz I, Frid L, Baransi H, Achiron A. The effect of balance training on postural control in people with multiple sclerosis using the CAREN virtual reality system: a pilot randomized controlled trial. J Neuroeng Rehabil. 2016 13(1):13.

[33] Sousa M, Vieira J, Medeiros D, Arsenio A, Jorge J, editors. SleeveAR: Augmented Reality for Rehabilitation using Realtime Feedback. Proceedings of the 21st International Conference on Intelligent User Interfaces(175-185).  ACM; 2016

[34] Agrawal AJ. How Virtual Reality Will Change Physical Therapy Forever., Inc., Accessed Nov 9, 2015.

[35] Beebe JA, Lang CE. Active range of motion predicts upper extremity function 3 months after stroke. Stroke. 2009 40(5):1772-1779.

[36] Robson N, Faller II KJ, Ahir V, Mhawesh M, Langari R. Developing Models for Predicting Physiologically Impaired Arm Reaching Paths. Engineering, Technology IJoM, Health, Biomedical, Bioengineering. 2017 11(4):139-147.

[37] Robson N, Faller II KJ, Ahir V, Deps Miguel Ferreira A R, Buchanan J, Banerjee A. Creating a Virtual Perception for Upper Limb Rehabilitation. Engineering, Technology IJoM, Health, Biomedical, Bioengineering. 2017 11(4):152-157.