RESNA Annual Conference - 2019

Kinetic Classifications for Assessment of Typical and Atypical Infant Play Behavior with Toys

C. Kather1, K. Lima1,2, W. Torres1,2, O. Sanders, PhD6, S. Pierce, PhD6,7, L. Prosser, PhD6, M.J. Johnson, PhD1-5

1Rehabilitation Robotics Lab, University of Pennsylvania, 2Department of Physical Medicine and Rehabilitation, University of Pennsylvania, 3Department of Mechanical Engineering and Applied Sciences, University of Pennsylvania, 4Department of Bioengineering, University of Pennsylvania, 5Integrated Product Design, University of Pennsylvania, 6Division of Rehabilitation Medicine, Children’s Hospital of Philadelphia, 7Institute for Physical Therapy Education, Widener University, 8Department of Pediatrics, University of Pennsylvania


Defined as birth at less than 37 weeks’ gestation, premature infants are at a significantly increased risk for behavioral, developmental, and motor delay and disability [1]. Premature birth rates are rising annually worldwide and affect nearly one in ten infants; the greatest prevalence of such occurring in developing nations often with limited access to reliable medical care [1]. When studied at the preschool age, severe disability is found to be common in children of extremely preterm birth, and children of a moderate and early preterm birth were shown to exhibit significant delays with fine motor, communication, and interpersonal development when compared to their full-term counterparts [2]. Early detection and intervention for infants of premature birth has proven to be clinically promising and ultimately beneficial in mitigation and eventual control over these risks [3].

Between the ages of birth to 4 months, typically developing infants should begin exhibiting the ability to move/rotate their head, focus their eyes, wrap their fingers around an object, smoothly move their legs, and reach for toys [4]. Between the ages of 4 to 8 months, they should begin to focus on an object and reach with volition [4]. Clinical guidelines for the evaluation of infant movement (such as Prechtl’s Method on the Qualitative Assessment of General Movements in Preterm, Term, and Young Infants) [5], rely heavily on the careful eye of a trained professional and are thus largely inaccessible to those lacking access to such. Research has shown that it may be useful to analyze the development of infants in non-structured environments in order to detect early signs of atypical development and to establish standards to measure the development of infants at risk of neurodevelopmental disorders [6]. Ongoing works affirm the possibility that instrumented play environments, such as those containing sensor-based smart toys, could be valuable for detecting and monitoring infant neurodevelopment [6].


This study aims to employ our developed smart play system to evaluate the efficacy of using a developed set of simple kinetic classifications (based on clinical movement definitions) to distinguish atypical behavior in a sample of infants, both preterm and full-term, aged 6 months or less.


Picture of the Play and Neuro Development Assessment (PANDA) Gym. The PANDA gym consists of a sensorized mat, and 3D printed structure that holds a set of three sensorized smart toys.The set of three sensorized smart toys: an orangutan to text bimanual dexterity, a lion to test lower body interaction, and an elephant to test upper body interaction. They are designed to hang down from the top of the PANDA gym.
Figure 1. (a) PANDA Gym full set-up, (b) sensorized toys.
The Play and Neuro Development Assessment (PANDA) Gym (Figure 1) is a tool capable of encouraging and measuring wholistic infant interactions with their environment using a pressurized mat, sensorized smart toys, and GoPro cameras positioned to give top and side views [7-9]. The PANDA Gym has a set of three sensorized, interchangeable smart toys—a lion, elephant, and orangutan—utilized for testing lower body interactions, upper body interactions, and bimanual dexterity, respectively. In an initial pilot study, a selection of 34 infants were individually placed into the PANDA Gym to interact with all three toys, and for the purposes of this study, their upper body (elephant) and lower body (lion) videos were chosen for further analysis. This set of 34 infants consists of subjects of various birth statuses (both preterm and full-term), ages (chronological and corrected), and clinical health statuses (based on evaluation from expert researchers in the field of pediatric movement and development, Bailey Infant Neurodevelopmental Screener (BINS) calculations, and further communication with the infants’ parents); full statistics represented in Table 1. Full-term infants exhibiting typical development (referred to here as “typical infants”) were expected to engage with the toys more overall than the preterm infants exhibiting atypical development (referred to here as “atypical infants”), who were expected to engage less overall than their typically developing, full-term counterparts.


Table 1. Infant Subject Data. (*corrected age used; †from initial case study)

Infant #
Age (months) Development and Clinical Notes
6† 5.00 Typical. Full-term.
8† 4.50 Typical. Full-term.
9 5.25 Typical. Full-term.
11 4.00 Typical. Full-term.
15 2.50* Atypical. Preterm.
16 5.50* Atypical. Preterm.
17 3.75* Atypical. Preterm. Normal muscle tone and mass. Later developed speech delay.
18 4.50 Typical. Full-term.
20 4.50 Typical. Full-term.
24 4.25 Typical. Full-term.
29† 5.00* Atypical. Preterm. Global hypotonia and developmental delay present on medical record.
30† 5.25* Atypical. Preterm. Showed signs of delay in early infancy but ultimately confirmed as typical in development milestones.
32 1.00* Atypical. Preterm. Normal muscle tone and gross motor skills when later evaluated at 11 months corrected age.
33 1.50* Atypical. Preterm. Mildly increased muscle tone but delayed gross motor skills when later evaluated at 14 months corrected age.
34 5.50* Atypical. Preterm. Global hypotonia and delayed gross motor skills when later evaluated at 9 months corrected age. Possible genetic condition.

A set of kinetic classifications (Table 2)—which were developed and evolved from an initial group of rudimentary movement definitions based  on visual observation and clinical literature review of infant movement [7]—was used as an objective, intuitive, and ideally effective alternative metric in distinguishing atypical from typical infant behavior, both preterm and full-term. These classifications are largely distinguished as either being voluntary or involuntary and further categorized according to body part and type of interaction. Photographic representations of the interactions are represented in Figure 2.

Table 2. Kinetic Classifications. (*involuntary interaction; †voluntary interaction)


Interaction Description


Unintentional; toy does not appear to be the intended target of contact. Interaction usually without visual engagement.


Direct eye contact and attention directed at the toy; determined by pupil direction and potential head angle.


Toy touches lips or enters mouth of infant.

Hand Touch†

Physical contact with toy but fingers and/or palm do not close around any part.

Hand Grasp†

Physical contact with toy but fingers and/or palm do close around any part.

Foot Touch†

Contact of foot with toy. Usually more prolonged than a kick.

Foot Kick†

Contact of foot with toy involving greater force than touch. Usually shorter term than a touch.

This paper follows up on a promising small case study in which 4 infants (n=2 preterm, atypical; n=2 full-term, typical) were analyzed according to this set of classifications with clear patterns that supported the hypothesis emerging [10]. Subjects from each group had two-minute video segments from their interaction sessions with the elephant and lion toys subjected to video coding with MAXQDA software by two individual researchers, independently. The raw data of the coding process was exported from the software and organized by type and time duration. Inter-rater reliability was examined through using an intraclass correlation (ICC) two-way mixed-effects model in SPSS; producing coefficient values of 0.978 for the elephant toy, and 0.990 for the lion toy. Such values fall well within what is considered to be “excellent reliability” (0.750-1.00) per literature review [11] and the codes were considered sufficiently objective for the purposes of proceeding onto the next trial group.

For sake of overall evaluation and effectiveness, the coded interactions were also compared side by side

for overall performance and reliability. Agreed upon coded interactions were considered effective metrics; and disagreed upon coded segments were

conversationally addressed between researchers to identify potential issues, with the results of which considered for future analysis if such an issue were to arise again (ultimately, finalized definitions are represented in Table 1 with slight grammatical modifications to reduce subjectivity).


Photographic example of infant voluntary interaction with upper body toy. Infant is visually engaged with toy while grasping left hand around trunk and touching left ear with right hand.
Figure 2 (b): Photographic example of involuntary interaction with lower body toy. Infant is not visually engaged with toy, so the slight foot touch is considered involuntary.
Figure 2. (a) clear example of voluntary interactions; (b) clear example of involuntary interaction.
Due to the ICC analysis from the initial n=4 case study calculating within the range of “extreme reliability,” the kinetic classifications were considered to be performing at an adequate/objective level strong enough to merit further testing. For this follow-up case series study, an additional selection of infants aged 6 months or less was considered (n=6 preterm, atypical; n=5 full-term, atypical). Similar to the initial case study, the trials selected for examination were two-minute video clips from each of the current infants’ session with the elephant toy and lion toy. The videos were coded using the same software as previously used, with each of the code sets analyzed no differently. Since the infants for this study were aggregated solely according to the bases of their age, the health statuses of the current sample were unknown to the researcher prior to coding and data analysis. Following completion of the coded videos, the data were gathered and analyzed at once. The findings for the upper body and lower body interactions are represented in Figure 3 and Figure 4, respectively. These  charts include all total infants coded thus far from both the initial and current case studies (n=7 typical, full-term; n=8 atypical, preterm).


Graphical representation of average of all infant interactions with standard deviation measures (initial case study of 4 and current case study of 11, giving a total of 15 total infants) for the elephant toy. Typical infants far outperformed atypical infants for voluntary interactions, and atypical infants slightly outperformed typical infants for involuntary interactions.
Figure 3. Mean total times (s) for voluntary and involuntary interactions for typical/atypical infants with elephant toy.
Figure 3 shows the mean total interaction values (in seconds) for atypical and typical infants with the elephant toy. Typical infants outperformed atypical infants in voluntary interactions with the elephant to a ratio of nearly 4.5:1 (188.42s to 40.41s). Atypical subjects, however, moderately outperformed the typical subjects with respect to involuntary interactions with the elephant toy (14.00s to 2.56s).

Figure 4 shows the mean total interaction values (in seconds) for both atypical and typical subjects with the lion toy. Typical subjects far outperformed the atypical subjects to a ratio of nearly 40:1 (72.03s vs 1.80s). Different from the elephant toy, however, typical infants again outperformed the atypical infants with respect to involuntary interaction (8.96s to 0.70s).

Table 3 represents the total aggregate values for overall interaction and total mean aggregate values per code type across all video trials. Values here present similar findings as in Figures 3 and 4, with overall voluntary mean total interaction times per video in a ratio of 6.2:1 (260.46s vs 42.21s) typical to atypical, and overall involuntary mean total interaction times per video in a ratio of 1:1.3 (14.70s vs 11.51s) typical to atypical.


Graphical representation of average of all infant interactions with standard deviation measures (initial case study of 4 and current case study of 11, giving a total of 15 total infants) for the lion toy. Typical infants outperformed atypical infants for both voluntary and involuntary interactions.
Figure 4. Mean total times (s) for voluntary and involuntary interactions for typical/atypical infants with lion toy.

Visual trends are clearly apparent when analyzing both Figures 3 and 4. For each toy, typical full-term infant subjects far exceed the atypical preterm subject interactions in total duration of interaction with respect to voluntary interactions, with the aggregate voluntary mean ratio of 6.2:1 (typical:atypical) in Table 3 further supporting this assertion. For involuntary interactions, atypical infants outperform typical infants in mean trials with the elephant toy, but the typical infants outperform the atypical infants with the lion toy; this discordant data is somewhat mediated through the aggregate mean involuntary ratio of only 1:1.3 (typical:atypical) as represented in Table 3.

Though not individually represented, clear outliers existed within both the typical and atypical subject populations. Most notably, infants 30 and 34 far outperformed their

Table 3. Overall Interaction Time Totals and Metrics. Former and current case study.
Infant Group Voluntary Interactions (s) Involuntary Interactions (s)
Total μ σ Total μ σ
Typical (n=7) 1,823.20 260.46 118.03 80.60 11.51 8.34
Atypical (n=8) 337.70 42.21 51.40 117.60 14.7 16.47

atypical counterparts, a scenario to which Table 1 provides clarification. Infant 30, through clinical notes, appeared to be exhibiting signs of atypical development at time of testing, but later proved to meet typical developmental milestones. Infant 34, though no clarification is made through clinical notes, is one of the oldest infants tested, and could have been performing at a higher level than what was initially expected.

The lion toy underperformed when compared to the elephant toy across the board. Younger infants will typically display more kicking behaviors than their older counterparts. This could explain the differences in play with the elephant toy versus the lion toy.

It’s worthy to note that for individual trials, typical subjects had six sessions with total interactive times greater than 2 minutes, indicative of simultaneous coded interactions; the atypical selection only had one such occurrence (outlier 30). Similarly, the atypical group had four sessions of no coded interaction at all with the toy which never occurred in the typical sample.


Typical infants on average engage with their surroundings more than their atypical counterparts, and to a greater degree of simultaneity. This is especially true for voluntary interactions, and for upper body engagement. Atypical infants, although were shown to outperform typical infants marginally in some trials, overall don’t engage as frequently with their surroundings as typical infants. Any outlier scenarios were clear and didn’t drastically affect overall coded interaction data, but further analysis of all infants is necessary to confirm.


[1] Preterm birth. (2018, February). Retrieved from

[2] Wood, N. (2001). Neurologic and developmental disability after extremely preterm birth. ACOG Clinical Review, 6(1), 4-5. doi:10.1016/s1085-6862(01)80007-8

[3]Majnemer, A. (1998). Benefits of early intervention for children with developmental disabilities. Seminars in Pediatric Neurology, 5(1), 62-69. doi:10.1016/s1071-9091(98)80020-x

[4] CDC's Developmental Milestones. (n.d.). Retrieved from /index.html

[5] Einspieler, C., & Prechtl, H. F. (2008). Prechtl's Method on the Qualitative Assessment of General Movements in Preterm, Term and Young Infants. Mac Keith Press.

[6] Cecchi, F., Sgandurra, G., Mihelj, M., Mici, L., Zhang, J., Munih, M., . . . Dario, P. (2016). CareToy: An Intelligent Baby Gym: Home-Based Intervention for Infants at Risk for Neurodevelopmental Disorders. IEEE Robotics & Automation Magazine, 23(4), 63-72. doi:10.1109/mra.2015.2506058


[7] Torres, W., Ho, E., Lysenko, S., Prosser, L., Johnson, M.J. (2018, July) The Impact Of Toy Design On Natural Play Interactions Of Premature Infants. Paper presented at the RESNA 2018 Annual Conference, Washington, DC. Retrieved from default/files/conference/2018/emerging_technology/Torres.html

[8] Goyal, V., Torres, W., Rai, R., Shofer, F., Bogen, D., Bryant, P., … Johnson, M. J. (2017). Quantifying infant physical interactions using sensorized toys in a natural play environment. In 2017 International Conference on Rehabilitation Robotics (ICORR). IEEE.


[9] Shivakumar, S. S., Loeb, H., Bogen, D. K., Shofer, F., Bryant, P., Prosser, L., & Johnson, M. J. (2017). Stereo 3D tracking of infants in natural play conditions. In 2017 International Conference on Rehabilitation Robotics (ICORR). IEEE.


[10] Kather, C., Torres, W., Lima, K., Ho, E., Pierce, S., Prosser, L., Johnson, M.J. (2018, November) Distinguishing Atypical Infant Behavior Using Elementary Kinetic Classifications. Paper Presented at the 2018 RESKO Technical Conference on Rehabilitation Engineering and Assistive Technology. Seoul, South Korea.

[11] Koo, T. K., & Li, M. Y. (2017). Erratum to “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research” [J Chiropr Med 2016;15(2):155-163]. Journal of Chiropractic Medicine, 16(4), 346. doi:10.1016/j.jcm.2017.10.001


We gratefully acknowledge the support of the National Institutes of Health (NIH)-1-R21-HD084327-01. We thank Vatsala Goyal, Roshan Rai, Samuel Gaardsmoe, Mayumi Mohan, the PANDA Gym team, the Children’s Hospital of Philadelphia, and Megan Johnson for contributions to infant recruitment and testing.