RESNA 27th International Annual Confence

Technology & Disability: Research, Design, Practice & Policy

June 18 to June 22, 2004
Orlando, Florida


Psychometric and Administrative Properties of Measures Used in Assistive Technology Device Outcomes Research

James A. Lenker, MS, OTR/L, ATP, Marcia J. Scherer, PhD, Marcus J. Fuhrer, PhD, Jeffrey W. Jutai, PhD, & Frank DeRuyter, PhD

ABSTRACT

This paper summarizes the measurement domains and properties of tools used in 82 assistive technology device outcome studies published between 1980 and 2001. Few authors offered adequate evidence supporting the reliability or validity of the tools used to collect data. Most offered no information regarding the preparation time required for learning the tools, administering them to research participants, or scoring them.

Keywords:

outcomes, measurement, research, psychometrics, domains

BACKGROUND

The properties of reliability, validity, and administrative burden associated with a particular measurement instrument support the credibility of data gathered using the tool [1, 2]. These properties are considered essential components of formal research reporting [3]. Although there have been numerous calls for increasing the quantity and quality of assistive technology (AT) outcomes research [4-6], no one has analyzed the nature of data that the field has been accumulating. This paper summarizes our evaluation of 82 outcome studies, published between 1980 and 2001, addressing AT devices (ATDs).

Research Questions

  1. How adequately have published ATD outcome studies reported the nature of the sample populations that were used?
  2. How adequately have published ATD outcome studies provided evidence of reliability, validity, and administrative burden for measurement tools used to measure outcome variables?
  3. How frequently have various outcome domains been assessed in published ATD outcome studies?

METHODS

Inclusion Criteria and Sampling

The objective was to identify all research studies published from 1980 to 2001 that involved a follow-up study of ATD users and that measured one or more domains of ATD impact. Minimally, a sub-set of the study participants must have been users of one or more devices in any of the following categories: mobility aids (including crutches, canes, walkers, and wheeled mobility devices), seating and postural support devices for wheelchair users; augmentative and alternative communication, computer-based ATDs, or environmental control systems. Articles excluded from consideration were case studies, ethnographic studies, review articles, meta-analyses, and conference papers. A comprehensive literature search, restricted to English language articles involving AT device outcomes research, was conducted using several bibliographic databases (CINAHL, Medline, EMBASE, ERIC, and PsychInfo).

Coding Syllabus

The coding syllabus was adapted from the format used by Dijkers and colleagues for evaluating the medical rehabilitation literature [2]. Outcome variables (i.e., dependent variables) and their corresponding measurement tools were identified from the Methods sections of the articles.

Procedure

Initially, five papers were coded in parallel by the first three authors in order to evaluate the coding instructions, uncover limitations in usability of the coding form, and refine the coding categories. Five additional papers were coded by the same three individuals in order to reach consensus regarding interpretation and coding of idiosyncratic scenarios. The first author coded the remaining papers.

RESULTS

The inclusion criteria were met by 82 articles involving 212 outcome variables. The majority of studies (56%) were conducted in the United States, with the rest based in Canada, Europe, and Australia.

Sample Characteristics

Table 1. Number of articles in which each AT device category was the predominant AT used by the sample population.

AT Device

# of articles (%)

Environmental control units

10 (12%)

Computer access

10 (12%)

Augmentative communication

8 (10%)

Self-care

8 (10%)

Seat cushion

5 (6%)

Manual wheelchair

4 (5%)

Mobility aid, non-wheeled

4 (5%)

Power wheelchair

2 (2%)

Unable to determine

31 (38%)

TOTAL

82 (100%)

The study samples in each article were evaluated in terms of disabling condition, type of ATD used, and age group. Nine categories of disabling condition were used to classify the studies' participants. The largest category by far was neurologic impairments. The dominant impairment category could not be determined for 30% of the articles due to imprecise descriptions of the proportions of disabling conditions present in sample populations. Nine categories were used to categorize the ATDs that were studied. Table 1 summarizes the number of articles for which various ATDs were the dominant device category. The dominant device category could not be determined for 38% of the studies because insufficient information was presented to make that determination. Three categories were used to record the mix of age groups present in each study: children (birth to age 17), adults (ages 18 to 65), and elders (greater than age 65). For 21 studies (26%), the modal age category could not be determined because insufficient information was presented. In summary, very few (~10%) studies included participants whose disability and age fell into a single category and who used a single category of ATD. Almost one-third (25 of 82) of the studies used a sample that included multiple categories of disabling condition and age who used multiple categories of ATDs.

ATD Outcome Domains and Measurement Tools

Table 2. Outcome domains represented in ATD research
Domain # of outcome variables (%)

Usability

71 (34%)

Use

49 (23%)

User Satisfaction

24 (11%)

Functional Level

24 (11%)

Quality of Life

17 (8%)

Role Participation

16 (8%)

Cost

11 (5%)

Total

212 (100%)

The 212 outcome variables were categorized into one of seven different domains, with ATD usability, usage, user satisfaction, and functional level comprising approximately 80% of the variables identified (Table 2). Of 212 reported outcome variables, 168 (79%) were measured using study-specific instruments, i.e., tools developed to serve the purposes of the particular study. Forty-four (21%) variables were measured using previously published measurement tools.

Data Source and Level

One hundred forty-nine (70%) outcome variables were obtained via participant self-report, and 28 (13%) were based on clinician observation and rating. Almost two-thirds of the variables were measured with either nominal or ordinal data. Approximately 12% of the variables were not defined well enough to identify the data level.

Psychometric Evidence and Administrative Burden

Few authors offered adequate evidence supporting the reliability or validity of the tools used to collect data (Table 3). Most offered no information regarding the preparation time required for learning the tools, administering them to research participants, or scoring them.

Table 3. Psychometric and administrative evidence presented for tools used in ATD research studies

Type of Psychometric or Administrative Evidence

  No report

  Some report, questionable or incomplete applicability

Satisfactory presentation of evidence & applicability

Test-retest reliability

195 (92%)

10 (5%)

7 (3%)

Inter-rater reliability

187 (88%)

11 (5%)

14 (7%)

Content validity

191 (90%)

7 (3%)

14 (7%)

Criterion validity

196 (93%)

6 (3%)

10 (5%)

Construct validity

196 (92%)

7 (3%)

9 (4%)

Training required to learn tool

207 (98%)

1 (<1%)

4 (2%)

Time to administer the tool

181 (85%)

17 (8%)

14 (7%)

Time to score the tool

211 (99%)

1 (<1%)

0 (0%)

DISCUSSION

The studies' sample populations exhibited substantial heterogeneity, characterized by merging a number of distinguishable subgroups (e.g., involving age, disabling condition, or type of ATD used) and not differentiating the findings in terms of those groups. Sample heterogeneity is desirable in studies hypothesizing that an intervention is robust across a range of subgroups. However, it poses hazards for exploratory studies that lack such hypotheses, because it allows the results to be confounded by factors that have neither been identified nor controlled. The effect is to muddle interpretation of the

findings and limit their contribution to a cohesive body of evidence-based literature [7].

A preponderance of the studies' measurement tools had not been published independently. In many cases the immaturity of the AT outcomes research area has forced researchers to develop their own measures. Nonetheless, serious questions arise from using psychometrically unproven instruments. Genuine treatment effects may go undetected if the reliability of measures is actually weak. Tools lacking validity can result in systematic under-estimation, over-estimation, or misrepresentation of treatment effects. Neglect in reporting psychometric background data is not unique to the AT field. Dijkers et al. found similar results in their evaluation of medical rehabilitation research literature [2].

CONCLUSIONS

Based on our findings, we offer several recommendations in order to elevate the quality of ATD outcomes research reporting:

  1. Sample populations should be described as clearly as possible in terms of age, impairment, ATD being used, and the length of time that participants have been using their ATDs.
  2. The rationale for selecting each measurement instrument should be described. Authors should provide information about the psychometric properties of each measurement tool, as well as the applicability of the psychometric data to the context of the current study. If psychometric information is nonexistent, authors should attach caveats to their findings.
  3. If an outcome measure is developed anew for a particular investigation, study resources should be devoted to an initial assessment of the measure's reliability and validity.
  4. Information should be provided about the administrative workload associated with using particular outcome instruments.

REFERENCES

  1. Law, M. (1987). Measurement in occupational therapy: Scientific criteria for evaluation. Canadian Journal of Occupational Therapy, 54, 133-138.
  2. Dijkers, M., Kropp, G. C., Esper, R. M., Yavuzer, G., Cullen, N., & Bakdalieh, Y. (2002). Reporting on reliability and validity of outcome measures in medical rehabilitation research. Disability & Rehabilitation, 24 (16), 819-827.
  3. Johnston, M. V., Keith, R. A., & Hinderer, S. R. (1992). Measurement standards for interdisciplinary medical rehabilitation. Archives of Physical Medicine and Rehabilitation, 73 (Supplement), 3-23.
  4. DeRuyter, F. (1997). The importance of outcome measures for assistive technology service delivery systems. Technology and Disability, 6 , 89-104.
  5. Fuhrer, M. J. (2001). Assistive technology outcomes research: Challenges met and yet unmet. American Journal of Physical Medicine and Rehabilitation, 80 , 528-535.
  6. Smith, R. O. (1996). Measuring the outcomes of assistive technology: Challenge and innovation. Assistive Technology, 8 , 71-81.
  7. Jongbloed, L. (1990). Problems of methodological heterogeneity in studies predicting disability after stroke. Stroke, 21 (Suppl II), II-32-II-34.

Acknowledgments

Funding was provided in part by grant H133A010401 from the National Institute of Disability and Rehabilitation Research. The first author extends heartfelt appreciation to Crystal Yalch for gathering most of the articles reviewed here.

Contact :

Jim Lenker,
Department of Rehabilitation Science,
School of Public Health and Health Professions,
University at Buffalo, 14214-3079.
(716) 829-3141, x109;
lenker@buffalo.edu

RESNA Conference Logo