-->

 

Validation Of The Accessplace Personal Accessibility Information Review Sorting

Dennis B. Tomashek1, Nathan Spaeth1, Nicole Latzig1, Ariana Pelkey1,Roger O. Smith1

1University of Wisconsin-Milwaukee, R2D2 Center

INTRODUCTION

Figure 1 is a screenshot of the AccessPlace reviews page. A dropdown menu is located on the left side of the page. A purple title bar with the word “AccessPlace” is in the center, and a small logo is on the left part of the bar. Below this is a list of reviews with star ratings. The dropdown menu is superimposed on a grayed out map with location markers
Figure 1. AccessPlace reviews page
The AccessPlace web app is designed as an interactive site where people with disabilities can both review and read others’ reviews on the accessibility of public buildings. AccessPlace was created by the R2D2 Center as part of the Access Ratings for Buildings (ARB) project, funded in large part by a grant from NIDIRR, as a multi-platform responsive designed app available to users on all devices (e.g., laptops, tablets, phones) and operating systems [1-3].  AccessPlace is designed to be similar to popular restaurant rating apps [4, 5]. A person can search for buildings nearby or in other cities, can search for specific types of businesses or buildings (e.g., Chinese restaurants, barbershops, campus buildings), and can leave star ratings (1-5) and comments for others to read. Figure 1 shows the review page.

Table 1. A Sample of Health Conditions and Functions in the AccessPlace user profile
Health Conditions Functions RatingScale
Mobility How difficult is climbing stairs? Easy-Unable
Vision How difficult is reading Easy-Unable
What is the severity of glare sensitivity? None-Severe
How difficult is walking? (Vision) Easy-Unable
What is the severity of color blindness? None-Severe
Hearing How difficult is hearing? Easy-Unable
Cognition How difficult is remembering? Easy-Unable
How difficult is following directions? Easy-Unable
How difficult is working with numbers? Easy-Unable
How difficult is navigating? Easy-Unable
Communicating How difficult is speaking? Easy-Unable
How difficult is understanding (spoken) speech? Easy-Unable
How difficult is reading? Easy-Unable
Upper Extremity How difficult is reaching objects off shelves? Easy-Unable
How difficult is lifting items? Easy-Unable
How difficult is it to grasp items? Easy-Unable
What is the severity of tremors experienced? (UE) None-Severe

On most rating apps, a person is forced to scroll through numerous reviews, with no way of being able to distinguish who the reviewer is or what the review is about. Reviews for a restaurant may include everything from the service, to the decor, to the food.  A restaurant review from a 20-something may not be relevant to a retired person, and vice-versa. A unique feature of AccessPlace is the Personal Accessibility Information (PAI) feature, which is based on a personal profile. The AccessPlace profile includes 9 Health Condition categories, with 29 functional impairments which a person can rate from easy or none [default] to unable or severe. Table 1 above shows a sample of the 9 health conditions and functional impairments.

Table 2. Building Elements in AccessPlace

Doorways

Elevators

Floor/Ground

Handrails

Parking

Ramps

Restrooms

Routes

Seating

Signage

Stairs

Restaurant Features

 

Highlighted elements were reviewed for this study

After a PAI profile has been created, AccessPlace will automatically order reviews so that those written by people “like me” rise to the top of the review list. Users are still able to see other reviews, but a person with a hearing problem, for example, does not have to filter through reviews left by those with mobility issues or upper extremity disabilities, but will see reviews left by others with hearing problems first. A major complication to this is that many people have more than 1 disability, and the levels of disability may differ. To address this problem, a modified vector distance formula was used. We did not use the square root, as the units did not matter in this case.

A second feature of AccessPlace that provides greater detail of information is the inclusion of 12 building elements, which can be rated independently. There is also an Overall Accessibility rating, to get a quick, summary view of the accessibility of the building. Table 2 lists the 12 building elements. The highlighted elements are those for which reviews were left for this study.

Methods

Eleven profiles of simulated users were created to test the known-groups validity of the AccessPlace sorting. Simulated participants have been found to have moderate to substantial reliability and low bias [6-8] Each of the 11 users were assigned functional impairment ratings between 1 (easy or none) and 5 (unable or severe) on five functional impairments (See table 3). Profiles were intentionally created to provide a broad selection of similar and dissimilar PAI profiles.

Table 3. Health conditions and functions for simulated users
Health Condition Function
Vision How difficult is reading?
Upper Extremity How difficult is reaching objects off shelves?
Sensory Sensitivity What is the level of sensitivity to stimulating environments?
Mobility How difficult is climbing stairs?
Communicating How difficult is speaking?

Star ratings were entered for 12 different restaurants for each of the 6 building elements highlighted in table 2 using all 11 simulated user PAI profiles. Star ratings varied, in order to simulate actual reviews. For instance, a user who was unable to read may have left a 1 star rating for one restaurant, and a 5 star rating for another. The AccessPlace PAI algorithm does not sort by rating levels, only on the level of ability on the functions in the profile. Table 4 provides examples of two simulated users PAI profiles with a low profile similarity. In this case, Simulated User 10 (SU10) has a functional ability of 1 (easy, none) for all five functional impairments, while Simulated User 11 (SU11) has a functional ability of 5 (unable, severe) for all five functional impairments. We expected that, when viewing reviews as SU10, reviews from SU11 would appear at or near the bottom of the reviews.

Table 4. Example of two simulated user profiles with low profile similarity
Simulated User 10 Simulated User 11
Health Condition Function Functional ability Health Condition Function Functional ability
Vision How difficult is reading? 1 Vision How difficult is reading? 5
Upper Extremity How difficult is reaching objects off shelves? 1 Upper Extremity How difficult is reaching objects off shelves? 5
Sensory Sensitivity What is the level of sensitivity to stimulating environments? 1 Sensory Sensitivity What is the level of sensitivity to stimulating environments? 5
Mobility How difficult is climbing stairs? 1 Mobility How difficult is climbing stairs? 5
Communicating How difficult is speaking? 1 Communicating How difficult is speaking? 5

Table 5 provides examples of two simulated users PAI profiles with a higher profile similarity. Because the two users have only one functional impairment that is dramatically different, we expected that reviews from Simulated User 11 would appear at or near the top of the reviews for Simulated User 9 (SU9).

Table 5. Example of two simulated user profiles with high profile similarity
Simulated User 9 Simulated User 11
Health Condition Function Functional ability Health Condition Function Functional ability
Vision How difficult is reading? 5 Vision How difficult is reading? 5
Upper Extremity How difficult is reaching objects off shelves? 5 Upper Extremity How difficult is reaching objects off shelves? 5
Sensory Sensitivity What is the level of sensitivity to stimulating environments? 1 Sensory Sensitivity What is the level of sensitivity to stimulating environments? 5
Mobility How difficult is climbing stairs? 5 Mobility How difficult is climbing stairs? 5
Communicating How difficult is speaking? 5 Communicating How difficult is speaking? 5

Data Analysis

An Interrater Intraclass Correlation Coefficient (ICC) was used, with reviewers used as the raters, and the restaurants as the ratees. ICC (2) two-way random effects model was used[9]. Data were analyzed in SPSS 22.0.0.0. Additionally, Kruskal-Wallis tests were conducted to see if the expected order was different than the actual order for each restaurant. Alpha was set at .05, and a confidence interval (CI) of .95 was used.

Hypotheses

  1. There would be a good ICC (ICC>.75) between the expected order and the actual order for 4 restaurants.
  2. There will not be a significant difference in rank ordering as determined by a Kruskal-Wallis test between the expected order and the actual order for 4 restaurants.
  3. There would be an excellent ICC (ICC>.90) between the actual order for 4 restaurants.

Results

  1. The ICCs indicated significant reliability for all four comparisons, ranging from .63 to .79.
  2. The Kruskal-Wallis test results were non-significant, indicating no difference between the rankings of the expected vs actual orders for all 4 restaurants.
  3. The ICC for the actual order between the 4 restaurants was significantly reliable at .92* (CI=.87-.96).
    Table 5. Results of ICCs and Kruskal-Wallis test for Expected order vs Actual
    Test Expected Vs R1 Expected Vs R2 Expected Vs R3

    Expected Vs R4
    ICC 0.73* (CI=.44-.87) 0.79* (CI=.56-.90) 0.76* (CI=.49-88)

    0.63* (CI=.21-.82)
    Kruskal-Wallis Χ2=.01 Χ2=.01 Χ2=.01

    Χ2=.01

    * indicates significance ≤ .05

Conclusion

Despite one comparison not fully supporting our hypothesis (Expected vs Actual Restaurant 4), the results were still significant, indicating that the method for ordering reviews works. This was further supported by use of the Kruskal-Wallis test, which indicated no difference in rankings between any of the restaurant and the expected data. Additionally, the ordering between SU9 & SU11 (high similarity) was as expected: SU11 appeared 2nd for SU9, SU9 appeared 3rd for SU11. This was also exhibited between SU10 & SU11 (low similarity): SU10 appeared last for SU11, SU11 appeared last for SU10.

 

Table 6. Expected and Actual order for simulated user 11
Position Expected Actual R1 ActualR2 ActualR3 ActualR4
1 7 7 7 7 7
1 9 5 5 5 5
3 5 9 9 9 9
4 1 1 2 1 1
4 2 2 1 2 2
6 3 8 8 8 8
6 4 6 4 4 4
6 6 4 6 6 6
6 8 3 3 3 10
10 10 10 10 10 3

Further analysis of the data from the vector difference outcomes indicates that some of the discrepancies in order between the expected and actual are due to ties in the data. For instance, for SU11, there was a tie between simulated users 7 & 9 at position 1, a tie between simulated users 1 & 2 at position 4, and a 4-way tie between users 3, 4, 6 & 8 at position 6. This would have an effect on lowering the ICC scores. This is reflected in the data presented below in table 6.

While the ordering of reviews is not the main focus of AccessPlace, it is expected to make for a better user experience with the app.

Limitations and Next steps

It should be noted that these were simulated users, specifically designed to explore the algorithm. We expect actual user data to be more complex. An important next step for this project will be to address this by obtaining real PAI profiles and reviews from actual people with disabilities. Further, the analysis presented in this paper is only a subsection of all the data available. Continued analysis of all 11 profiles across all 12 restaurants will be conducted. This study also only examined the Overall Accessibility ordering, and not for the 8 (out of 12) building elements for which reviews were left for each of the 12 restaurants.

References

[1]        R. O. Smith and J. Schwartz, "The development of Access Ratings for Buildings: Apps for community access," presented at the 3rd Annual Occupational Science Summit, Philadelphia, PA, 2014.

[2]        J. K. Schwartz and R. O. Smith, "Access Ratings for Buildings: Measuring Building Accessibility in the Community Environment " presented at the Second Annual Occupational Therapy Summit of Scholars, Chicago, IL, 2013.

[3]        R. O. Smith, "Access Ratings for Buildings," presented at the First Annual Occupational Therapy Summit of Scholars, St. Louis, MO, 2012.

[4]        N. L. Spaeth, D. B. Tomashek, and R. O. Smith, "AccessPlace: Personalized accessibility information for buildings," in RESNA 38th International Conference on Technology and Disability: Research, Design, Practice and Policy (platform), Denver, CO, 2015.

[5]        K. Edyburn, J. Schwartz, and R. O. Smith, "A case study: Development of Access Ratings for Buildings "Consumer" mobile app," in RESNA 36th International Conference on Technology and Disability: Research, Design, Practice, & Policy, Bellevue, Washington, USA, 2013.

[6]        R. Shavelson, P. W. Mayberry, W. Li, and N. M. Webb, "Generalizability of job performance measurements: Marine Corps rifleman. ," Military Psychology, vol. 2, pp. 129-144, 1990.

[7]        N. V. Vu and H. S. Barrows, "use of standardized patients in clinical assessments: Recent developments and measurement findings," Educational Researcher, vol. 23, pp. 23-30, 1994.

[8]        G. Adamo, "Simulated and standardized patients in OSCEs: achievements and challenges 1992-2003," Medical Teacher, vol. 25, pp. 262-270, 2003.

[9]        T. K. Koo and M. Y. Li, "A guideline of selecting and reporting intraclass correlation coefficients for reliability research," Journal of Chiropractic Medicine, vol. 15, pp. 155-163, 2016.