Repeatability And Reproducibility Of A Procedure For Measuring Color Value Contrast On Products, Including Fitness Equipment

Stephanie D. Schnorbus, Ph.D., Seanna L. Hurley, MS,  and Peter W. Axelson, MSME

Beneficial Designs, Inc., Minden, NV


To help increase the accessibility of fitness equipment for people with visual impairments (VI) or color blindness, a color value contrast method was developed for an ASTM standard focused on the universal design of fitness equipment. This method is used to determine whether labeling, console buttons, movable components, and other key elements of the fitness equipment are visually discernible for VI access. A repeatability and reproducibility study was completed with eleven laboratories to determine the precision limits of this test method. The results indicate that the method can assist manufacturers of fitness equipment to quickly and inexpensively assess and improve the color value contrast on their equipment. This procedure can also be adapted for use by public and private entities seeking to comply with ADA guidelines in the public right-of-way and by private companies desiring to improve the visual accessibility of their products.


In the U.S. alone, the number of people with visual impairment (VI) increased 27% from 2000 to 2010, from 3.3 million to 4.2 million (NEI, n.d.; NEI, 2015). Approximately 8.5 million additional people have some form of color blindness (CBS News, 2014; U.S. Census Bureau, 2013). The number of people and proportion of the population with VI is predicted to continue increasing.

People with VI and color blindness can best discern differences in color based on color value. Color has three properties: hue, saturation, and value. Hue is commonly called color. Saturation (or intensity) is the richness of a hue compared to a gray that reflects the same amount of light. Value (or lightness) is the amount of light reflected back to the viewer’s eye. For people with VI or color blindness different hues can appear as the same color gray. Different values make objects visible, as in black-and-white photography or cinematography. Hence, creating a sign with at least a 70% color value contrast improves visual recognition for people with VI and color blindness.

To increase the accessibility of the built environment for people with VI and color blindness, a prior version of the Americans with Disabilities Act Accessibility Guidelines (ADAAG, 2002) referenced this 70% minimum contrast value for colors on signage and detectable warnings in pedestrian rights-of-way in an appendix. Due to the lack of a simple, low-cost test method, companies and governmental entities were unable to measure how well they met the contrast value guideline in ADAAG. In fact, this specification was removed and is not in the current ADAAG. Improving accessibility by increasing the ability to recognize important information in places such as grocery stores, gyms, and pedestrian rights-of-way would allow for people with VI and color blindness to complete more instrumental activities of daily living, such as shopping and using public transport or public rights-of-way to travel independently.  

A simple, low-cost method to assess color value contrast using common photographic light meters has been under development (Hilderbrand, Hurley, & Axelson, 2010). A quality photographic light meter can be obtained for less than $1000 and does not require extensive specialized training, unlike other technologies, such as spectrophotometers (approx. $5000). For testing, the light meter needs to measure the amount of light that falls on an object (illuminance) and the amount of light an object reflects (luminance). Illuminance is measured in lux or footcandles, and can be thought of as the brightness of the light in the room. Luminance is measured in candela per square meter (cd/m2), and can be thought of as the perceived brightness of an object. The color value contrast test method uses measurements of luminance to quickly and easily assess the color value contrast of two colors.

The ASTM F08.30 Fitness Products Committee assisted in the development of the test method, which is published in the ASTM F3022 Standard Test Method for Evaluating the Universal Design of Fitness Equipment for Inclusive Use by Persons with Functional Limitations and Impairments (ASTM, 2015). Hoogwout, Hilderbrand, Hurley, and Axelson (2013) completed a preliminary repeatability study, showing repeatability in the range of 1.4%–2.5% variation in color value contrast calculations for a single person performing the test method and an inter-operator variation in the range of 1.9%–3.9%. The only variable that changed in that study was the operator—light meter, location, and lighting all remained constant. Additionally, that study was done at light levels common in office areas (450–900 lux), based on the requirements in the draft of ASTM F3022 at that time. A further study was needed to test reproducibility with the revised lighting level in ASTM F3022 (100–400 lux), different light meters (of the same model), different locations, and different lighting. This current study was completed to fulfill that need for testing the reproducibility of the test method in ASTM F3022 for future use by fitness equipment manufacturers in determining and improving the color contrast on their fitness equipment. Test labs have also expressed interest for applications in other areas, such as accessibility assessments of signage and labeling.


The purpose of this repeated measure study was to investigate the following research questions:

  1. What is the repeatability precision (within laboratories) of the color value contrast method (as specified in ASTM F3022)?
  2. What is the reproducibility precision (between laboratories) of the color value contrast method (as specified in ASTM F3022)?



Eleven testers from ten laboratories performed the color value contrast measurement procedure. One tester chose to do testing at only the high light level; the other ten did testing at low light and high light levels.

Instrumentation and Instruction

Testers used a Sekonic L758CINE light meter and a standard 18% gray card and an X-Rite ColorChecker Classic color card. Testers were provided with background information and instruction regarding the technology and general test method, including a YouTube video demonstrating the procedure. The study protocol referred to the ASTM F3022-15 color contrast test methods.

Setup and Procedure

Table 1: Consistency statistics, h and k, 300–400 lux
Oi Set A    Set B    Set C  
  h k h k h k
O1 -0.661 1.058 -1.435 1.219 0.191 0.896
O2 -0.074 1.009 0.973 0.928 -0.632 1.972
O3 0.419 0.000 1.192 0.000 -1.007 0.000
O4 0.419 0.000 1.103 0.611 -0.944 0.000
O5 -0.090 0.000 -0.300 0.000 0.779 0.000
O6 0.558 0.000 -1.425 1.724 2.013 1.345
O7 1.638 0.734 1.053 0.986 -0.164 0.493
O8 -1.926 2.385 -0.768 1.017 -0.894 2.063
O9 -0.105 1.193 0.396 0.761 -0.944 0.000
O10 -1.201 1.101 -0.579 1.277 0.779 0.000
O11 1.021 0.000 -0.211 1.044 0.822 0.000

Before tests were performed, testers were instructed to read through the protocol and practice using the equipment. The Test Coordinator was available for questions regarding the procedure throughout the testing period.

Each tester was requested to perform the protocol at two different light levels—at 100–150 lux and 300–400 lux. For each light level, testers were instructed to find a test environment that could maintain a constant level of light over one square foot of wall at the tester’s eye level. Following ASTM F3022-15, testers checked the light meter’s calibration, ensuring that there were no visible shadows on the gray card.

After affixing the color card to the wall in place of the gray card, testers performed the color value contrast measurement procedure on three different color pairs, testing each color pair three times, resulting in nine complete color value contrast calculations per tester, following the procedure in Hoogwout, et al. (2013). Using those measurements, the testers calculated the color value contrast of the color pair using the following contrast equation from the ADAAG (2002):

Contrast :   [ ( L 1 L 2 ) L 1 ] 100                                (1)


L1 = the luminance of the lighter color

L2 = the luminance of the darker color

The colors in each color pair are as follows: Set A, bluish green and black; Set B, yellow green and blue; Set C, white and yellow. These color pairs match those used in the preliminary repeatability study (Hoogwout, et al., 2013).


Table 2: Consistency statistics, h and k, 100–150 lux
Oi Set A    Set B    Set C  
  h k h k h k
O1 0.528 0.000 -0.165 0.000 0.044 1.701
O2 -0.443 0.000 2.004 1.400 -1.896 0.000
O3 1.014 0.000 0.568 1.894 0.774 1.370
O4 1.430 0.000 0.736 0.000 0.600 0.000
O5 0.609 1.170 -0.671 0.000 1.283 0.000
O6 -0.789 0.000 0.605 0.000 -0.405 0.000
O7 -1.876 1.116 -1.154 1.482 -0.351 1.985
O8 0.239 1.637 -0.077 1.482 -1.303 0.000
O9 -0.917 1.827 -0.583 0.000 0.738 1.087
O10 0.205 1.170 -1.264 0.247 0.517 0.331

After all nine color value contrast tests were completed, the testers were requested to reaffix the gray card to the wall and repeat the illuminance and luminance readings to verify calibration and steady light levels.

Those testers who performed testing at both light levels then adjusted the light levels in the room or found another area with the required light level and performed the setup and test procedure again.

Data Analysis

ASTM E691-11 was used to analyze the data and calculate the repeatability and reproducibility of the color value contrast method (ASTM, 2011). The standard includes all equations and critical values.


For both light levels, the test results from each tester for each color set were statistically analyzed to see whether they were consistent at the 0.5% significance level. The value for h is the between-laboratory consistency statistic, determining whether or not the average of the tester’s results falls within the results that can be expected from different testers 99.5% of the time, using the test method on that color pair. If h is at or higher than the critical value of 2.34 for the high light level (Table 1) or 2.29 for the low light level (Table 2), the tester’s results exceed that 0.5% significance level and are considered a statistical outlier. No statistical outliers for h were found.

The value for k is the within-laboratory consistency statistic, examining the standard deviation of the results from that tester with that set of materials compared to the standard deviation expected between tests by the same tester 99.5% of the time. If k is at or higher than 2.13 for the high light level (Table 1) or 2.11 for the low light level (Table 2), then the result is a statistical outlier. One statistical outlier was found, which is shaded in Table 1 (O8 Set A k 2.385). The k value was not much over the critical value, and after further review of the tester’s adherence to the protocol, it was determined the test method was followed. Therefore, this data set was included in all analysis.

Table 3: Repeatability and reproducibility, 300–400 lux
  Set A Set B Set C
Average contrast (%) 85.49 74.51 34.44
sr (repeatability standard deviation) 0.63 0.99 1.29
r (repeatability limit) 1.76 2.79 3.61
sR (reproducibility standard deviation) 2.22 3.45 4.82
R (reproducibility limit) 6.22 9.65 13.49

The values in Tables 3 and 4 represent the average color value contrast calculated for each color set at that light level. Based on the statistical analysis run according to ASTM E691, there is a 99.5% chance that tests conducted by a single tester will fall within r (repeatability limit), with standard deviation sr, for each color set at that light level. Likewise, there is a 99.5% chance that testing conducted across testers will be within R (reproducibility limit), with standard deviation sR. The repeatability limit for each color set was lower (better) at the low light level; inversely, the reproducibility limit was higher (worse) for each color set at the low light level.


The statistical outlier for Set A at the high light level was in the k value, or the comparative variability in the test results from that tester versus the variability in test results from other individual testers. Upon investigation, it appears the tester was blocking part of the light during the spot meter testing, although no visible shadows were seen. The test method in ASTM F3022 currently requires testers to check for a visible shadow when testing illuminance, but does not emphasize that the tester should ensure the path of light to the testing surface is not blocked during the spot meter testing. Hence, the tester followed the protocol. This suggests a clarifying note should be added to ASTM F3022.

This study was done at lower light levels than the preliminary repeatability study, which was done at light levels common in an office setting, based on a draft version of ASTM F3022 (Hoogwout, et al., 2013). Before the ASTM F3022 standard was published, additional preliminary data collected indicated that the average light levels in gyms was 100–400 lux, in contrast to the average office lighting of 450–900 lux. This study tested color value contrast levels within the range of ASTM F3022—average light levels in gym environments—and provides further insight into how lower levels of light affect color value contrast testing. Even at lower levels of light, this study showed that color value contrast readings taken by a single individual tend to be fairly consistent, but lower levels of light result in lower reproducibility. Measurements taken by different individuals in different settings vary by up to 15.8%, with less variation the higher the color value contrast and the brighter the illuminance. The luminance of lighter colors (e.g., white and yellow, Set C) vary the most based on illuminance levels, which helps explain the larger reproducibility limits for that color set—luminance readings for those colors are very sensitive to slight changes in illuminance.

Table 4: Repeatability and reproducibility, 100–150 lux
  Set A Set B Set C
Average contrast (%) 83.08 72.25 36.66
sr (repeatability standard deviation) 0.49 0.70 1.22
r (repeatability limit) 1.38 1.96 3.42
sR (reproducibility standard deviation) 2.91 4.58 5.66
R (reproducibility limit) 8.15 12.84 15.84

Overall, the repeatability and reproducibility limits found in this study show that the color value contrast method in ASTM F3022 is valid for use on fitness equipment and in other applications to confirm color value contrast for use in labeling and signage. Because of the reproducibility limits, manufacturers should be encouraged to choose color pairs whose color value contrast is close to 80% to ensure meeting the 70% minimum recommended contrast. Doing so will increase visual recognition and hence accessibility for people with VI and color blindness.

It should be noted that a total of seven different Sekonic L758CINE light meters were used by the testers for the high light level, and there were six different light meters used by testers for the low light level. At each light level, there were three sets of two labs that used the same meter due to the constraints of renting the light meters for labs that did not purchase them. Two of the testers were from the same laboratory and were counted as two separate testers for this study.


This study shows that the color value contrast test method allows fitness equipment manufacturers to increase the visibility of labeling, displays, moveable parts, and other key components on their equipment to increase access for people with VI and color blindness. It is expected that the repeatability and reproducibility limits found through this study will also enable other public and private sector entities to adapt this method to assess the color value contrast of current labeling, signage, and visual cues in the built environment and on products.

Future research will need to be conducted regarding the effects of several variables, including limits on which photographic light meters can be used reliably with this method. Currently, this method seems to work best with light meters calibrated at a tungsten setting versus a daylight setting. Additional work should be done to examine the effects of different lighting (especially LEDs) on color value contrast as tested by this method. In addition, field testing should be performed. The effect of a lighted sign in a dark environment, essentially switching the lighting source from the current method, should be studied. People with VI and color blindness have indicated that signs with extreme color value contrast (e.g., black and white) can also be difficult to read. The possible need for a maximum color value contrast should also be examined. Subject testing should be done to confirm that color value contrast pairs that pass the 70% minimum meet the needs for VI access.

The results of this study would support the inclusion of a 70% minimum contrast requirement in the body of the Americans with Disabilities Act Accessibility Guidelines (ADAAG). This would greatly increase accessibility for people with VI and color blindness, as entities required to follow ADAAG would improve the color value contrast of signage and other important visual cues. It is also anticipated that this method will be widely adapted to meet ADAAG in the built environment and for the testing and usability of consumer products.


Americans with Disabilities Act Accessibility Guidelines (ADAAG). (2002). Clause A4.30.4. Retrieved 23 November 2012, from

adaag/html/ adaag.htm#A4.30.5.

ASTM. (2011). Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method (E691-11). West Conshohocken, PA: ASTM International. DOI: 10.1520/E0691-11.

ASTM. (2015). Standard Test Method for Evaluating the Universal Design of Fitness Equipment for Inclusive Use by Persons with Functional Limitations and Impairments (F3022-15). West Conshohocken, PA: ASTM International. DOI: 10.1520/F3022-15.

CBS News. (2014, August 27). Who’s most at risk for color blindness. Retrieved 11 February 2016, from

Hilderbrand, H., Hurley, S.L., Axelson, P.W. (2010). Measuring luminance color contrast on products, including fitness equipment, for people with visual impairments. in Proceedings of the RESNA 2010 Annual Conference. Las Vegas, NV: RESNA Press.

Hoogwout, M.R., Hilderbrand, H.M., Hurley, S.L., Axelson, P.W. (2013). Repeatability of a procedure for measuring color contrast on products, including fitness equipment. in Proceedings of the RESNA 2013 Annual Conference. Atlanta, GA: RESNA Press.

National Eye Institute (NEI). (n.d.). All vision impairment. Retrieved 11 February 2016, from


National Eye Institute (NEI). (2015). Facts about color blindness. Retrieved 11 February 2016, from

U.S. Census Bureau. (2013). 2010 Census of Population and Housing, Summary population and housing characteristics (CPH-1-1, United States). Washington, D.C.: U.S. Government Printing Office.


The contents of this paper were developed under a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number 90RE5009-01-00).  NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS).  The contents of this paper do not necessarily represent the policy of NIDILRR, ACL, HHS, and you should not assume endorsement by the Federal Government.