Home
Global Supplier Directory
APPLIANCE Engineer
Supplier Solutions
APPLIANCE Line
Whitepaper Library
Calendar of Events
Association Locator
Contents Pages
Market Research
Subscription Center


 
issue: January 2007 APPLIANCE Magazine

Quality & Testing
Use of Gage R&R in Developing Metrics for Subjective Data


 Printable format
 Email this Article
 Search

by Aruna Pochampally, senior consumer scientist and Barbara Godden, senior research technician, Whirlpool Corporation

With the proper utilization of Six Sigma tools like Gage R&R studies, the development of metrics for evaluating subjective characteristics becomes possible.

Quantifying subjective data such as load condition after washing is a big challenge in evaluating the performance of laundry appliances. Historically, there are no proper standards developed to evaluate subjective features. However, with the proper utilization of Six Sigma tools like gage repeatability and reproducibility (Gage R&R) studies, the development of metrics for evaluating subjective characteristics becomes possible. The following deals with the development of a metric to evaluate one such type of subjective data, potential damage to the clothes load in a washing machine, utilizing Gage R&R studies.

Background

The Performance Evaluation is one of the major steps in the product development process of all appliances. The evaluation process consists of testing all the product characteristics that are important to a consumer. These characteristics include properties of both objective and subjective nature. Objective properties involve little or no human subjectivity in the test results. In other words, the test results do not vary much from operator to operator. On the other hand, testing of subjective properties requires a lot of human intervention in gathering the test results. The testing of subjective properties yields results that vary to a greater extent from operator to operator. It is not unusual to get different results even if tested by the same operator.
The subjective nature of some properties becomes a major concern in proper usage and interpretation of testing results. However, it is imperative to evaluate subjective properties along with properties of objective nature.  
Technological advances in the industry in some cases help in overcoming the subjectivity involved in evaluating the products. A good example is the comparison of the color of two products. Before the development of color measuring devices like the spectrophotometer, comparison of color was made by human eye. Therefore, it was hard to quantify the data owing to many factors, such as the very subjective nature of testing and differences in the perception ability from person to person. Spectrophotometer quantifies color, taking into consideration all the aspects of color, and it yields repeatable and reproducible results.
Technological developments can in most instances help in quantifying the data and eliminating the subjectivity involved in evaluating the products. However, quantifying some subjective data, like the appearance of the clothes load after laundering, is still a big challenge facing the home appliance industry. It is hard, if not impossible, to develop devices that can quantify results of testing properties of highly subjective nature. So it becomes necessary to create testing methods and standards that can yield more repeatable and reproducible results. Better utilization of Six Sigma tools can help in creating strong measuring metrics with respect to repeatability and reproducibility. 

 

Figure 1. Visual aid for buckle

Procedure

This paper deals with the development of a metric for evaluating the damage done to the clothes load in a washer, utilizing Gage R&R studies. The items that are found in a clothing load that are not made of fabric but attached to the garments include buckles, slides, metal buttons, plastic buttons, and hooks. These items have the potential of being damaged when put in a washer and dryer, and they can also damage the appliances. It is easier to assess the damage done to the laundry appliances by these items compared to the damage done to the items themselves during laundering. There is much subjectivity involved in evaluating the damage done to these items. A pass/fail criteria can be used in accepting/rejecting the damaged items. However, it is very difficult to come to a conclusion if the item passes or fails without proper standard procedures and a strong measurement system for evaluation.
Gage R&R studies, the tool that determines the consistency of test results within and between operators, can be utilized to create a strong measurement system for evaluating such type of characteristics. Gage Repeatability measures the consistency of the results with the same operator, whereas Gage Reproducibility evaluates the consistency of results between operators. This paper focuses on developing a pass/fail metric for evaluating the damage done to the items in a washer and dryer. Hence, attribute Gage R&R was utilized in this study.
 Several iterations of Gage R&R studies were necessary to create a strong metric for evaluating damage done to the items. This paper is focused on the metric for assessing the damage done specifically to metal buckles. This process needed four Gage R&R iterations.
The first step in the process was creating visual aids for evaluating the buckles. Photographs of several buckles served as visual aids. A panel of experts (consisting of design engineers, consumer scientists and research technicians) made decisions on the level of acceptability of damage from a consumer’s perspective.  The photographs were then categorized into acceptable and unacceptable divisions. These photographs were used as visual aids for further evaluations. Figure 1 is an example of a visual aid used.
The next step in the process was creating samples for evaluation. A required number of buckle samples was created simulating possible damage that can occur to buckles during laundering actions.
These samples were then evaluated by operators using the visual aids created by a panel of experts. The data collected from the evaluations were analyzed using Minitab software to determine the consistency of operators. The agreement between operators was found to be only 50 percent, which is considered to be unacceptable for attribute data according to industry standards. Assessment agreement within operators ranged from 78 percent to 90 percent. The Minitab output is as follows:


Minitab Output for Gage R&R First Iteration

Within Appraisers
Assessment Agreement

Appraiser    #Inspected    #Matched    Percent    95 % CI
Operator 1    50               39              78.00        (64.04, 88.47)
Operator 2    50               45              90.00        (78.19, 96.67)

# Matched: Appraiser agrees with him/herself across trials.


Between Appraisers
Assessment Agreement

#Inspected    #Matched    Percent    95 % CI
50                25               50.00       (35.53, 64.47)

# Matched: All appraisers’ assessments agree with each other.


The inconsistencies of scores between and within operators were explored further. Based on the findings, it was concluded that a gage would be needed to evaluate the three-dimensional aspect of bent buckles and also that photographs should be scaled to the actual size of the buckles to help the operator accurately compare the pictures to the actual buckles.
A metallic gage was developed to slide under the buckle in order to evaluate the extent of bending seen in the damaged buckle. The photographs were also scaled to fit the actual size of buckles. The second iteration of Gage R&R study was conducted on the revised visual aids and also utilizing the metallic gage to slide under the buckle. The assessment agreement between the operators increased to 68 percent from 50 percent in the first iteration and the assessment agreement within operators ranged from 82 percent to 94 percent as opposed to 78 percent to 90 percent in the first iteration. See the Minitab analysis output below.


Minitab Output for Gage R&R Second Iteration

Within Appraisers
Assessment Agreement

Appraiser    #Inspected    #Matched    Percent    95 % CI
Operator 1    50               47              94.00        (83.45, 98.75)
Operator 2    50               41              82.00        (68.56, 91.42)
# Matched: Appraiser agrees with him/herself across trials.


Between Appraisers
Assessment Agreement

#Inspected    #Matched    Percent    95 % CI
50                34               68.00       (53.30, 80.48)
# Matched: All appraisers’ assessments agree with each other.

Although the assessment agreement improved substantially from first to second iterations, it was not sufficient to be considered a strong metric. Hence, the scores were studied to explore the inconsistencies in assessments. The operators indicated a necessity for written instructions in addition to visual aids and a metallic gage.
A written criterion for evaluating buckles was created, and the pictures were refined further to make them simple and easy to follow. The third iteration of Gage R&R study followed the development of written criterion and refinement of visual aids. The assessment agreement between operators increased to 70 percent, and the assessment agreement within operator narrowed down to 86 percent - 90 percent. See the Minitab analysis output below.


Minitab Output for Gage R&R Third Iteration

Within Appraisers
Assessment Agreement

Appraiser    #Inspected     #Matched     Percent    95 % CI
Operator1    50                 45                90.00       (78.19, 96.67)
Operator2    50                 43                86.00       (73.26, 94.18)

# Matched: Appraiser agrees with him/herself across trials.


Between Appraisers
Assessment Agreement

#Inspected    #Matched    Percent    95 % CI
50                35               70.00        (55.39, 82.14)

# Matched: All appraisers’ assessments agree with each other.


Further exploration of inconsistent scores indicated that the non-agreement of assessments was due to the fact that the operators were not utilizing the metallic gage in a similar way. So detailed instructions for using the metallic gage were necessary.
Instructions were now inserted in the written evaluation criterion and also in the visual aids.
Figure 2 indicates the method to hold the metallic gage while evaluating the buckles. Figure 3 is an example of the photograph used in the visual aids depicting the buckle with all its parts labeled.
The final iteration of Gage R&R was done using the improved written evaluation criterion and visual aids. The Gage R&R study results indicated an assessment agreement between operators of 84 percent and the assessment agreement within operators ranged from 90 percent to 98 percent. See the Minitab analysis output for the fourth iteration on the next page. 

Minitab Output for Gage R&R Fourth Iteration

Within Appraisers
Assessment Agreement

Appraiser    #Inspected    #Matched    Percent    95 % CI
Operator 1    50              49                98.00       (89.35, 99.95)
Operator 2    50              45                90.00       (78.19, 96.67)

# Matched: Appraiser agrees with him/herself across trials.

Between Appraisers
Assessment Agreement

#Inspected    #Matched    Percent    95 % CI
50                 42              84.00       (70.89, 92.83)

# Matched: All appraisers’ assessments agree with each other.


Results

After conducting four iterations of Gage R&R studies for the development of damage metric, we can see that the four areas that need to be focused to create a robust metric for subjective data are written instructions, visual aids, training, and standard usage of equipment. These factors will be discussed in greater depth in the following sections. 

 

Figure 2. Metallic gage under a buckle

Written Instructions

Figure 3. Parts of a buckle

During the first and second iterations of Gage R&R only verbal instructions regarding the acceptability/unacceptability of damage done to the buckles were given to the evaluators. A written document for instructions was created before the third iteration of the study and this improved the assessment agreement. So, it can be understood that a standard operating procedure is absolutely necessary for obtaining consistent results between and within evaluators.

Visual Aids

Photographs of the buckles were used as visual aids in the evaluation procedure. In the first iteration of the Gage R&R study, it was not possible for the operators to place the buckles on the photographs to compare and determine if the damage was acceptable or not. Before the second iteration, the photographs were scaled to fit the size of the buckles so that it became easy for the evaluators to compare the buckles with the photographs. The resizing of photographs improved the assessment agreement significantly. It can be noted that resizing the photographs may not be possible in all instances, especially if the items under consideration are too large or too small to fit a sheet of paper. However, care must be taken to fit them to appropriate dimensions so that the comparison with the real objects becomes possible.  

Training

During the first iteration of the Gage R&R study, instructions on evaluating the damage done to the buckles was given verbally but formal training was not given to the evaluators. So the assessment agreement was very poor. From the next iteration, inconsistencies of the scores were discussed with the evaluators and in each step, the reasons for inconsistent scores were explored, and aspects that needed clarification were given more focus in the next training sessions. All evaluators were trained at the same time, rather than doing individual training sessions, to make sure all the evaluators got the same set of instructions for scoring the buckles.

Standard Usage of Equipment

To evaluate the extent of bending that occurred in the buckles due to wash action, a metallic gage was devised after the first iteration of the Gage R&R study. The metallic gage helped to improve the assessment agreement significantly. At the end of the third iteration of Gage R&R study, it was noticed that the evaluators were not holding the gage in a consistent manner. So a consistent method for holding the gage was incorporated in visual aids and the written instructions document. This further enhanced the strength of the metric. It can thus be reiterated that standard operating procedure for equipment is absolutely necessary for developing a robust evaluation metric.

Conclusions

The development process of metric for evaluating the damage done to the buckles in laundry shows that Gage R&R is a powerful tool that can be used to create standards to evaluate subjective characteristics of the appliances by converting them to objective characteristics. Several iterations of Gage R&R are needed to create a robust metric. The results from each iteration of the study bring to light the reasons for inconsistent scores between and within operators. This helps in improving the consistency of the metric. The work discussed in this paper highlights the necessity for clear visual aids, written instructions, training, and standard usage of equipment. Clear written instructions are very much essential to achieve consistency of scores between and within operators as can be seen from the degree of improvement in the assessment agreement achieved by incorporating written instructions in this work. This study also emphasizes the importance of the Gage R&R tool in creating standard metric for evaluating factors that are of a very subjective nature.

Acknowledgements

The author would like to thank Curt Tremel, manager, R&D Performance Lab, Maytag Laundry Appliances, for his time in reviewing the paper and giving valuable suggestions. The author would also like to thank Sarath Jayatellika, R&D reliability engineer, Maytag Appliances, for his support and encouragement in writing this paper.

References

1. Implementing Six Sigma, Smarter Solutions Using Statistical Methods,  Second Edition, Forest W. Breyfogle, page 307.

This paper is an edited version of a paper presented at the 57th Annual International Appliance Technical Conference, held in March 2006.

About the Authors

Aruna Pochampally is a senior consumer scientist for Whirlpool Corporation. Previously, she was a Textile Scientist with Maytag. She has a Masters of Science in Textile Science from the University of Nebraska, a BTech in Textile Technology from Osmania University, Hyderabad, India, and is currently pursuing her MBA from the University of Iowa.
Barbara Godden is a senior research technician at Whirlpool Corporation. She holds an Associate of Arts degree from Des Moines Area Community College and is currently pursuing a Bachelor of Arts from Buena Vista University.

If you would like to contact Pochampally or Godden, e-mail editor@appliance.com

 

Daily News

...........................................................

Nov 26, 2014: Hamilton Beach 3Q income down slightly

Nov 26, 2014: LG's sales of Inverter Linear Compressor refrigerators top 10 million

Nov 25, 2014: Robotics Exhibits Grow 25% at 2015 CES

Nov 25, 2014: Index shows HVAC/R contractors more positive than last year

Nov 25, 2014: Consumer Confidence Index Declines

More Daily News>>

RSS Feeds
.........................................................
Appliance Industry
Market Research

...........................................................

November 2014: U.S. Appliance Industry: Market Value, Life Expectancy & Replacement Picture 2014
October 2014: Portrait of the European Appliance Industry
September 2014: Appliance Industry Focus: HVAC
June 2014: Appliance Magazine Market Insight: April 2014




 
Contact Us | About Us | Subscriptions | Advertising | Home
UBM Canon © 2014  

Please visit these other UBM Canon sites

UBM Canon Corporate | Design News | Test & Measurement World | Packaging Digest | EDN | Qmed | Plastics Today | Powder Bulk Solids | Canon Trade Shows