ASTM E3093-20 - Standard Guide for Structured Small Group Product Evaluations

1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product Evaluation” (SGPE) is used.

1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training, the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these SGPE are used to inform decisions and determine next steps including the risks involved with each of these.

SGPE are widely used, and when properly conducted, are an option in the sensory professional’s toolbox. SGPE should be conducted only when the risks are known, stated, and shared. Limited timing and resources alone are not adequate reasons to utilize SPGE testing and forgo formal sensory testing. Risks in doing so must be clearly communicated and agreed to by all involved parties.

The proper uses of SGPE are several: to screen variables, to establish hypotheses, to gain information about a product set or category, to take a course of action where a low risk product decision is needed or for product learning throughout a development program. In all of these cases, the team must accept the risks that come with having SGPE outputs to inform a decision. One risk involved in SGPE is missing small differences among products (beta risk), when the goal of the evaluation is to find such differences, particularly those differences that might be important to the consumer. An SGPE failure to find differences does not mean that product similarity or equivalence is established, since much larger sample sizes than are common to SPGE’s are required to establish similarity/equivalence.

1.3 This guide covers the planning and implementation processes, including objective setting, method determination, number and types of participants, ballots, sample preparation, decision criteria, products to be included, review of information collected, and management of the post-product evaluation discussion to arrive at a decision within the small group. Documenting and communicating SGPE outputs are also covered, as well as next steps if a decision cannot be reached. Worked examples across industries including food, household, and personal care are included. The different types of SGPE covered include those commonly executed but is not exhaustive.

1.4 This guide does not cover the use of small group evaluations to pilot research or test protocols before implementation in larger scale testing. In addition, the use of small group evaluations to substitute for larger evaluations that incorporate formal hypothesis testing and statistical analysis or to replace hedonic testing are neither recommended nor included within this guide. SGPE that are regular activities of a quality function and product reviews that are done for demonstration or informative purposes with no defined decision criteria are also not covered in this guide.

1.5 See 5.2 for a best practice recommendation for the role of the sensory professional or trained delegate in the planning, designing, conducting, or oversight of structured SGPE.

1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.

1.7 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.