Companies in the food industry have made significant strides in increasing the quality and safety of the food supply. The Food Safety Magazine, April/May 2012 issue notes that quality is defined differently within the food industry based on the customer.  Whatever the definition used, the business of quality in the food industry must be proactive.  With this in mind, the article goes on to review two programs and tools used to address quality and manage risk.  These are the Hazard Analysis and Critical Control Points (HACCP) program and the Failure Mode and Effects Analysis (FMEA). A remarkable amount of focus and energy has gone into developing comprehensive HACCP plans that mitigate and control risk in order to ensure high quality standards are met within the food industry.   The FMEA is now seen as an important analysis tool to be used in conjunction with the HACCP to ensure quality and brand protection.

The FMEA is done to analyze the potential for error and define mitigation strategies where that risk is identified.  The importance of this tool and the need for diligent self-regulation in food testing laboratories is made more clear by an article published in the Journal of Food Safety, October 2009:  Pathogen Detection in Food Microbiology Laboratories:  An Analysis of Qualitative Proficiency Test Data, 1999-2007.  This piece details cumulative findings from a study of proficiency test data in which labs were testing for the presence or absence of four common pathogens, Escherichia coli 0157:H7, Salmonella spp., Listeria monocytogenes and Campylobacter spp. in proficiency samples. The study reported, “The cumulative 9-year false-negative rates were 7.8% for E. coli 0157:H7, 5.9% for Salmonella spp., 7.2% for L. monocytogenes and 13.6% for Campylobacter spp…. Percentages of false-positive results were below 5.0% for all four pathogens.”  The study goes on to suggest the high rate of false-negative results may be attributed to process variation, including insufficient staff training, improper temperatures and inadequate incubation times.  The authors propose that the magnitude of these errors in the study may well be less than those during normal operation, testing real food samples, as testing personnel would not necessarily be treating the sample with the extra care given to those samples designated for proficiency testing. 

Roka Bioscience has expanded the application of hazard analysis placing focus specifically on pathogen testing methods within the food testing laboratory.  The FMEA facilitated detailed analysis and identification of potential process risks inherent in multiple molecular and immunoassay pathogen testing methods.  Phase one of this study is concentrated on two of the top five pathogens contributing to foodborne deaths¹, Listeria spp. and Salmonella enterica.

Key Elements of the Study

• Manufacturers’ Product Insert Sheets and User Guides
• Failure Modes and Effects Analysis Tool
• Process  and Food Pathogen Microbiology Experts

It was important to begin with a complete understanding of the most commonly used methods for detection of Listeria and Salmonella on environmental surfaces, and in raw and finished products.  Methods selected for comparison analysis are listed below.

• Listeria Testing:  Molecular Methods
  • Roka Bioscience Atlas® - universal environmental and food protocol
  • DuPont™ BAX® Standard – environmental and food protocol
  • DuPont™ BAX® 24E - frankfurters, spinach, shrimp, queso fresco cheese and stainless steel
  • Bio-Rad IQ Check™ - Easy I protocol - stainless steel, plastic, ceramic and sealed concrete
  • Assurance GDS™ Listeria spp. – deli meats, hot dogs, seafood, dairy products, produce, soft cheese, stainless steel, rubber, plastic and concrete.
• Listeria Testing:  Immunoassay Methods
  • bioMérieux VIDAS® - universal food protocol
  • bioMérieux VIDAS® - environment protocol
  • bioMérieux VIDAS® - LSX universal food protocol
  • bioMérieux VIDAS® - LSX environmental protocol
• Salmonella Testing:  Molecular Methods
  • Roka Bioscience Atlas® - universal environmental and food protocol
  • DuPont™ BAX® Standard – environmental and food protocol
  • Bio-Rad IQ Check™ - Easy I protocol - eggs, raw chicken, raw beef, cantaloupe
  • Bio-Rad IQ Check™ - Standard I protocol - food products, animal feed and environmental samples
  • Assurance GDS™ Salmonella – foods, ingredients and environmental samples
• Salmonella Testing:  Immunoassay Methods
  • bioMérieux VIDAS® - universal environmental and food protocol
  • bioMérieux VIDAS® -  Easy Salmonella - universal environmental and food protocol

See Appendix A for package insert and/or user guide reference information for each method.

Developing the Failure Modes and Effects Analysis (FMEA)
Manufacturers’ product insert sheets and user guides were reviewed for each test method to complete this detailed comparison analysis of all requirements in the processing of a sample.  The scope of this study begins with primary enrichment and ends with final result documentation. Sample types within scope were environmental swabs, raw product and finished product.

The FMEA is a quality tool commonly used in industries where risk prevention is important to the successful development of a product as well as the output or final result of that product.  By providing products to consumers through a zero-defects environment risk for both the customer and brand can be greatly reduced.  The FMEA is a key tool in the quantification of total risk and identification of individual process steps posing highest potential risk.

An FMEA provides documentation of the following data for each step in the test method under study:

1. Potential failure mode(s) – All potential defects or ways there could be a failure in the process.
2. Consequence of the defect – Specifically how the sample or result would be affected if the defect does occur.
3. Current control(s) – All controls or procedures put in place to identify and or mitigate the potential for the defect to occur.

In order to accurately document this data a team including process experts and food pathogen microbiology experts examined each test method.  The first step was recording the individual process steps.  Next, all associated potential failure modes/defects, consequences of those defects and controls consistently in place to mitigate or help the operator quickly recognize a defect were documented.  

Example Process Step:  Place lysis tube in 37° (+/- 2°) C heat block for 30 minutes

1. Potential failure mode/defect:  Heat block not within tolerance range for temperature.
2. Consequence of defect:  Ineffective lysis  resulting in indeterminate or incorrect result.
3. Current controls – Temperature dial on heat block, and thermometer placed in heat block.

Each potential failure mode/defect on the FMEA was ranked for the following:

1. Severity (S) –The severity of damage relative to the end customer when the defect does occur.
2. Frequency (F) – How often the defect is likely to occur based on current controls.
3. Detection (D) – The likelihood of detecting a defect and when it may it be detected.

For purposes of this study it was important to have the group of food pathogen microbiologists come to agreement on these ratings in each process step.  The group consisted of microbiologists who have experience with all methods under study.  Standard rating scales were developed with 3 options for each category.  Ratings of 1, 5, and 9 were assigned.  The rating of 1 is associated with the option having the lowest potential risk to the sample, final result or customer.  The rating of 9 is associated with the option having the highest potential risk to the sample, final result or customer.  See Appendix B to review the standardized rating scales. 

Example Potential failure mode/defect:  Heat block not within tolerance range for temperature.

1. Severity – 9 – Rating assigned due to the potential for an incorrect result
2. Frequency – 1 – Rating assigned based on inability of the team to determine how often the defect is likely in laboratories with highly variable conditions.  This (1) was the rating most commonly assigned for F throughout the study.
3. Detection – 5 – If the dial and thermometer are properly calibrated and the operator is not too busy to review them the defect may be recognized.  Human or mechanical error remains possible resulting in the assignment of 5.

The risk priority number3 (RPN) is derived through the simple calculation (S x F x D).  Upon completion of the FMEA for each selected test method the total RPN for that method was calculated by summing the RPNs for each individual potential failure mode/defect.

Example Potential failure mode/defect:  Heat block not within tolerance range for temperature.

• RPN = (S x F x D) = (9 x 1 x 5) = 45
• Total RPN = 45 + Sum(RPN for all potential failure modes)

Summary of All-Method FMEA
Additional, valuable data derived from the FMEA development process included the total number of process steps, touches per sample4 by all operators in the process and defect opportunities.  The total RPN of each test method in both studies, Listeria and Salmonella, correlates well with the number of process steps and the number of required touches.  The methods with the highest number of process steps and required touches have the highest RPN, therefore higher risk of error in the process and potential for direct negative effect on the customer.

Listeria:

The Roka Atlas System is a continuous flow automated system requiring only one transfer step per sample following primary enrichment.  Controls are present in every sample tube and calibrations, (adding 2 company-prepared calibrators to an Atlas rack and placing on the system) are required only one time every four days.  One reagent kit lasts for 250 samples.  By comparison, following primary enrichment, the molecular testing systems, DuPont BAX, Assurance GDS and Bio-Rad IQ Check, require multiple transfers as well as positive and negative control preparation, lysis preparation and PCR mix preparation with each batch of 94 or fewer samples.  It follows that these systems are significantly less efficient, adding 8-17 process steps (depending on the protocol) and minutes of operator time to prepare the sample for load onto the thermo cycler and complete result validation*.   The resultant RPNs are two and a half to over four times higher than the RPN for the Atlas System. 

We see similar results comparing the Atlas System to the immunoassay test system, bioMérieux VIDAS.   Depending on the protocol a secondary enrichment and incubation may be required.  The operator is required to keep track of temperature and time-dependent steps, and positive and negative controls are required for each batch.  Resultant RPNs are one and a half to over two times higher than that of the Roka Atlas System. 

Laboratories utilizing the Atlas System experience much higher efficiency and productivity when testing for Listeria than those using the other molecular and immunoassay methodologies studied, as shown by the number of sample touches and process steps.  Perhaps more important to the brand and reputation of the laboratory and its customer, the potential for an error to occur at some point in the overall process is decreased with the Atlas System, as shown through comparison of the total number of potential defects and calculated RPN for each method studied.   

*Assuming rerun of sample not necessary.

Summary of All-Method FMEA:  Salmonella
The same analysis was conducted for the Salmonella test methods.  The low level of complexity required by the Roka Atlas System, enrich, transfer and load on the Atlas continuous flow system, is the same for Salmonella and Listeria.  The molecular methods, DuPont BAX, Assurance GDS and Bio-Rad IQ Check, require multiple transfers as well as positive and negative controls, lysis preparation and PCR mix preparation with each batch of 94 or fewer samples.  These systems are significantly less efficient, adding 8-18 process steps (depending on the protocol) and minutes of operator time to prepare the sample for load onto the thermo cycler and complete result validation*.   The resultant RPNs are between two and three times higher than the RPN for the Atlas System. 

We see similar results comparing the Atlas System to the immunoassay test system, bioMérieux VIDAS.   Secondary enrichment and multiple transfer requirements increase the probability of contamination and error during the process.  This is most evident in the standard universal protocol that includes transfers to Rappaport Vassiliadis (RV) broth, tetrathionate (TT) broth and M broth.  The operator is required to keep track of and sync multiple transfers and incubation timing throughout the process.  As with the Listeria VIDAS methods positive and negative controls are required for each batch.  Resultant RPNs are two to more than three times higher than that of the Roka Atlas System. 

Laboratories utilizing the Atlas System for Salmonella testing experience much higher efficiency and productivity than those using the molecular and/or immunoassay methodologies studied, as shown by the number of sample touches and process steps.  As with Listeria, perhaps more important to the brand and reputation of the laboratory and its customer, the potential for an error to occur at some point in the overall process is decreased with the Atlas System, as shown through comparison of the total number of potential defects and calculated RPN for each method studied.   

*Assuming rerun of sample not necessary.

Conclusion
In an industry highly dependent on product testing to ensure consumer risk is as low as possible it is necessary to understand and evaluate the error potential in all test methodologies in order to make an informed decision on the right pathogen testing system.  The Roka Atlas System has been shown through this strict FMEA and resultant comparison analysis to induce significantly less risk into the testing process for both Salmonella and Listeria.   By decreasing the process steps required, human interaction with the sample is decreased significantly resulting in fewer potential errors and the ability to use testing personnel in more efficient ways.

By Maureen Harte, Lean Six Sigma Master Black Belt, HartePro Consulting LLC

For more information on FMEA and risk of error in the food industry see the following:

http://www.food-safety.com/magazine-archive1/april-may-2012/an-integrated-approach-to-food-quality-and-safety-a-case-study-in-the-cookie-industry/

http://onlinelibrary.wiley.com/doi/10.1111/jfs.2009.29.issue-4/issuetoc

http://blog.nelsonjameson.com/failure-mode-and-effects-analysis-in-the-food-industry

http://reliabilityweb.com/index.php/maintenance_tips/fmea_used_throughout_food_industry_to_improve_product_quality/

Appendix A:  Package Insert / User Guide Test Method Reference Information

• Listeria Testing:  Molecular Methods
  • Roka Bioscience Atlas® - universal environmental and food protocol
    • PRT 00255.  Atlas® Listeria Detection Assay
    • AOAC Validation, license number 011201
  • DuPont™ BAX® Standard – environmental and food protocol
    • Genus Listeria Standard assay, Kit #11000147
    • AOAC Validation, license number 030502
  • DuPont™ BAX® 24E - frankfurters, spinach, shrimp, queso fresco cheese and stainless steel
    • Genus Listeria 24E assay, Kit #D1360815
    • AOAC Validation, license number 050903
  • Bio-Rad IQ Check™ - Easy I protocol - stainless steel, plastic, ceramic and sealed concrete
    • IQ Check™ Listeria spp. Kit, Cat #357 8113
    • AOAC Validation, license number 090701
  • Assurance GDS, Genetic Detection System, Listeria spp.
    • AOAC Performance Tested Method 070701
    • Catalog No. 61009-100
• Listeria Testing:  Immunoassay Methods
  • bioMérieux VIDAS® - universal food protocol
    • VIDAS® Listeria (LIS), REF 30 700
    • AOAC Validation, certificate number 981202
  • bioMérieux VIDAS® - environment protocol
    • VIDAS® Listeria (LIS), REF 30 700
    • AOAC Validation, certificate number 981202
  • bioMérieux VIDAS® - LSX universal food protocol
    • VIDAS® Listeria species Xpress (LSX), REF 30 224
    • AOAC Validation, certificate number 100501
  • bioMérieux VIDAS® - LSX environmental protocol
    • VIDAS® Listeria species Xpress (LSX), REF 30 224
    • AOAC Validation, certificate number 100501
• Salmonella Testing:  Molecular Methods
  • Roka Bioscience Atlas® - universal environmental and food protocol
    • PRT 00250.  Atlas® Salmonella Detection Assay
    • AOAC Validation, license number 031201
  • DuPont™ BAX® Standard – environmental and food protocol
    • Salmonella (Standard Assay), Kit #D11000133
    • AOAC Validation, license number 100201
  • Assurance GDS, Genetic Detection System, Salmonella
    • AOAC Performance Tested Method 050602
    • Catalog No. 61008-100
  • Bio-Rad IQ Check™ - Easy I protocol - eggs, raw chicken, raw beef, cantaloupe
    • IQ Check  Salmonella II Kit, Cat #357 8123
    • AOAC Validation, license number 010803
  • Bio-Rad IQ Check™ - Standard I protocol - food products, animal feed and environmental samples
    • IQ Check  Salmonella II Kit, Cat #357 8123
    • AFNOR Validation, BRD 07/06-07/04
• Salmonella Testing:  Immunoassay Methods
  • bioMérieux VIDAS® - universal environmental and food protocol
    • VIDAS® Salmonella (SLM) REF 30 702
    • AOAC Validation, certificate number 996.08
  • bioMérieux VIDAS® -  Easy Salmonella - universal environmental and food protocol
    • VIDAS® Salmonella (SLM) REF 30 702
    • AOAC Validation, certificate number 020901

Appendix B:  Rating Scales for FMEA

1. Centers for Disease Control and Prevention. CDC Estimates of Foodborne Illness in the United States. http://www.cdc.gov/foodborneburden/2011-foodborne-estimates.html.

2.  Zero defects is an aspect of total quality management that stresses the objective of error-free performance in providing goods or services, http://dictionary.reference.com/browse/zero+defects

3. Risk Priority Number (RPN) definition:  The RPN is a numeric assessment of risk assigned to a process, or steps in a process, as part of an FMEA in which a team assigns each failure mode numeric values that quantify the severity of impact if/when an identified failure occurs, likelihood of occurrence, and the timing and ability to detect a failure.

4. “Touches per sample” refers to one or more operators touching/working with the sample in all forms as it moves through the process – sample to final result.