Estimates of Uncertainty in Microbiology Testing
“At Wickham Laboratories, food clients can be sure that UKAS accredited food microbiology tests are consistently monitored for levels of proficiency”
When conducting food analysis and examination, there is much to be said for establishing the certainty of quoted results. Indeed, these days it is a necessary requirement for any UKAS-accredited testing organisation to demonstrate its proficiency at performing those procedures covered in its schedule of accredited tests. In short, uncertainty comes down to determining the value of potential errors encountered while performing a particular procedure. Such an approach is a vital concept within the requirements of ISO17025 and at Wickham Laboratories, food clients can be sure that UKAS accredited food microbiology tests are consistently monitored for levels of proficiency.
Without complicating the issue with statistics and equations, a brief explanation of uncertainty of measurement follows thus. If a procedure is followed x number of times, it is highly unlikely that the result will always be the same. For instance, if the procedure were to start a stopwatch counting and then stop it dead on sixty seconds to hundredths of a second, and this was attempted ten consecutive times, there would be little chance of stopping it even once dead on 60.00 seconds. However, if all the stop times were recorded, probability would dictate that half of the attempts would stop before the 60.00 second target and half to stop after the 60.00 second target. If this exercise was repeated many times, and the differences between the recorded and target times were plotted on a graph, we would create what is commonly referred to as a normal distribution curve, as illustrated below with the majority of attempts being very close to the target time and only a few outlining results well off the mark either side. In food analysis and examination, the same principles apply in that for any particular procedure, various influencing factors will mean that for any repeated analysis, it is unlikely that the exact same result will be found.
Where there is no known target, values obtained from replicates of the same procedure can be used to calculate an average or mean value for the experimental result. Again, in normal distributions, there should be the same number of values lower than the mean as above the mean value. To determine how well, or with what level of proficiency, the procedure is performed, we look at how widely spread are the individual values either side of the mean.
When plotting experimental data in the above format, there will obviously be a small proportion of replicate results, which lie furthest from the mean value. These so-called "outliers", if low in number, are considered to be insignificant and so the proficiency of a test method is based on the number of results that are closest to the mean value. This is where the expression "confidence interval" originates, such that phrases like 95% confidence limits indicate the ± range either side of the mean value in which 95% of the replicate results will fall. It is generally accepted when calculating estimates of uncertainty that repeatability data should be interpreted to provide 95% confidence limits for that procedure. A 95% limit can be calculated as 2 x square root of the variance often referred to as the standard deviation and is a measure of the dispersion of probability about the mean value.
Where the value of the standard deviation is small compared to the mean value of a series of replicate results, the distribution curve will be narrow, indicating good reproducibility of the procedure and hence a demonstration of high proficiency. If, compared to the mean value, the standard deviation is large, the reproducibility is correspondingly low. Using a chemistry related example where the fat content of a single food sample is determined many times, let's say the mean value is determined to be 11.5%. This mean was determined from ten replicates, their individual values being:
10.7 11.2 12.2 12.7 11.6 11.3 11.8 10.5 11.8 11.2
The calculated standard deviation equals 0.66% and by multiplying this by two gives 95% confidence limits. This means that 95% of results if the test were repeated over and over again should fall between 11.5% ± 1.3 or 10.2 and 12.8%. This would be considered to be a fairly narrow range and so the test proficiency can be considered favourable. Now suppose the ten replicates giving the same mean value were as follows:
8.3 14.6 12.2 12.7 7.1 9.3 13.5 14.9 11.5 10.9
In this case the standard deviation equals 2.6%, giving a 95% confidence limits of 9.4 and 14.1%. This range is much greater than for the first example and if were applied to a test value on an unknown product, then the fat percentage could be out by as much as five percent - a figure that may not sound much until you consider legislative requirements for some food products. If a legislative standard dictates that a product may have no more than 20% fat, but a manufacturer for commercial reasons wants to be sure of attaining a fat content as close to this limit as possible, is it certain that the laboratory results for that product are not going to lay the company open for prosecution, should the fat content analysis return a figure of 18% with the true figure being nearer 23%?
Now that we have a better understand of the concept of uncertainty, it is worthwhile identifying where the causes of variation between results originate. When dealing with food analysis (chemical determinations) or examination (microbiological assessment) there are a number of common sources of variation introduced while performing a particular test. Examples of these are given in the table below.
| Storage conditions prior to and after sampling | Environmental factors that can affect both chemical and microbiological properties of foods prior to testing may not be full appreciated. Most food industry people know the effects of increased temperatures on micro-organisms growth, but how about comparing bacterial levels in a frozen product compared to a chilled product. Some micro-organisms are particularly susceptible to freezing and will not survive. |
| Sampling | When a food product is presented for testing, it is usually the case that only a proportion of the material is required for testing, e.g. 10 grams. Selection of this proportion introduces variation as food materials are infrequently homogenous in their distribution of properties being examined. This is especially true where micro-organisms are concerned. |
| Equipment functionality | Whether it be digital balances for weighing, or incubators for encouraging growth of organisms, all laboratory instrumentation and equipment which either provides a value or operates to a defined target will have some inaccuracy associated with it. For instance a thermometer may well read 37°C but this will have a confidence limit associated with it which may be ± 0.5°C. |
| Reagent purity | Microbiological media and chemical reagents may not be guaranteed free of impurities that may affect results. These impurities may not be uniformly distributed throughout a particular batch of the reagent |
| Measurement conditions | Again equipment functionality is to some extent affected by the conditions it is operating in. For example laboratory humidity may affect materials; ambient temperature may not match the standard requirements recommended for volumetric glassware. Expansion/contraction of measuring devices will alter associated measurements. |
| Operator effects | Despite training to a standard operating procedure, human nature means that subjective differences will exist between operators. In the worst cases, systematic or unintentional bias may be introduced. |
| Counting errors | Particularly applicable to microbiology, distribution of organism counts derived from colony-forming units from serial dilutions, will invariably lead to no two counts being exactly the same if a sample is examined in replicate, no matter how good the laboratory and its staff. |
| Random effects | Along with all of the above, there is always room for further errors or causes of variation. Also bear in mind that where microbiological methods are concerned, some of the above factors when combined with others may act synergistically, magnifying the level of error encountered. Where chemical determinations are concerned, each error usually will not influence the next and so a gauge of the overall estimate of uncertainty may be calculated from the sum of individual component errors. |
All this is very well, but how does a laboratory provide an estimate of uncertainty for a test determination when it does not know what the true value should be. Laboratories operate quality control systems whereby, for each method under observation, known samples are processed alongside customer unknown samples. Even then assessing estimates of uncertainty for microbiological methods still cause difficulties in knowing exactly how many micro-organisms should be present in a food sample before analysis begins. The closest microbiologists can get to testing a food with a "known" number of organisms is by use of external proficiency schemes and in-house spiked control samples. Wickham Laboratories operates the HPA Food EQA scheme, whereby periodically freeze-dried preparations of organisms originally isolated from food are despatched to participating laboratories for examination. Results for laboratory enumeration, or presence/absence testing of these culture preparations are returned to the scheme organisers and compared to “intended results”. Even these intended results cannot be interpreted as exact counts for each sample despatched to each laboratory, and are instead are mean values achieved through extensive repeated analysis by HPA laboratories throughout the UK.
Compilation of HPA results over the several years has enabled estimates of uncertainties to be determined for many of our standard UKAS-accredited test methods, both enumerations and presence/absence testing of pathogens. These can be expressed in a similar manner to fat analysis example illustrated above.
Wickham Laboratories is able to supply its food customers with estimates of uncertainty upon request. Where necessary certificates of analysis can be provided to carry the test result together with a ± uncertainty value (95% confidence interval) and appropriate units of measurement together with the following statement:
"The reported expanded uncertainty is based on a standard uncertainty multiplied by a coverage factor of k=2, providing a level of confidence of approximately 95%, *but excludes the effects of bacterial distribution in the sample tested. For further explanation of how this figure relates to your result and its limitations please contact Wickham Laboratories Limited."