advertisement

Chemometrics Method comparison Department of Chemical Pathology, University of Pretoria, Dr R Delport 2003 Chemometrics Performance characteristics that are taken into account: Precision, Accuracy, Interference, Working Range, and Detection Limit. http://www.westgard.com/lesson21.htm Chemometrics To carry out a good method validation study, you need to do the following: • Define a quality requirement for the test in the form of the amount of error that is allowable, preferably an allowable total error, • Select appropriate experiments to reveal the expected types of analytical errors, • Collect the necessary experimental data, • Perform statistical calculations on the data to estimate the size of analytical errors, • Compare the observed errors with the defined allowable error, and • Judge the acceptability of the observed method performance. http://www.westgard.com/lesson21.htm Chemometrics An experimental plan can be formulated by: • Recognizing the types of errors that need to be assessed for this test and method, • Identifying the appropriate experiments and the amount of data needed to estimate those types of errors, then • Organizing these experiments to perform the quick and easy ones first and the ones taking more time and effort last. http://www.westgard.com/lesson21.htm Chemometrics Types of errors • Imprecision or random errors, • Inaccuracy, bias, or systematic errors, which can be of two types •Constant systematic error or •Proportional systematic error. http://www.westgard.com/lesson21.htm Chemometrics The dashed line in the middle of the figure represents ideal method performance where the test method and the comparative method give exactly the same results. The bottom line in the figure shows the effect of a proportional systematic error, where the magnitude of the error increases as the test result gets higher. The top line shows the effect of a constant systematic error, where the whole line is shifted up and all results are high by the same amount. Note that these results will also be subject to the random error of the method, therefore the actual data points would scatter about the line as illustrated in the figure. The range of this scatter above and below the line provides some idea of the amount of random error that is present. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Description of different experiments. Replication experiment • Provides information about random error • Is performed by making measurements on a series of aliquots of the same test samples within a specified period of time, usually within an analytical run, within a day, or over a period of a month. • Preliminary experiment involves determining within-run imprecision. • Final experiment requires at least 20 working days to provide estimate of the total imprecision, which includes within and between run components. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Description of different experiments. Interference experiment • Provides information about the constant systematic error caused by the lack of specificity of the method. • One test sample is prepared by adding the suspected material to a sample containing the analyte. • A second aliquot of the original sample is diluted by the same amount with solvent, then both samples are analyzed by the test method and the difference determined. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Description of different experiments. Recovery experiment • Provides information about the proportional systematic error caused by a competitive reaction. • Test sample is prepared by adding a standard solution of the analyte being tested to an aliquot of a patient specimen. • A baseline sample is prepared by adding an equal amount of the solvent used for the standard solution to a second aliquot of the same patient specimen. • The two samples are then analyzed by the test method and the amount recovered is compared to the amount http://www.westgard.com/lesson21.htm added. Chemometrics Experiments for estimating analytical errors Description of different experiments. Comparison of methods experiment • Is primarily used to estimate the average systematic error observed with real patient samples. • Can also reveal the constant or proportional nature of that error. • A series of patient specimens are collected and analyzed by both the test method and a comparative analytical method. • The results are compared to determine the differences between the methods, which are the analytical errors between the methods. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Get the method working and establish an operating protocol. • Set up the instrument, prepare the reagents, calibrate the methods, and obtain results from test samples. • Check the standards and be sure the method is properly calibrated, otherwise calibration errors will show up throughout the experimental studies. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Determine the working range. • The working range will vary from test to test and must be defined as part of the specifications for the method. • Check by analyzing a series of solutions, in duplicate or triplicate, covering the concentrations range of interest. • If detection limit is a critical characteristic, it may be assessed at this time or in the next phase of preliminary experiments. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Determine within-run imprecision, recovery, and interference. • The replication experiment might include 20 samples of two or three materials whose concentrations closely match the medical decision levels of interest for the tests. • Interference experiments should test common problems such as hemolysis, lipemia, and high bilirubin. • Recovery experiments assess whether there are any competitive reactions due to the matrix or other materials in the native specimens. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Comparison of methods experiment. • Fresh patient specimens, and stored specimens. • Minimum of 40 well-chosen patient samples should be tested over a minimum of 5 working days. • Distributed one-third in the low to low-normal range, onethird in the normal range, and one-third in the high abnormal range. • Method acceptability should be judged on the basis of the sizes of the random, systematic, and total analytical errors. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Comparison of methods experiment. Difference plot • Used as initial graph if the two methods are expected to show one-to-one agreement, • Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Comparison of methods experiment. Difference plot • Used as initial graph if the two methods are expected to show one-to-one agreement, • Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure. • Differences should scatter around the line of zero differences, half being above and half being below. • Repeat measurements if indicated. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Comparison of methods experiment. Correlation coefficient • The correlation coefficient is a statistic that is almost always calculated and reported to describe the results from a comparison of methods study. • The new or "test" method values are plotted on y-axis and comparison values on x-axis. http://www.westgard.com/lesson21.htm Chemometrics Experiments for estimating analytical errors Walking tour of the plan. Comparison of methods experiment. • A value of 1.000 indicates perfect correlation between the results of two methods. • Other statistics (such as slope, intercept, and standard deviation of the residuals) can also be calculated from the same data to estimate the size of errors occurring between the methods. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Random error, RE, or imprecision • Can be either positive or negative. • Direction and exact magnitude cannot be predicted. Random error • Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Random error, RE, or imprecision • Can be either positive or negative. • Direction and exact magnitude cannot be predicted. Random error • Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. • As the SD often increases as the concentration increases, the coefficient of variation (CV) is calculated to express the SD as a percentage of the mean concentration from the replication study. • Maximum size of a random error is commonly expressed as a 2 SD or 3 SD estimate to help understand the potential size of the error that might occur. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Systematic error, SE, or inaccuracy • Is always in one direction. • A systematic shift displaces the mean of the distribution from its original value. Systematic error • In contrast to random errors that may be either negative or positive and whose direction can not be predicted, systematic errors are in one direction and cause all the test results to be either high or low. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Systematic error, SE, or inaccuracy Systematic error • How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. • Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Systematic error, SE, or inaccuracy Systematic error • How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. • Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis. • SE may stay the same over a range of concentrations, in which case it can also be called constant error, or it may change as concentration changes, in which case it can be called proportional error. http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Total Error, TE, • It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. Total error http://www.westgard.com/lesson15.htm Chemometrics Analytical errors Total Error, TE, • It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. Total error • It represents a "worst-case" situation, or just how far wrong a test result might be due to both random and systematic errors. • Because laboratories typically only make a single measurement for each test, that measurement can be in error by the expected SE, or bias, plus 2 or 3 SD, depending on how you quantitate the effect of RE. http://www.westgard.com/lesson15.htm Chemometrics analysed with: A nalyse-It + Clinical Labo rato ry v1.40 Test Bias plots NCCLS EP9-A example dataset Method: Comparative v Test Performed by Analyse-it Software, Ltd. Identity line A =B 300 Date n 1 February 1999 40 Mean of Method - Test 250 Bias 95% CI 200 -0.1 -2.3 to 2.2 150 95% limits of agreement Lower Upper 100 50 0 0 100 200 Mean of Method - Com parative 300 -13.7 13.6 95% CI -17.4 to -10.0 9.9 to 17.3 50 Chemometrics 0 0 100 200 300 Difference between methods 15 10 5 0 Zero bias -5 -10 -15 0 100 200 Mean of Method - Com parative 300 0 5 10 15 Chemometrics Difference between methods (%) 15 10 5 0 Zero bias -5 -10 -15 0 100 200 Mean of Method - Com parative 300