advertisement

In this section, The American Statistician publishes articles and notes of interestto teachers of the firstmathematicalstatistics course and of applied statisticscourses. To be suitable for this section,articlesand notes should be usefulto a substantialnumber of teachers of such a course or should have the potential for affectingthe way in which the course is taught. fundamentally ResearchDesigns ComparativeAnalysesof Pretest-Posttest DONNA R. BROGAN AND MICHAEL H. KUTNER* Two common methodsof analyzingdata froma two-grouppretestposttestresearchdesign are (a) two-samplet test on the difference score between pretest and posttest and (b) repeated-measures/ split-plot analysis of variance. The repeated-measures/split-plot analysis subsumes the t test analysis, althoughthe formerrequires more assumptions to be satisfied. A numericalexample is given to illustratesome oftheequivalences ofthetwo methodsofanalysis. The investigatorshould choose the methodof analysis based on the researchobjective(s). analysis; t test; PreKEY WORDS: Repeated-measures/split-plot test-posttestdesigns. 1. INTRODUCTION A commonresearchdesignis the two-grouppretest/ posttest design with one dependent variable where subjects are not matched and may or may not be randomlyassignedto thetwo groups(Cook and Campbell 1979). When the two groups are not formedby randomassignmentof subjects,a randomsample from each of the two groups is necessary. This design can be extended to more than two groups; an example is the comparison of several differenttreatmentswith each otheror witha controlgroupin whicheach group is measured on a pretestand posttest. The statistical analysis for these designs can be approached fromseveral viewpoints.If the dependent variable is measured on an intervalor ratio scale, a common analysis is to definea differencescore for each subject (posttestminuspretestor vice versa) or a relative differencemeasure (the differencedivided by the pretest)and then test the null hypothesisthat the means or medians of the (relative) differencesare equal for each group. In many cases the t test or analysis of variance is used, althoughnonparametric tests could also be used, for example, the MannWhitneyU test, or the median test, or theiranalogs formore thantwo groups. Covariance analysis,wherethe pretestscore is used as the covariate,is anothermethodused foranalyzing thisdesign. The differencescore methodis essentially * Donna R. Broganis Professorand Michael H. Kutneris Associate Professorin the Statistics and BiometryDepartment,School of Medicine, Emory University,Atlanta,GA 30322. Work on this articlewas partiallysupportedby NCI ContractNo. CB-74101 and USPHS GrantNo. RR39. a special case of the analysis of covariance where the regressioncoefficientof the postteston the pretestis assumed to equal unity.Neter and Wasserman (1974, p. 717) and Cox (1958, pp. 55-56) pointout thatifthe common slope is not near one the covariance analysis probably will be better than the differencescore analysis. We note thatwhen an experimentalgroupis to be compared to a controlgroup, it is oftenlikely thatinequalityofslopes willprevailamonggroups,thus violatingan assumptionof the analysis of covariance. of a Bock (1975, Sec. 7.3) compares the interpretation analysis and covariance analysis and difference-score suggestsguidelinesregardingwhich analysis to use. Still another method of analyzingthis design is to view the pretestand posttestas a repeated-measures/ split-plotdesign or as a profileof two measurements designs foreach subject. Repeated-measures/split-plot are discussed in detail by Winer (1971) and Steel and Torrie (1980), whereas both repeated measures and profileanalysis are discussed in Morrison(1976, Secs. 4.5, 4.6, and 5.6). This articleillustratessome of the equivalences and differencesbetween the differencescore analysis and or profile analysis. the repeated-measures/split-plot The numericalexample and major discussion are for designwhere subjects are a two-grouppretest/posttest not matched.Concludingremarksindicatehow the resultscan be extendedeasily to morethantwo groups. 2. A NUMERICAL EXAMPLE We consider data fromRikkers et al. (1978), who report results of a prospective randomized surgical trial allocating cirrhoticpatients who had bled from varices to eithera nonselectiveshunt(standardoperation) or to a selective shunt (new operation). The dependentvariableis themaximalrateofurea synthesis (MRUS), whichis a quantitativetest of liverfunction. Poor liver functionis associated with a low MRUS value. MRUS was measured preoperativelyand early postoperativelyin eight selective shunt patientsand thirteennonselectiveshuntpatients.The purposes of the studywere to assess preoperativelythe comparabilityof the selective and the nonselectivegroupsand evaluate the change in liver function to longitudinally C) The American Statistician,November 1980, Vol. 34, No. 4 229 1. Pre and Post MaximalRate of Urea Synthesis Level (mg urea N/hr/kg BW314)and Sample Cell Means, by Group Group Subject Selective Shunt (new operation) 1 2 3 4 5 6 7 8 Pre Post 51 35 66 40 39 46 52 42 48 55 60 35 36 43 46 54 All = 46.375 Mean Nonselective Shunt (standard operation) /L12 = 34 40 34 36 38 32 44 50 60 63 50 42 43 9 10 11 12 13 14 15 16 17 18 19 20 21 Mean =21 /412 - /11 422 = 47.125 - 11) - (G22 - tk21) = 0 0 = 0 3. REPEATED-MEASURES ANALYSIS 31.462 of the two groups. Table 1 reportsthe MRUS values foreach patientforthepreoperativeand postoperative periods and the respectivecell means. For completeness Table 2 displays the standard repeated-measuresanalysis of variance table (analysis of means method);in thisexample the totalnumberof subjects, n, is 21. The hypotheses of interestto the researchers were the interactiontest and the simple effects test on equality of preoperative population means between groups. The test for interactionis significant(F = 11.36 with 1 and 19 df, p < .005); therefore,it is concluded that the pre/postaverage differchange in the nonselectivegroupis significantly ent fromthe pre/postaverage change in the selective group (see the figure).In the presence of a significant interactioneffectitis generallyofinterestto testsimple effectsratherthan main effects(Winer 1971, p. 529). The Bonferronimultiple-comparisonprocedure (see Neter and Wasserman 1974) was adopted to test the followingcontrasts: G412 -21 = Using an experimentwise errorrate of .05, we conclude that the interactioneffect is significantand /21 is significantly greater than 922. Therefore,significantdeteriorationof liverfunctionoccurredin the nonselectivepatientsbetween preoperativeand early postoperativeevaluationperiods,whereasthe selective group had no apparentdeleteriouseffect.Two points are worth noting in the example just cited: (a) The equality of slopes test using the preoperativeMRUS values as a covariate is rejected(p < .02); and (b) the significantinteractioneffectrequires special handling when testingthe last contrast since, for the pretest level, we have a two-groupexperimentin whichthere are no repeated measures. Therefore,the appropriate errortermfor this type of comparison is MS (within cell). For a more extensive coverage of this pointthe reader is referredto Winer(1971, pp. 529-532). 16 36 16 18 32 14 20 43 45 67 36 34 32 43.538 /21 /422- and 0 11 = We now discuss the statistical properties of the repeated-measures analysis of variance for this example and compare it withthe statisticalproperties of the differencescore analysis. Using the model proposed by Winer(1971, p. 519), we have Xijk = /U + ai + rk(i) + /3j+ a/i j + 131jk(i) + Em(ijk) (3.1) j = 1, 2 (pretest= 1, posttest= 2), i = 1, 2 (group 1 = 1, group 2 k = 1, 2, . . . , = 2), n1, m = 1, where Xijk is the observed value of subject k within groupi at timej, , is the overall mean, ai is the effectof groupi, f1k(i) is the effectof subject k nested withingroupi, ,Bjis theeffectoftherepeated-measuresvariable] (i.e., pretestand posttest), aj3ij is the interactionof groupi withlevelj of the repeated measures factor, 2. Repeated-MeasuresAnalysisof VarianceforMaximalRate of Urea SynthesisLevel 230 Source of Variation df Between Subjects Groups Subjects WithinGroups WithinGroups Pre/Post Groups x Pre/Post (Pre/Post)x Subjects WithinGroups 20 (n - 1) 1 19 (n - 2) 21 (n) 1 1 19 (n - 2) Sum of Squares Mean Squares 847.48 4440.00 847.48 (MSG) 233.68 (MSE) 317.69 407.41 681.21 317.69 (MSP) 407.4 (MSGP) 35.85 (MSPE) C The American Statistician,November 1980, Vol. 34, No. 4 F Ratio 3.63 (MSG/MSE) 8.86 (MSP/MSPE) 11.36 (MSGP/MSPE) and is the interactionof subjectk withingroupi with levelj of the repeated-measuresfactor. 50 13n1jk(i) The following constraints are imposed on the parameters: where a. = 3. a 8i. = aBJ3.j= 0, af.j= a ,8ij,and so on. coo 0 -t " z40 (3.2) In the designunderdiscussion,the repeated-measures factorand the groupfactorare each at two levels. The generalanalysis of variance also is indicatedin Table 2, where n is the total number of subjects. Note thatit is not necessaryforeach groupto contain the same number of subjects. Assuming the group factorand thepre/postfactorto be fixedeffects,Winer (1971) shows thatthe appropriateF tests are as indicated in the F ratio column of Table 2. It is worthnotingexactly what null hypothesesare tested in Table 2. The ratio MSG/MSEtests the null hypothesisthatthereis no group main effect.This is equivalentto testingwhetherthesum ofthepretestand posttest observations on each subject has the same populationmeaninthetwogroups.The ratioMSP/MSpE tests the nullhypothesisthatthereis no prelpostmain effectand is equivalentto testingwhetherthe population mean of the pretestobservationsis the same as the populationmean of the posttestobservations.The ratio MSGP/MSPE teststhe null hypothesisthatthereis no interactionbetween the group main effectand the pre/postmain effect.This ratio also tests whetherthe differencebetween pretestand posttest observations has the same populationmean in both groups. This is the test manyresearchersare interestedin when using this research design, since they oftenwish to assess has had any effectupon an experiwhethera treatment mentalgroup. Note thatthisF test has (1, n - 2) df, which will correspondto the t test with(n - 2) df,as discussed in the next section. Two assumptionsare requiredto arriveat theF tests indicatedin Table 2 (Winer 1971). 1. The pretest and posttest population variancecovariance matricesforeach group are assumed equal. 2. The randomeffectsHk(i), PIjk(i), and Em(Ujk) from the model in (3.1) are all independentlyand withmeanzero and variances normallydistributed ( and OE2, cH2, respectively. I2. (TO Assumption(1), equalityof the variance-covariance matrices,impliestwo otherresultsworthnoting.First, the variation of subjects within the two groups is homogeneous. That is, if each subject's pretestand posttestobservationsare added together,thissum has the same population variance in both groups. This allows pooling over groups to calculate SSE. Second, the variationof the interactionof subject and the pre! post factoris homogeneousforthetwo groups.That is, if the differencescore between pretestand posttestis N E Shunt v e ~~~Selective ~~~o--o NonselectiveShuntb - 30 E Post (AfterSu rgery) Pre (Before Su rgery) Mean Pre and Post />aximal Rate of Urea Synthesis Level (MRUS) by Type of Surgery defined for each subject, the population variance of the difference scores is the same for both groups. This allows pooling over groups to calculate SSPE:. 4. DIFFERENCE-SCORE ANALYSIS Using dik dik model (3.1) and forming a difference score for each subject k nested in group i yields = Xi2k Xilk- = (,31 - + P2) +(Befor Sureir) Poslk(i)(t + (Em(ilk) Srk(r) EM(i2k)) * (4 .1) - a,8i2) iS a parameter associatedwith The term(MR,US group i and measures the "effect" of group i on the differencescore dik. The null hypothesisto be tested - a12 is How pooli - ove12=g p21 - cal22 or l1 Ho: = Ui821 - md822 = a.The differencescores in (4.1) can be viewed as a one-way classification model in which the errortermis the sum of the followingtwo terms: (WIlk(i) - P2k(i)) and (Em(ilk) - Em(2k)) If we assume this errortermand also homogeneous variances forthe two groups, an appropriatetest statisticis the Studentt testfortwo independentsamples with (n - 2) df. theerortrmi the sum of the followin poteteifrms: two 5. COMPARISON OF THE TWO ANALYSES homogeneousedt term an tw-amlso weassmchisberro Ifo thetogroups m ans appropriatetheststavracesmfore The followingthree results are useful computationally samples t tes for0 two indtepsuaendent titcaisltheStden and can be verifiedeasily with the example from etfrGop nTbe2with df. F(n1,12) C The American Statistician,November 1980, Vol. 34, No. 4 231 2. Ifthedifference betweenthepretestand theposttest is formedforeach subject and a two-sample t testis used to comparethegroupmeans ofthese differences,thenthe calculated t = 3.371 and is thesquare rootoftheF testforGroupsx (pre/post) in Table 2 with(1,19) df. 3. Ifall 21 subjectsare consideredto be inone group, thenthe t statisticto test the nullhypothesisthat the mean differencescore is zero has 20 df and equals 3.158. From Table 2, if we reanalyze the within-subjectscomponentby assumingthatthe Group x (pre/post)interactionis zero and SSGP is pooled with SSPE, then the F test for the main effectpre/postyields F = 9.97 with (1,20) df, whichequals thesquareoftheprecedingt statistic. These resultsdemonstratethatthevariousF testsin the repeated-measuresanalysis of variance can be obtainedby usingsimplet testson linearcombinations ofthepreand post scores. It can be shownalgebraically thatinterpretations (1), (2), and (3) of theF tests hold in the particular research design discussed in this article,thatis, a two-grouppretest/posttest design. In fact,thenumericaloperationsin (1) and (2) of summing and differencing observationsare the pretest/posttest used by the latestversionof the BMD P2V programin calculatingsums of squares in repeated-measuresdesigns (Dixon and Brown 1979). Since the difference-score analysis is embedded in therepeated-measuresanalysis,therepeated-measures about the data at analysis provides more information hand. Fewer assumptions,however,are requiredinthe difference-score analysis.The difference-score analysis assumes only homogeneous variances for the difference scores and a normallydistributederrortermwith mean zero. It is easy to show thatif the assumptions oftherepeated-measuresanalysisare satisfied,thenthe assumptionsof the difference-score analysis are also met. However, the converse is not true. In our experience,theresearcherrarelyis interested in onlytheinteractiontest,thatis, thedifference-score analysis. Furthermore,simpleeffectsare commonlyof interesteven in the no interactioneffectexperiments. Therefore, we advocate the use of the repeatedmeasures/split-plot analysis in most instances. However, we urge the user to empiricallyvalidate the underlyingassumptions. In theexamplediscussedin Section2, theresearchers wouldhave been interestedinassessingthesignificance 232 of the main effecttime (pre vs. post) if the Groups x pre/postinteractionhad been nonsignificant. That is, thenonsignificant interactionwould have indicatedthat the two groups did not differsignificantly on their MRUS differencescores. The pre/postmaineffecttest would then indicate whether the MRUS difference score in both groups was significantly from different zero, that is, whetherthe treatmentdid or did not effectboth groups. 6. GENERALIZATION OF FINDINGS If there are more than two groups, similarresults can be obtained. The difference-score analysis would no longerbe performedby a t test but by an F test in a one-way analysis of variance. A furtherextension can be made wherethedifferent groupsbeingcompared maybe definedby severalfactorsin a factorialdesign. However, there is no logical extension of this discussion to more than two levels of the repeatedmeasuresfactorsince a simpledifference score analysis would no longerbe appropriate. [Received July1977. Revised June 1980.] REFERENCES Bock, R. Darrell (1975), Multiv'ariate Statistical Methods in? Behavioral Research, New York: McGraw-Hill Book Co. Cook, Thomas D., and Campbell, Donald T. (1979), QuasiExperimentation:Design and Analysis IssiuesJorField Settings, Chicago: Rand McNally. Cox, D.R. (1958), Planning of Experiments, New York: John Wiley & Sons. Dixon, WilfredJ., and Brown, MortonB. (eds.) (1979), Biomedical ComputerProgramsP-Series, Berkeley:Universityof California Press. Morrison,Donald F. (1976), MultivariateStatistical Methods (2nd ed.), New York: McGraw-HillBook Co. Neter, John, and Wasserman, William (1974), Applied Linear StatisticalModels, Homewood, Ill.: Richard D. Irwin. Rikkers,Layton F., Rudman, Daniel, Galambos, JohnT., Fulenwider, J. Timothy, Milliken, William J., Kutner, Michael H., Smith,Robert B., Salam, AtefA., Sones, Peter J., and Warren, W. Dean (1978), "A Randomized, ControlledTrial of the Distal Spenorenal Shunt," Annals of Surgery,188, 271-282. Steel, Robert G.D., and Torrie, James H. (1980), Principles anld Procediures of Statistics (2nd ed.), New York: McGraw-Hill Book Co. Winer, B.J. (1971), Statistical Principles in ExperimentalDesign (2nd ed.), New York: McGraw-HillBook Co. (? The AmericanStatistician,November 1980, Vol. 34, No. 4