SDASA One-Day Conference
Best Practices in Statistical Consulting
October 16, 2013
Colleen Kelly, Ph.D.
Principal Statistician
What are the attributes of the ideal
consultant?
Condensed from the list prepared by the ASA Committee to
examine the training needs of statisticians (1980):
• Well trained in theory and practice of statistics
• generalist rather than specialist
• Good oral and written communication skills
• Can work within the constraints of the real world
• Team player
What situations do we want to avoid?
Examples of Dissatisfied Clients: (from Derr 2000)
“The experience was unacceptable to me . . . [Consultant] was
consistently late to meetings – sometimes not showing up at all. .
.[recently] he did not return any email and dropped out of
sight?!”
“The time frame in waiting . . . was longer than expected”
“I received poor advice. I did not know it at the time . . . but I
ended up changing the model [after consulting with someone
else].”
What situations do we want to avoid?
1. Not delivering results in a timely manner/not meeting an
important deadline
2. Sending an unexpectedly large bill
3. Errors of the third kind in statistical consulting (Kimball 1957):
“Solving precisely the wrong problem”
Errors of the third kind
What the client wanted
What the consultant
understood
The solution conceived
by the consultant
Errors of the third kind
How the client was billed
The solution actually
delivered
What the client really
needed
Tips for Successful Consulting
1. Plan for good communication
2. Be a good communicator
3. Check that you have communicated what you wanted to.
Tips for a Successful First Meeting
1. Plan for good communication
• Know your client
• Be interested in their problem
• Have a list of questions that you need answered
2. Be a good communicator
• Listen
• Ask questions
3. Check that you have communicated what you wanted to.
• Describe to the client your understanding of the problem
• Discuss what tasks you will do and what tasks they will do
• Discuss timeline of project
• A written statement of work (SOW) may be appropriate.
Tips for Study Design: look for ways
to be efficient
Tips for Study Design: look for ways
to be efficient
• Parallel v. cross-over design
• Blocking, pairing, adjusting for covariates (what factors affect
response?)
• More accurate measurements?
• Repeated measurements?
• Reducing drop-outs.
Tips for Study Design:
Sample Size Rules of Thumb
Normally distributed response (van Belle 2002):
1. One arm study (pre- v. post- comparison):
n =8/D
2
(n= 32 for medium effect, n=13 for large effect)
2. Comparison of two groups (i.e. two-sample t-test):
n =16 / D
2
(n=64 for medium effect, n=26 for large effect)
Tips for Study Design:
Sample Size Rules of Thumb
Proportions (van Belle 2002):
1. The rule of threes: Given no observed events in n
trials, an upper 95% bound for the rate of
occurrence is:
3/ n
2. Comparison of two proportions:
n =16(1- p ) / (p 0 - p1 )
2
(n=64 to compare 10% to 30% )
Tips for Study Design:
Other Design Rules of Thumb
Regression (van Belle 2002):
1. Using continuous variables is generally preferable
(i.e. more powerful) than dichotomizing them.
2. Multiple logistic regression:
•
You need at least 10 events per variable for
stable regression coefficients.
Tips for Statistical Modeling
•
Educate yourself on the science involved (Box 1976 argues
that the best applied statisticians are more appropriately
called scientists).
•
Your solution should be as simple as possible, but not overly
so.
• Choice of a statistical model:
“Since all models are wrong, the scientist must be alert to what is
importantly wrong. It is inappropriate to be concerned about mice
when there are tigers abroad.”
- Box (1976)
Tips for Statistical Modeling
The mice: “In nature there never was a normal distribution, there never
was a straight line, yet with normal and linear assumptions, known to
be false, he can often derive results which match, to a useful
approximation, those found in the real world.”
- Box (1976)
The tigers: Violating assumptions of independence (see Hurlbert 1984),
not adjusting for fishing expeditions (see Dmitrienko et al. 2010 and
Wood et al. 2007) and improper analysis of missing data (Little and
Rubin 2002) can result in more serious errors.
The Tigers:
1.Violating assumptions of independence:
Story A: To assess the precision of a new device, a client plotted
the errors (new device - reference device) against the reference
values. The errors were negatively correlated with the reference.
Story B: To assess the precision of a new device, 10 subjects
were measured 15 times (every 2 minutes) with new and
reference devices. A confidence interval for the mean error was
constructed using the 150 measurements.
The Tigers:
2. Not adjusting for fishing expeditions:
A. In clinical trials, there are often multiple response variables
of interest or there may be a number of doses to compare to
placebo.
B. Data mining (searches for biomarkers or genetic signatures of
disease).
The Tigers:
3. Analysis of missing data:
High-Rise Syndrome: A 1987 article in JAVMA reported a
study of 115 New York city cats that fell 2 to 32 floors. On
average, they fell 5.5 floors. 90% survived. Mortality and
injury rates decreased with increasingly large falls.
Somewhat embarrassingly, it wasn’t the scientific
community that questioned the result, it was Ann Landers’
readers:
“Where in the world is the SPCA? Where are the police? Does
anyone really care how a cat lands? Who goes around dropping
cats anyway?”
Deadly Statistical Sins
Tips for a Successful Presentation of
the Solution
1. Plan for good communication
• Write an organized report
• Write an organized powerpoint presentation
• Explain the results in the client’s language.
2. Be a good communicator
• Listen
• Ask questions
• Avoid statistical jargon, or explain it
3. Check that you have communicated what you wanted to.
Other Tips
• Manage the client’s expectations with regard to time, cost and
deliverables
• Be aware of the client’s timeline and budget
•
Be aware of regulations for the analysis
• developmental or pilot study
• confirmatory or pivotal study
• FDA submission?
• Pharmaceutical v. device v. diagnostic
Conclusions:
1. Communicate, communicate, communicate!
2. These are goals to strive for – when we fall short of the “ideal
consultant”, we can take it as a opportunity to learn how to
do better.
3. The rewards of consulting are to continually learn new
science and statistical methods and to make contributions to
science.
References:
Box G. 1976. Science and Statistics. JAMA 71: 791-789.
Cabrera J. and McDougall A. 2002. Statistical Consulting. Springer-Verlag, New
York, NY.
Dmitrienko A., Tamhane A. and Bretz F. 2010. Multiple Testing Problems in
Pharmaceutical Statistics. CRC Press, Boca Raton FL
Derr J. 2000. Statistical Consulting: A Guide to Effective Communication. Brooks
Cole, Canada.
Hurlbert S. 1984. Pseudoreplication and the Design of Ecological Field
Experiments. Ecological Monographs 54:187-211.
Kimball A.W. 1957. Errors of the Third Kind in Statistical Consulting. JAMA 52:
133-142.
Little R. and Rubin D. 2002. Statistical Analysis with Missing Data, Second
Edition. John Wiley and Sons, Inc. Hoboken NJ
Van Belle G. 2002. Statistical Rules of Thumb. John Wiley, New York, NY.
For More Information
Contact:
Colleen Kelly
Kelly Statistical Consulting, Inc.
760-846-6763
www.kellystatisticalconsulting.com
kstat.consulting@gmail.com
Types of Consulting
Academic
Consulting
Industry
Consulting
Expert Witness
Testimony
Type of client
More statistically
sophisticated
Less statistically
sophisticated
Least statistically
sophisticated
Deadlines
Longer and more
flexible
Shorter and less
flexible
Short and
inflexible (but
erratic)
Pay
Low
Good
Very good