Statistical Consulting
Neil Scott
Medical Statistics Team, University of Aberdeen
Overview
1. Experience of operating a statistical consulting service
(Medical Statistics Team, University of Aberdeen)
2. Review of relevant literature:
• Not systematic
• Considers any type of statistical consultancy (not just medical
statistics)
3. General discussion – your experiences with statistical
consulting
Carter et al. (1986)
• Unique role of statistics: both a science and a part of the
methodology for all scientific subjects
• Statistical thought is needed at every stage of quantitative
research investigations
• A statistics department may assume academic
responsibility beyond home department
Types of statistical consulting
•
•
•
•
•
Ad hoc advice
Drop-in clinic
Appointment system
Formal agreement to produce data analysis/ report
Overlap with research/teaching roles
Who consults?
•
•
•
•
Students
University staff
NHS staff
External
Statistical Consultancy Service - Medical Statistics Team
•
•
•
•
•
Students (taught course)
• Daily student clinic 1-2pm (nine statisticians contribute to rota)
• May need to queue
PhD students/MSc by research
• Appointment system (supervisor must attend)
University of Aberdeen staff
• Appointment system (first half hour is free, then charged)
NHS Grampian staff
• Appointment system (free advice/analysis unless large amount of
work)
Project collaboration (sometimes begins as consultancy)
Logged consultations (2012)
Number of
logged entries
Mean length (minutes)
Total time (hours)
Student clinic
315
21.8
114
University staff / PhD students
40
50.1
33
NHS staff
97
61.5
99
Total
452
32.8
246
Student consultations by month (2012)
When do clients consult a statistician?
•
Stages of the research process:
•
•
•
•
Planning
• Study design
• Sample size calculation
• Grant application/ethics form
Data collection
• Data management/ setting up a database
Data analysis
• Most common time to seek help
Dissemination
• Publication process
• Responding to reviewer comments
• "To consult a statistician after an experiment is finished is
often merely to ask him to conduct a post mortem
examination. He can perhaps say what the experiment
died of."
R. A. Fisher, in an address to the First Indian Statistical
Congress in 1938
Organisational issues (personal experience)
•
•
•
•
Drop-in student clinic has peaks and troughs of demand
Some clients want immediate appointments/ specific times
Many people are “hybrids” (University staff/ NHS staff/ student)
Occasional demand for remote consultation for staff/students
based off site:
• face-to-face (or phone) preferred to e-mail
Common situations with face-to-face consultations
• Short time scale: “I need to present this next Tuesday”
• Understanding unfamiliar topic area/ clinical jargon overwhelming
• Overcomplicated studies (small sample size, too many variables), e.g.
waste five minutes searching to find correct variable in SPSS
• Repeat clients (regular customers)
• Timing of advice (e.g., if they had come at design stage, the analysis
stage would have been easier)
• Client wants to copy a method used in another published paper: i.e.
“me too” publication
Common situations (when performing data analysis for client)
•
•
•
•
•
Messy data: e.g. Excel spreadsheet with blank spaces / mixture of text and numbers
Hand (2010):
• “90% of data analysis is data cleaning”
• Analysing the data is a cyclic process: interaction with the client
Timescale to complete work or get back to someone – how long is reasonable?
Unclear/vague projects:
• Client has only vague idea of research aims - these may evolve over time
• Unclear about definitions of variables/ exactly which variables to analyse
• Multiple requests for analysis that add up: “Prof Bloggs wants me to add another
three p-values / look at another subgroup”
Clients sending data with identifiable information
Clients outside applied health research
• Laboratory research:
• Designed experiments
• Repeated measurements on small number of samples
• Psychology:
• Everything expressed as effect sizes!
Generalist versus specialist
• Consultant statistician analogous to being a GP
• Questions can be on any area (from trivial to extremely
complex/specialised)
• May have to emphasise you are not expert on every aspect of
statistics:
• OK to:
• Say you don’t know the answer
• Get back to them
• Refer to colleague with appropriate knowledge
Financial terms
•
•
•
Discussing charging can be awkward
Difficult to judge total time on project
Survey of academic consulting units [Niland (1995)]
• Most aim to recover costs
• Some degree of conflict between service and research roles of academic
statistician:
• “To maintain a collegial relationship with investigators, it was suggested that
only time spent in more routine tasks such as data entry and programming be
charged back to investigators, and more conceptual activities such as
proposal design, grant, and manuscript preparation be provided without
charge.”
Statistical Consulting Cycle
1.
2.
3.
4.
5.
[Kennett & Thyregod (2006)]
Problem elicitation
Data collection
Data analysis
Findings formulation
Findings presentation
Statistics courses in academia tend to focus on step 3
Problem elicitation
• Active listening
• “Determine what the client wants to know: may not be what they say
at first (they are not statisticians)” [Hand (2010)]
• First question often is: “How exactly would you describe your research
question?”
• Common question after client gives too much information to take in
at once: “Could you go back a step?”
Kennett & Thyregod (2006)
• “Hunter (1979)…described that two chemists entering his
office, when asked to describe the problem they came to
get advice on, entered a lengthy discussion that eventually
led to a common reformulation of the problem.
• Once they had the problem clearly formulated they knew
how to solve it and did not need the help of a statistical
consultant any more.”
Consultancy skills (American Statistical Association, 2003)
•
•
•
•
•
•
•
•
Knowledge of statistics
Problem solving
Communication skills
Flexible/ can meet deadlines
Patience
Experience of topic area
Understands resource constraints
Shows respect for statistical and research ethics
Hand (2010)
•
•
•
•
“Statistical consulting requires skills and expertise completely distinct
from academic and technical expertise”
“What you need to be a successful consultant:
• Affability
• Availability
• Ability
• in decreasing order of importance!”
“Client relationship is the most important issue”
“Put yourself in the client’s shoes”
Johnson and Warner (2004)
• Studied factors associated with consultation being success
• Used factor analysis/ logistic regression [!]
• Most important factors:
• Statistical/collaboration skills
• Personal qualities
• Client requested particular consultant
Clinician – statistician communication
甲状腺刺激ホルモン受容体遺伝子の生殖細胞突然変異を
引き起こす自己免疫性常染色体優性甲状腺機能亢進症
Bạn đã cố gắng một mô hình hồi quy hoặc một bài kiểm tra
chi bình phương và bạn đã kiểm tra rằng các giả định được
đáp ứng?
Effective communication
• Use appropriate language: ask client about knowledge of statistical
tests / p-values to ensure communicate at correct level
• “Use the language of the customer” [Kennett &Thyregod (2006)]
• “Be aware of the client’s level of technical knowledge
• They won’t have your statistical expertise
• They will know about the problem and the data
• They are paying you
• They did not come in for a lecture on statistics” [Hand (2010)]
Organisational problems of consultancy [Sprent (1970)]
1. Lack of time or background knowledge to become a full member of an experimental
team
2. The statistician may not have adequately adapted outlook to the computer era
3. The statistician may be called in too late or dropped too early
4. There is little or no training in the art of consultancy
5. In some universities little account is taken of the consultancy load in staffing and
successful consultancy is not always a criterion for promotion
6. In research institutes there is a danger that over-specialization may lead to stagnation,
as modern developments may be overlooked
7. There is little informed criticism of consultancy, and many of us are not keen on
practising the art of self-criticism
Client stereotypes
• Hyams (1971), Sprent (1970)
• “Probabilist” – only interested in a significant p-value
• “Sporadic leech” – stops the consultant in the hall for a brief
friendly chat
• “One-technique amateur statistician”
Learning from experience
• Both client and statistician learn from experience of
consultation
• Joliffe (2001)
• The consulting activity of statisticians can only enhance their
teaching
• Wilson (1992)
• Problems encountered in consultancy settings can lead to new
statistical methods
• Carter et al. (1986)
Benefits of consulting
•
•
•
•
Exposure to variety of methods
Real world problem solving
Collaboration with others
Rewarding (helping others/ ensuring quality of
research)
Potential disadvantages
•
•
•
Recognition - consultancy comes under heading of “service”: less prestige than
teaching or research [Gibbons and Freund (1980)]
Not initiating own research
Acknowledgement, authorship or nothing at all?:
• May do a lot of work without any reward
• Concern that do not get to check published material with your name on it!
• Checking a paper thoroughly is time-consuming
• Additional work may be required following referee comments
• May work mainly on small-scale, low-impact research
Russell (2011)
• “Advertisements for University consultants frequently offer
the position at Lecturer level, or even at Research Fellow
level, suggesting that the position is seen as a low-level
role requiring not too much statistical maturity. Nothing
could be further from the truth.”
Discussion
• Your experiences with statistical consulting:
•
•
•
•
•
Demand for consultancy in your institution
Organisation of consultancy services?
Who consults? Staff/students?
Staff time/costs
Etc.
References (1)
•
•
•
•
•
•
•
American Statistical Association (2003). When you consult a statistician … what to
expect
Carter RL, Scheaffer RL, Marks RG (1986). American Statistician, 40(4), 260-264.
Gibbons JD, Freund RT (1980). Organizations for statistical consulting at colleges and
universities. American Statistician, 34(3), 140-145.
Hand DJ (2010). Statistical consultancy.
Hyams L (1971). The practical psychology of biostatistical consultation. Biometrics,
27(1), 201-211.
Johnson HD, Warner DA (2004). Factors relating to the degree to which statistical
consulting clients deem their consulting experiences to be a success. American
Statistician, 58(4), 280-289.
Joliffe F (2001). Learning from experience. In: Batanero C (ed.) Training researchers in
the use of statistics, 355-370.
References (2)
•
•
•
•
•
•
Kennett R, Thyregod P (2006). Aspects of statistical consulting not taught by
academia. Statistica Needlandica, 60(3),396-411.
Kirk RE (1991). Statistical consulting in a university: dealing with people and
other challenges. American Statistician, 45(1), 28-34.
Niland (1995). A survey of biostatistical consulting units throughout North
America. American Statistician, 49(2), 183-189.
Russell (2011). Advice for the potential statistical consultant. Proceedings of
the Fourth Annual ASEARC Conference, Parmatta, Australia.
Sprent (1970). Some problems of statistical consultancy. JRSS(A), 133(2), 139165.
Wilson (1992). Statistical consulting is scholarship. American Statistician,
46(4), 295-298.