An Approach to Reviewing Mixed
Methods Literature
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed
Methods Literature
The research was entitled:
The effectiveness of occupational therapy local authority social services' interventions for older people in Great Britain: A critical literature review.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed
Methods Literature: study purpose
Was to identify, from literature since 2000, policy, legislative and organisational factors influencing the delivery, effectiveness and any similarities or differences in social care occupational therapy in relation to older people’s ability to remain living in their communities with or without appropriate packages of care in England, Scotland and
Wales.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed
Methods Literature
(Mountain 2000)
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed
Methods Literature
In this study, the literature investigated was:
 Published reports and evaluations
 Current research activity
 Grey literature such as local authority service evaluations and unpublished dissertations from January 2000 to February 2012
Dr G. Boniface & Dr J Riley 04/12

Literature related to:



Older people living in the community
Local authority based occupational therapy services
Local authority occupational therapists’ interventions
Excluding literature which was:

 published before 2000 related to non occupational therapy provision
 solely related to services outside Great Britain
Duplications were also obviously excluded
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Theoretical approach
Was pragmatic and flexible and took into account the historical social, political and organisational contexts. This was necessary due to the diverse nature of the types of evidence (Pope et al 2007,
SCIE 2010).
Thus a traditional systematic review approach was deemed to not be inclusive enough for this type of study.
Dr G. Boniface & Dr J Riley 04/12

The search strategy was shaped by the research questions and the team members’ social care experience and was based in:


Electronic databases
Library holdings


Websites (particularly important for policies)
Using occupational therapy networks to locate local reports and evaluations
Initial search terms were expanded as the review progressed.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: data management
1.
2.
3.
Endnote and Word were used to record included and excluded studies and other material used in the review.
Reasons for inclusion and exclusion were recorded on a summary sheet.
Appraisal results were copied into an overview document detailing the reference, aim of the paper, key findings and implications for practice and methodological quality (Petticrew and
Roberts 2006).
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Appraisal Process
Templates
A range of templates were piloted by the two reviewers on different types of literature to ascertain their suitability such as CASP (University of Oxford 2005), Long et al
(2002) Long and Godfrey (2004). This entire process was also influenced by the SCIE guidelines (2010).
Following this pilot, a new customised review template
(based on Long and Godfrey 2004) was created which included an assessment of the contextual and policy implications of the literature being appraised in more detail.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Data evaluation
1.
2.
3.
4.
Literature was screened and assessed for relevance by two reviewers independently via scan reading the literature rather than simply relying on abstracts.
The new customised template was used by the two reviewers independently to appraise the literature in terms of content, implications for practice and policy as well as the literature’s quality.
The independent appraisals were discussed and the results were recorded on a consensus form.
This was used to populate the overview document detailing the reference, aim of the paper, key findings and implications for practice and methodological quality
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: research team collaborator involvement
This project included three Cardiff University researchers, three service provider collaborators (one from England,
Scotland and Wales) and a service user/carer in the wider research team. In order to address the following:
1.
An expected methodological outcome to be gained from engaging occupational therapists in the development of a process of critical analysis of literature, which will benefit services in the future
2.
The involvement of a service user/carer in the project to assist in identifying search terms, refining appraisal templates and in dissemination of the findings.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: research team collaborator involvement in team meetings
This collaborative group held three team meetings during the project’s life as follows to:
1.
Review the search results, refine terms and revisit the project’s aims.
2.
3.
Evaluate the review procedures and identify any problems or issues arising and discuss the procedures for data synthesis.
This final meeting discussed and evaluated the project’s progress in relation to current practice, policy and organisational drivers and decided upon the structure of the final report and methods of dissemination.
Two collaborators also reviewed papers (with the second researcher) which had been written by the principal investigator.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Data analysis
The data in this project were the consensus key findings and the implications for practice and policy, not the literature itself. These were recorded in the overview document once consensus had been reached and a view on their reliability taken based on the quality assessment.
At this point data was descriptive and needed to be analysed
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Data analysis
Data analysis took the form of a thematic synthesis, which followed the stages described by Thomas and Harden
(2008). The findings from included studies entered into the overview document were independently coded by the two reviewers for meaning and content. Line by line coding allowed for the 'translation' of concepts from one study to another. Codes from each study were added to a coding framework and new ones developed as necessary. During this stage, the reviewers checked for consistency in interpretation and patterns and descriptive themes were identified. A clear audit trail was also created in order for cross checking of analysis to occur.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Data analysis
Thus the data analysis resulted in an aggregative synthesis that summarised and described the data under key concepts (Thomas and Harden 2008; Dixon-Woods et al
2005, 2006) but remained close to the original findings of included studies. The next stage moved beyond this, through further interpretation, using the descriptive themes to address the review questions and develop more analytical themes. Each reviewer did this independently and then together. The process was repeated until new themes were sufficiently refined
(Thomas and Harden, 2008: 3).
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Quality Appraisal Process
In this review, no data was initially excluded on the basis of its quality alone.
Instead, the customised appraisal template was used to assess the quality of the wider range of published, unpublished, grey literature and reports identified. The following quality issues were considered at this point:
 Rigour



Credibility and reliability of findings
Relevance (to the aims of the study)
The extent to which the setting was described (as local issues were found to be paramount).
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Quality Findings




Most occupational therapy related research based literature reported on small scale and localised studies (in both qualitative and quantitative ways).
Larger scale studies tended to be on more general issues e.g reablement rather than on Occupational Therapy alone, making it difficult to disaggregate occupational therapy from general services.
The few pragmatic randomised control trials compared services within and between local authorities, but tended to have unrealistic time scales for occupational therapy intervention.
The unpublished evaluation and service reports were very varied in quality.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Quality Findings
The main weaknesses identified in the literature reviewed were:



A lack of detail on the context or study location
Limited description of participants or explanation of the service, team or intervention
A lack of reflexivity where the researcher appeared to be part of the settings.
Dr G. Boniface & Dr J Riley 04/12

An Approach to Reviewing Mixed Methods
Literature: Conclusion
The researchers in this review knew from the outset that if they had carried out a systematic review as is common, that much of the literature located would have been rejected on grounds of quality.
Therefore, in order to get at and evaluate the findings from a wide range of occupational therapy literature, a different approach to the review needed to be taken.
Thus a customised template was developed, which included a quality judgement, but offered a way of interpreting that quality that did not lead to an exclusion of the findings.
Dr G. Boniface & Dr J Riley 04/12

References






Dixon-Woods, M. et al. (2005) Synthesising qualitative and quantitative evidence: A review of possible methods, Journal of Health Service Research and Policy, 10(1),
45-53.
Dixon-Woods, M. et al. (2006) How can systematic reviews incorporate qualitative research? A critical perspective, Qualitative Research, 6(1) 27-44.
Long, AF. et al. (2002) Developing Evidence Based Social Care Policy and Practice.
Part 3: Feasibility of Undertaking Systematic Reviews in Social Care, Leeds, Nuffield
Institute for Health.
Long, AF., Godfrey, M. (2004) An evaluation tool to assess the quality of qualitative research studies, International Journal of research Methodology, 7 (2), 181-196.
Mountain, G. (2000) Occupational therapy in social services departments, a review of the literature, London, College of Occupational Therapists.
Petticrew, M. , Roberts, H. (2006) Systematic reviews in the social sciences: A practical guide, Oxford, Blackwell.
Dr G. Boniface & Dr J Riley 04/12

References




Pope, C. et al. (2007) Synthesizing qualitative and quantitative health evidence: A guide to methods, Berkshire, Open University Press.
Social Care Institute for Excellence (SCIE). (2010) SCIE systematic research reviews: guidelines (2nd edition), London, Social Care Institute for Excellence.
Available at: http://www.scie.org.uk. Accessed on: 1/8/11.
Thomas, AT., Harden, JH. (2008) Methods for the thematic synthesis of qualitative research in systematic reviews, BMC Medical research Methodology, 8(45).
University of Oxford. (2005) Critical appraisal skills programme: making sense of the evidence, Oxford, University of Oxford.
Dr G. Boniface & Dr J Riley 04/12