Review 1
Major compulsory revisions
1. The overall grammar of the paper needs some major overhaul. It is suggested
that the authors hire a grammar editor to help with the overhaul. Also, headers
need to be corrected (grammar). For example “Category Importance Per Each
Responder” has some tautology – “Per each”.
2. There is need to present the inconsistency values for the local and global
weights in Table 2. These values should be discussed in relation to the
acceptable threshold (Saaty 1980).
3. The Spearman’s correlation table (Table 3) seems to lack meaningful
information.
We believe that we’ve now corrected the grammatical errors.
Thank you for this suggestion. Now it is reported in all the tables that all the
responders achieved a CR<0.1. It is also stated that this is the acceptable
threshold suggested by Saaty.
The homogeneity of correlations was tested calculating the matrix of p-values
for testing the hypothesis of no correlation against the alternative that there is a
nonzero correlation. Each element of this matrix is the p-value for the
corresponding element of RHO. If the p-value(i, j) is less than 0.05, then the
correlation RHO(i, j) is significantly different from zero. This means that there
may be a correlation between responder-i and responder-j, which will be
proportional to RHO.
We have now explained this better in method section 2.9, which has been
completely re-edited.
I am concerned about the homogeneity of correlations 1(<0.5).
In Tables 3 and 4, the values in bracket are the p-values, as explained in
section 2.9. In tables 3 and 4, only the significant correlations are reported
(those with p-value <0.05, or 0.01 when achieved), as the RHO with p-value
greater that this threshold are not statistically significant.
In none of the cells in Tables 3 and 4 is there a number 0.5 in bracket. We have
now inserted a third digit (the zero to left of the decimal point) to avoid any
possible misunderstanding by the reader.
4.
The authors need to use a more robust consensus indices
[See Uzoka F.M.E (2008): A fuzzy-enhanced multicriteria decision
analysis model for evaluating university Academics ’ research output,
Information Knowledge Systems Management (2008), Volume: 7, Pages:
273-299].
Expert Choice software can produce both the consistency of rating and group
consensus.
We were not looking for consensus. We were looking for correlation among
users prioritizations. Therefore, we used the Spearman rank correlation, which
is a method to measure mathematically if two sets of elements are in the same
order. We stated clearly this in section 2.9.
5.
There is the need to present and discuss the respondents’ demographics. Is
there any correlation between the results and the age, gender, respondent’s
experience, size of hospital, etc. It is not just adequate to mention that the
respondents had over 20 years experience in medical practice.
We have now reported this information in the methods section (section2.11,
Responders), but unfortunately, 5 responders is insufficient for us to draw
conclusions based on demographics.
We have stated this as a possible limit of this study (page 22, rows 6-10).
6.
The questionnaire is organized around a five point scale (much less, less,
equally, more, much more). How does this translate to the Saaty’s scale, which
Many authors applying AHP in medical decision making, especially when
patients and lay users are involved, reduced the scale (i.e. see additional
We have now added additional text to the introduction (“However … devices”,
page 8 rows 1-5).
has eight points on each side of the ‘equal importance’ rating?
material of Ijzerman 2012*)
Moreover, this issue was presented and widely discussed in previous papers
presented to the International Symposium on AHP (ISAHP) in Pittsburgh
(2009) and in Sorrento (2011). These papers are now cited in section 2.3,
where a brief paragraph explains why this 5-point scale was chosen and how
this can be translated onto the Saaty 9-points scale.
*Ijzerman MJ, van Til JA, Bridges JF: A comparison of analytic hierarchy
process and conjoint analysis methods in assessing treatment alternatives for
stroke rehabilitation. The patient 2012, 5(1):45-56.
7.
I am sure that this study has a number of limitations and policy implications
for CT Scanner manufacturers.
The authors should address these in the Conclusion section.
Minor Essential Revisions
Figure 2 is not the entire questionnaire and should not be labelled as such. The
questionnaire should be presented in the Appendix instead of inside the body
of the work.
9. Re-write the description of the section “Correlation among Responders’
Preferences” to clarify if you are referring to item consistencies or respondents
consensus.
8.
We have now added a sentences in the conclusion section ‘These results also
have … during the development of new devices’
We specified that the Figure 2 aim to illustrate the questionnaire layout. This is
now stated clearly in Figure caption.
We thank the reviewer for the suggestion and recognize that this was a
confusing point. We have now rewritten this section, emphasising that we aim
to investigate differences between user needs, stratifying clinicians according
to specialization and intervention.
10.
The Consistency Estimation should indicate the consistency threshold (from
literature)
We inserted this threshold in section 2.6 and reported explicitly in Tables 1 and
2.
11.
12.
Number the sections to better organize the work and enhance readability.
The “Judgment Matrix” section contains the number 251658240. What is the
meaning of the number?
We numbered the sections.
These numbers were inserted in error and we thank the reviewer for noting
this.
The second paragraph of the section on “Relative Importance of Needs into
Each Category” is confusing. Please, re-phrase the entire paragraph. Also, the
section title is grammatically wrong.
14. The Lit review portion needs to highlight some of the existing multi-criteria
decision methods and the superiority of AHP to such methods. Please, see
section 3 of Uzoka et al (2011);An Experimental Comparison of Fuzzy
Logic and Analytic Hierarchy Process for Medical Decision Support
Systems. Computer Methods and Programs in Biomedicine, 103(1), 10-27.
15. The line (Introduction section) that says “In this study, the AHP method was
applied to elicit user needs of clinicians using a complex medical device” is
13.
We have now rewritten this section.
We mentioned in the introduction section some studies that support the
application of AHP in comparison with other methods (“Other methods…
healthcare decision-making”, from page 6 row 22 to page 7, row 21)
It is the latter, we modified this line to clarify.
confusing. Do you mean the AHP is a complex medical device or the clinician
use a complex medical device?
16. The section on “Hierarchy Definition” mentions a focus group that evaluated
the ‘Tree of Needs’. What is the composition of the focus group? What was
the outcome of such evaluation?
17.
We added the following description:
“This focus group involved 4 medical doctors in charge of the units, of which 2
are co-authors of this paper (AR and AS), 3 biomedical engineers with
extensive experience of the design, assessment and management of medical
devices, of which 2 are co-authors of this paper (LP and LM) and 1 clinical
engineer of the hospital. This group identified 12 needs, based on their
personal experience and the pertinent scientific literature, and organized them
into meaningful categories. LP acted as the facilitator and, based on his
experience of AHP, designed the hierarchy, which was then reviewed with the
other participants to check that it was accurate and comprehensive.”
We modified this reference.
19.
20.
We have structured the paper according to the journal’s guide to authors,
which stipulates the section titles: Background; Methods; Results; Discussion;
Conclusions.
The author guidelines ask for this section to be in the methodology.
We considered this discretionary suggestion out of the scope of this paper.
Reference number 22 has some missing components, which should be fixed.
Discretionary Revisions
18. The introduction section could be split into Introduction and lit review. It is
currently not well presented.
I would prefer “Ethical Considerations” not to be part of the methodology.
It would be nice to see what the aggregated group results look like and be able
to compare this with the individual results (for each respondent) to determine
deviations from the aggregate
Reviever 2
The application of the AHP method to the case study of purchasing a CT
scanner – measured from multiple perspective – is an interesting one and on its
face value, would be relevant to this journal. This said, there are five (5)
serious flaws in the paper that make it unacceptable for immediate publication.
Each of these can be views as compulsory revisions or as serious barriers to
publication.
22. The authors mix notions of subjective needs to guide the development of
future technologies (a separate, but increasingly important, literature) with the
deliberative process behind the decision to purchase a CT.
21.
This causes much ambiguity within the paper, and at time serious
inconsistencies.
We especially thank the reviewer for this comment, which gave us the
opportunity to improve the whole paper.
Actually, we used AHP because we are interested in both aspects. At hospital
level, the majority of studies applied AHP for prioritizing attributes and
performances alternative of alternative technologies for selection and
purchasing. Nonetheless, this aspect has a growing impact on medical devices
development. In the last year, device life-cycle has been strongly reduced and
the R&D divisions of company producing devices work closer than ever with
Marketing ones.
This lead us to choose AHP, which proved to be satisfactory in user need
elicitation for device development, but was previously widely used for device
purchasing (“In our elicitation.. device purchasing”, page 7, rows 11-13). We
state this clearly in the paper now, and chanced the papers around to reduce
confusions.
23.
AHP is an important and practical tool for the explication of decision making
weights across criteria (and potentially subjective performance). Its main
purpose is to facilitate the delivery of a decision. The authors use it primarily
to study the importance weights.
If this was the primary objective, then other research methods such as conjoint
analysis (both compositional and decompositional), discrete choice experiment
or best-worst scaling would have provided a more direct and theoretically
grounded method to study what is essentially attribute importance.
Many studies compared AHP with other methods for user need elicitation.
Some authors (see references 35, 36, 37) concluded that AHP is slightly better
than other methods when complex decision (i.e. more than such a number of
attributes) have to be considered. Others argued that AHP may increase
inconsistencies (reference 38). We described widely the state of the art in the
introduction now.
Regarding our chose to use AHP (and not other methods typically used of user
need elicitation) to prioritize user need, although no alternative had to be
selected, we did this because the study was conducted at hospital level, and
AHP was previously used to devices selection at this level.
Thus, we believe that the use of AHP for user need elicitation may facilitate
internal (clinicians with clinical engineers and top management) and external
communication (clinical with device designers). We explained better this in the
introduction section (“In our elicitation … decision-making”, page 7, rows 1121).
24.
AHP is normally used within a group decision making process.
This normally requires that the group meet to compare and discuss their
weights and implied decisions as a means to develop consensus around group
weights and, more importantly, a group decision.
The authors appear to have conducted five N=1 studies as a means to compare
the results, but it is concerning that these individual results lack reliability and
generalizability.
25.
The authors have a great deal of discussion and present a lot of material
(mainly methodological) that really is not linked to the core thesis of the paper.
Furthermore, the authors have very shallow discussions of vital information
Although aware about this standard procedure, we were not interested in
achieving consensus. The goal of this study was to explore the differences
between user needs, stratifying responders according to clinical specialization
and medical intervention area (elective versus emergency medicine). Thus, we
were interested in AHP applicability to explore differences in categories of
users. We describe wider this aspect in discussion section.
We introduced this difference in section 1 background (“AHP is normally
used… medical devices”, from page7, row 21, to page 8, row 5) and dicsussde
in section 4, discussion (“it should be noted… clinical specialization”, page 20,
rows 15-20).
Generality of results achieved are hard to achieve considering their dependency
from specific hospital assets. However, we state this as a limit of the study in
this paper. (page 22, rows 8-10).
Reviewer one appears to be satisfied with the methodology and in several
places has requested more information. We therefore believe that we have the
correct balance in terms of information for the reader
(such as their discussion with other clinicians to examine the face validity of
the results). I would have liked to have seen some details here, both in terms of
the process of validation and the results (even if they were qualitative).
26.
Many of the important contributions to the literature applying AHP to health
care have been ignored by the authors.
Here I am thinking of the important advance and applications of the method by
James Dolan in the US and some work specifically applying the method to
medical technologies by Maarten Ijzerman in the Netherlands.
Furthermore, there is an emerging literature on the application of conjoint
analysis, discrete choice experiments and best-worst scaling to the study of
needs, priorities and preferences as they relate to health care and medical
technologies.
Editor
Please include a 'Competing interests' section between the
Conclusions and Authors' contributions
28. Please include an Authors' contributions section before the
Acknowledgements and Reference list.
27.
In the submitted version of the manuscript, references 26 and 27 refer to some
important contributions of James Dolan, from USA, while references 24 and
25 are two works of Maarten Ijzerman and Hummel J Marion from
Netherland.
Perhaps the Reviewer refers to studies conducted from these scientists with
other Authors, comparing among AHP and other methods. In this new version
of the manuscript, we gave more space to comparisons among AHP and other
methods for user needs elicitation in medicine. (“Other methods… healthcare
decision-making”, from page 6 row 22 to page 7, row 21)
According to this interpretation, we considered more literature on this topic,
inserting also some more references: from 32 to 38.
We inserted this section.
We inserted this section.