Transactions in GIS, 2008, 12(3): 293– 321
Research Article
C
Transactions
TGIS
©
1467-9671
1361-1682
Original
XXX
GIScience
Caron,
2008 The
Articles
S
Journals
Roche,
Authors.
in GISDRanking
Journal
Goyer and
and
compilation
AEvaluation
Jaton © 2008 Blackwell Publishing Ltd
Blackwell
Oxford,
UK
Publishing
Ltd
GIScience Journals Ranking and
Evaluation: An International Delphi Study
Claude Caron
Stéphane Roche
GeoBusiness Group
Université de Sherbrooke
Center for Research in Geomatics
Université Laval
Daniel Goyer
Annick Jaton
CGI – Groupe Affaires électroniques
Montréal (Québec), Canada
Department of Geomatics Sciences
Université Laval
Keywords
Abstract
Researchers’ fame in most scientific fields is closely linked to their publishing
capacity, both in terms of quantity and quality. In GIScience, as in other fields, this
situation demands that the researcher evaluate and to be very familiar with the
scientific journals in which they could publish. Some specialized journals (e.g. Journal
of Citation Reports or JCR) are devoted to ranking these reviews according to various
methods and criteria. Compared to other scientific communities, GIScience is
relatively new and constantly evolving. Therefore, the journals of this field do not
benefit from any real formal ranking yet. The objective of this paper is to present
the process and results of a study aimed at addressing this gap. More specifically,
the challenge is to elaborate an importance ranking of the scientific journals in the
field of GIScience. To do so, both a qualitative (Delphi study carried out with 40
international experts) and a quantitative (JCR impact factor) approach has been
implemented. This triangulation method leads to an early global ranking of the
journals of this field.
1 Introduction
Researchers’ reputation in most scientific fields is closely linked to their publishing
capacity (Hamilton and Ives 1980, Latour 1987, Hardgrave and Walstrom 1997). Their
Address for correspondence: Claude Caron, GeoBusiness Group, Faculté d’administration, Université
de Sherbrooke, Sherbrooke (Québec), Canada, J1K 2R1. E-mail: Claude.Caron@USherbrooke.ca
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
294
C Caron, S Roche, D Goyer and A Jaton
production is an important evaluation criterion during the “tenure track” period, as
well as for potential promotions. This production is evaluated not only on the quantity
of their published works (for instance, the number of papers published per year), but
also on their quality (awards for the best papers, personal invitation to publish, etc.), as
well as on the way these publications impact their area of expertise (particularly the
frequency with which articles are quoted) (Dubois and Reeb 2000). This modus operandi
also implies an evaluation of the scientific journals in which researchers publish their
results. Consequently, in both applied science and humanities, these journals are being
classified according to various criteria, based on various methods. Specialized journals
are dedicated to this task, such as the Journal of Citation Reports (JCR), which is one
of the most respected by the international scientific community.
Numerous research studies have tackled these issues of evaluating and ranking
scientific journals (Garfield 1972, Hamilton and Ives 1980, Tijssen and Van Raan 1990,
Gillensen and Stutz 1991, Hardgrave and Walstrom 1997, Moed et al. 1998, Dubois
and Reeb 2000, Rousseau 2002). These works have facilitated the evaluation of the
specific contribution of each journal regarding the dissemination of research results
within the scientific community, as well as the structuring of communities of practice.
It is now clear that the issue of evaluating and ranking scientific journals is an
important one. Yet, in some relatively new fields, this type of ranking is still quite
informal. This is particularly the case in GIScience. As far as it is known, journals
related to GIScience have not been subject to any formal evaluation. No overall evaluation
exists enabling prioritization of the scientific journals in which researchers from our
community publish their works. These journals, however, do not have the same impact
on the dynamic of this specific field, and on the transfer of knowledge between GIScientists.
Some studies have been carried out in the related field of geography, the most notable
of which are Gatrell and Smith (1984) and Lee and Evans (1984, 1985). These studies
are interesting as some of the journals already ranked in the 1980s are still respected by
researchers in GIScience (Annals of the Association of American Geographers, Environment
and Planning A, Environment and Planning B, for instance).
Without venturing into any epistemological analysis of GIScience, we assume that
there is de facto a community of specialists and scientists who do care about the issues
related to the formalization, analysis and use of geographical concepts and data (e.g.
University Consortium for Geographic Information Science (UCGIS), Association of
Geographic Information Laboratories for Europe (AGILE)). The GIScience community
has been developing for more than a decade (Goodchild 1992, Hendriks 1998, Ottens
1999, Caron 2000, Fryrear et al. 2001, Murayama 2001, Gatrell 2003). According to
Bruno Latour’s (1987) analysis, GIScience as described in Goodchild et al. (1999) is
more and more considered to be a special field. Therefore, on the basis of this very
active community, we wish to identify, then evaluate and rank the journals considered
as being related to GIScience by the practitioners of the field.
Evaluating the specialized journals in GIScience is highly relevant because it will
give a clearer picture of how papers contribute to the advancement of knowledge
(Gatrell and Smith 1984). This evaluation will also identify trends that lead to the
development of research policies (Wilson 1985), and that characterize the scientific
community (identification of the sub-domains, possible differentiation of the researchers’
behaviors according to their type and age, etc.) (Lanegran 1992). The improvement of
the comprehensive knowledge about the GIScience literature also offers to junior
researchers, early in their careers, the possibility to identify which journals are the most
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
295
relevant to publish in. Furthermore, depending on their publishing strategy, this knowledge
allows them to target selective and prestigious journals or ones that are not too demanding
(Hamilton and Ives 1980, Dubois and Reeb 2000).
It is with a view to improving the knowledge of the GIScience-specialized literature
that we have carried out this study. The objective is to offer a preliminary formal
ranking of the scientific journals in GIScience. To do so, we chose a methodological
approach focusing on both quality and quantity. First of all, a Delphi study1, qualitative
and consensual by nature, was carried out on nearly 40 international experts of the
field. Then, the results obtained were consolidated, using triangulation and comparison,
with quantitative approaches (particularly the JRC impact factor).
First we develop a contextual framework by providing an update on the major
scientific journal ranking schemes. Then, we explain the methodological approach, and
more particularly the implementation parameters of the Delphi study. Finally, we
present the results and the consequent ranking. We conclude with the limits and
perspectives of this research.
2 Research Conceptual Framework
2.1 GIScience in Action
Before building up a ranking system of the GIScience journals, it is necessary to identify
all the journals in this area. Therefore, a major precondition is to define the current
boundaries of GIScience. Yet, this is a very difficult task since, reciprocally, the decision
of considering whether or not each journal is a part of GIScience and contributes to the
very definition of the field. Bruno Latour’s works offer in this context an interesting
framework (Latour 1987). He believes that publications and journals fully participate
in the actors’ network, on which scientific fields develop. Consequently, whatever the
difficulty, proposing a reading and understanding grid of this actors’ network means
contributing to a better understanding and definition of GIScience.
What does GIS mean? GIS tools, systems or science? : This is the type of questions
on which the scientific community, involved in geographic information research, has
been focusing for over a decade (Chrisman 1999, Pickles 1997, Wright et al. 1997, Mark
2000, Schuurman 2000). Our purpose is not to continue this epistemological debate, but
rather to emphasize how much this community is alive, how much this new “discipline”
or “science” is going through a very rich development period. Whether this dynamism
confers on this discipline the status of science or not is hard to say, but it surely highlights
the researchers’ strong will to formalize its basic concepts and to define its boundaries.
The last decade has been particularly rich with discussions that fostered the
progressive emergence of a (more or less) shared vision of GIScience. Numerous forms
of research, and numerous positions have been proposed since the Spatial Data
Handling conference keynotes presented by Michael Goodchild in Zurich, July 1990,
and then in Brussels, April 1991, during the EGIS – European GIS conference (Schuurman
1999). In 1991, Jean-Paul Cheylan also developed the GIScience concept in an editorial
of the Revue internationale de géomatique (previously Sciences de l’information
géographique et de l’analyse spatiale). In Canada, the 1980s and the 1990s marked the
conceptualization of the field of geomatics (Paradis 1981), and many other publications
followed (e.g. Gagnon and Coleman 1990, Gagnon and Bédard 1994). In 1993, the
Office de la langue française du Québec (Canada) defined geomatics as the:
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
296
C Caron, S Roche, D Goyer and A Jaton
Discipline dedicated to the management of spatially referenced data, and thus
relies on the scientific concepts and technologies implied in the acquisition,
storage, analysis, and distribution of the data (Bergeron 1993).
The first attempts to formalize GIScience plead for their recognition as a full
scientific discipline. This reasoning emerged out of a strong distinction between GIScience
and GIS software, and went even beyond the debate “tools versus system” that
animated the community at the beginning of the 1990s. Michael Goodchild (1992) gave
no formal definition of GIScience, but he clarified the scope of this field by identifying
its main characteristics. One of the first definitions was proposed by Frank (1992):
GIScience addresses the spatial representation of the data allocated to represent a part
of the Earth’s surface.
The 1990s provided a wealth of thoughts and debates. The book coordinated by
John Pickles (1995) is, in this regard, a significant contribution to the reflections on
which GIScience developed. A few more papers on that subject were published during
this period (Pickles 1997, Wright et al. 1997). The “critical GIS” stream arose around
the same time, and was widely supported by the publications (articles and research
reports) produced through the National Center for Geographic Information Analysis’s
(NCGIA’s) Research Initiative 19. The Varenius project, which followed the NCGIA’s
initiatives, began with the publication of a series of papers in the International Journal
of Geographic Information Science (the “S” was then no more for system but for
sciences, as was the case for other journals that were renamed as the GIScience concept
emerged). In the introduction of the Varenius project, Goodchild et al. (1999) devoted
a large part of their article to marking the boundaries of GIScience, to define, but also
to justify their relevance to scientific, societal and technological influences.
GIScience deals with the fundamental issues resulting from the creation, processing,
storing, representation, distribution and use of geographical information. That is to say,
and as Goodchild et al. (1999) also specified it, the terms geomatics, geoinformatics,
spatial information sciences can be considered as synonymous. A recent definition
posted on the Geographic Information Science Center at the University of CaliforniaBerkeley website (see http://www.gisc.berkeley.edu/about/whatisgis.html for additional
details) supports this statement:
GIScience has recently emerged as the field where problems of data capture,
encoding, storage, analysis, retrieval, synthesis, and dissemination of geographic
information are studied. These problems have increased in significance as new
computer-based technologies such as geographic information systems (GIS),
digital remote sensing (RS) and Global Positioning Systems (GPS) evolve.
GIScience is the foundation upon which these GI technologies are built.
Inspired by the above, we marked the boundaries of GIScience as they have developed
for the last two decades, and as they are considered today. We decided to adopt this
formal and operational definition of GIScience through all the current study. On this
basis, we constituted the initial corpus of journals for the Delphi study (section 3).
The previous timeline briefly emphasizes the dynamics of GIScience development
and underlines the significant role of publications and journals in the formalization of
this scientific discipline. They are crucial in communicating ideas within communities of
scientists, because of the concepts and ideas they explicitly formalize through their
names (which often change as the discipline evolves), and finally because of the way they
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
297
are recognized. GIScience emerged and developed mainly in and through communications
between scientists, more formally through publications and journals.
Apart from any formal ranking, all GIScience journals do not count equally. There
are differences, especially between peer reviewed journals (such as Transaction in GIS)
and non-reviewed journals (such as GIM Magazine). Among peer reviewed journals,
some score significantly higher than other ones. This is the reason why the GIScience
journal evaluation and ranking methodology chosen here is so important, not only for
the development of this still relatively new field, but also as an active component of
GIScience in action.
2.2 Journal Ranking Methodologies
The ranking approach elaborated here required the development of a conceptual framework, built on the basis of an analysis carried out to identify the main priority-ranking
methods for scientific journals. The criteria being used are variable: some are quantitative,
some are qualitative; some rely on statistical or bibliometric methods, or on qualitative
evaluation processes. Singleton (1976), in his work dealing with the ranking of physical
sciences journals, proposed bringing all of these methods together into three major
approaches: the analysis of journal article citations; scientists’ assessment of journals in
their field of study; and the evaluation of the size or productivity of the journals.
The first approach adopts the analytical techniques dedicated to the citations
published in journal articles. The “impact factor” of the Journal of Citations Report
(JCR) is surely the most common quantitative evaluation process for citations (Rousseau
2002). This approach made clear the fact that the quality of scientific journals is linked
to the impact factor, a term created and used for the first time in 1963 by Eugene
Garfield, founder of the Science Citation Index at the beginning of the 1960s. Amin and
Mabe (2000) underlined that the value of the impact factor is directly affected by the
research field at stake, the type and size of the journal, and may fluctuate over time for
some journals (Gillenson and Stutz 1991). Amin and Mabe (2000) also mentioned that
the “impact factor” is not only a means to describe the impact of journals on the scientific
literature of a specific field, but it is a strong measurement process for the ranking of
scientific journals, yet this type of indicator should only be used independently for each
discipline. Some secondary journals in one specific field may be primary journals in
another field. This type of approach was developed by Gatrell and Smith (1984) to
assess the relationships between 22 geography journals. In this study, the authors
calculated how many times journal articles were cited by other journals.
The second methodological approach relies on the researchers’ judgement and
appraisal (often qualitative) of journals in their own field. It may have to do with the
“freshness” or mean elapsed time between the submission of an article and its publication,
or with the calibre and recognition of the members appointed to the peer reviewers or
an editorial review board. It may also be related to the review process itself (anonymous
or not), or with the calibre of the journal. Hamilton and Ives (1980), Gillenson and
Stutz (1991) and Walstrom et al. (1995) focused on assigning a rating to various journals
with the help of a numerical scale. For instance, Hamilton and Ives (1980) asked
experts in information system management to systematically score the relative importance
of journals in related fields. Their results revealed that the experts’ perception is
very important. Katerattanakul et al. (2003) went even further, asking academics to
give their opinion about the relative value of computer science journals via a series of
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
298
C Caron, S Roche, D Goyer and A Jaton
questionnaires. Formal techniques have also been developed to assess researchers’
perceptions towards journals in their field of study, for instance: Koong and Weistroffer
(1989) in management information systems (MIS) and Dubois and Reeb (2000) in international business. Following this approach, Sutter and Kocher (2001) measured, over
10 years, the fluctuating recognition of economics journals so as to establish an overall
ranking through time.
The third methodological approach relies on the quantitative evaluation of the size
or productivity of journals. Many indicators are used: number of readers, number of
articles submitted, number of articles accepted, number of issues sold (or delivered),
diversity of the subject fields covered, etc. For instance, Todorov and Glaenzel (1990)
developed a journal ranking method on the basis of computerized bibliometric
techniques. The journals were organized by speciality areas, relying on the measures of
dispersion (or of concentration) of the subjects treated. Closer to our own field, Lee and
Evans (1984, 1985) used this approach to rank 34 geography journals according to the
number of issues delivered. In this regard, Lanegran (1992) mentioned that “The growth
in the number and circulation of geographical journals has led to speculation about their
prestige and impact”.
So, in many disciplines, studies have been carried out to establish scientific journal
operational rankings. In disciplines closely related to GIScience, issues relating to journal
ranking have also been tackled. For instance, Brancheau et al. (1996) determined the
critical issues with regard to information systems. Sabrin (2002) developed a classification
of the academic institutions offering information systems training in the United States.
Walstrom et al. (1995) dealt with the ranking of information systems conferences. Yet,
nothing of the kind formally exists in GIScience.
Granting agencies, or research institutions themselves have a true interest in possessing
such a ranking system, since publications still constitute today one of the major criteria
to evaluate researchers (obtaining grants, tenure and promotion, etc.). For instance, the
French Centre National de la Recherche Scientifique (CNRS) launched in 2004 an
action to assess the place and influence of French journals dedicated to human and
social sciences at the international level. An inquiry based on the impact factor principle
was then started. Finally, 18 disciplines were chosen, and the journals were analysed
between 1992 and 2001. This analysis ranked 125 journals in three categories: (A) very
highly rated international journals, (B) highly rated international journals and (C)
national audience journals. This ranking is now a basis for the evaluation of researchers
and French laboratories.
3 Description of Methodological Approaches
We believe that an intersectional approach could strengthen the ultimate outcome;
consequently we have opted for a methodology integrating the first and second approach.
The first step is mainly based on the Delphi method (Loo 2002). The rationale for
choosing this qualitative and consensual method was two-fold: the appropriateness of
being directly in contact with the researchers in a still young and developing field;
and the lack of quantitative data relating to GIScience journals (with particularly few
journals mentioned in the JCR). This step allowed us to generate a consensus as well as
an initial ranking of GIScience journals, so as to better interpret the results obtained
with the Delphi.
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
299
3.1 The Delphi Method
The Delphi method has been the subject of many studies in various fields: health
(Demeyrick 2003), information systems (Brancheau et al. 1996, Schmidt et al. 2001),
banking (Bradley and Stewart 2003), and marketing (Mitchell and McGoldrick 1994).
Yet, in GIScience, this method of investigation has not been used extensively (Roche et
al. 2003, Péribois et al. 2005). Consequently, before going further, it is useful to review
some of the elements relating to this approach. There are several variants (Linstone and
Turoff 1975, Mitchell and McGoldrick 1994, Brancheau et al. 1996). We have chosen
Loo’s method (2002), which appears to be the more straight forward one. Loo (2002)
identifies four major steps: (1) define the problem; (2) select the participants; (3) determine
the size of the sample; and (4) conduct the iterative surveys of the Delphi method.
Kaynak and Macaulay (1984) underscored the fact that the Delphi method is used
in response to a situation or a phenomenon rather than to test hypotheses. That is to
say, the Delphi method is efficient at explaining a problem or defining a situation, rather
than assessing the relationships between hypotheses. Therefore, we consider that it is
well adapted to develop a GIScience journal ranking methodology.
General approach. Successive questionnaires are sent to a targeted group of experts
on a given theme, until a consensus on the questions posed is reached. This method is
used to establish a consensus of opinion on a more or less quantitative specific subject.
It allows the researchers to predict, develop and explore the attitudes of a group, as well
as their needs and priorities (Jairath and Weinstein 1994). Iterations allow for feedback
from one survey to the other. Usually, three iterative surveys are enough to complete the
study (Duffield 1988).
Jury. The Delphi study is directed by a small group of jurors. The members of the
jury are not chosen at random but selected according to their subject matter knowledge
(Deitz 1987).
Expert. The definition, the choice and the maintenance of anonymity of the experts
are the key elements of a Delphi study. They prevent personal opinions from leading
the study. Moreover, anonymity stimulates ideas, as well as observations without any
pressure of external factors or from pairs (Goodman 1987).
Group of experts. The constitution of the group of experts is critical to the rigor of
the Delphi study (Linstone and Turoff 1975). The chosen respondents must be representative of the group and be familiar with the discipline or issue. The constitution of
the group of experts in the present study is mainly based on Martino’s (1983) theories.
In some specific fields, experts are too few in number to be chosen randomly. There is
no formal rule regarding the size of a group of experts (Linstone and Turoff 1975, Loo
2002). Martino (1983) proposes a rule of thumb: a group of 15 to 30 experts is necessary
for a heterogeneous population, where a group of 5 to 10 experts is enough for a
homogeneous population. To test a medicine, a heterogeneous population will consist
of physicians in an academic health-science centre, whereas a homogeneous population
will consist of specialist physicians in cardiology, for instance.
Limitations/Sources of error. The Delphi method does not escape criticism. Sackman
(1975), and then Woudenberg (1991), raised a series of questions about the scientific
bases of the Delphi method, whereas ample scientific evidence clearly demonstrates its
usefulness. Linstone and Turoff (1975), Martino (1983), and Loo (1997, 2002) confirmed
the good performance of the Delphi technique, but also specified the precise criteria that
are required to ensure efficiency and exactness. Constituting the group of experts and
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
300
C Caron, S Roche, D Goyer and A Jaton
determining the sample size are particularly important criteria. Choosing experts form
different geographical locations, or confirming the experts’ reputation through informal
discussions is also essential. Based on successful research outcomes, Mitchell and
McGoldrick (1994), Blow and Sprenkle (2001) and Bradley and Stewart (2003) all
agree. In order to mitigate the limits of the Delphi method, Dootson (1995) suggested
adopting a triangulation approach, using other complementary methods as well, which
is adopted in the present study.
The sampling of experts. The sampling process of experts is non-probabilistic in
nature, since the individuals are not chosen randomly. Sacks (2000) mentioned that the
more qualitative studies typically imply small samples, and enhance the sample representativeness rather than its size. We used Martino’s (1983) guidelines to gather a
homogeneous group.
In this study, the experts were chosen from a thorough study of the peer-review
journals in the fields of geomatics, geography, geographical information systems, and
other related disciplines. We listed all the Internet sites of these scientific journals so as
to identify the members of their respective editorial boards. This identification took
place over a period of several weeks. Based on an initial population of 1,110 experts
reported in the editorial review boards, a first sample of 174 experts was drawn. This
filter was carried out according to the following criteria: experts serving on at least two
editorial review boards and with an expertise in GIScience. The group includes selected
experts from all over the globe.
3.2 Constitution of the Initial Corpus of GIScience Journals
First, an initial listing of GIScience journals was compiled. The sampling process, used
to generate this basic corpus from the Delphi study, is non-probabilistic since the journals were not chosen at random. More specifically, we have mainly used the following
information sources:
•
•
•
•
•
•
•
Journal of Citation Reports 2003 – Social Sciences Edition, Institute for Scientific
Information (ISI), Thomson Scientific, Philadelphia, microfiches.
A selection of journals from the lists of the following sources : http://www.crg.ulaval.ca/
revues/revues.asp (June 2002), http://www.geomatique.georezo.net (June 2002), and
http://www.forum-pggq.com/liens/index.asp?section=10 (June 2002).
Research guide 2002–2003, Business Administration Department, Université de
Sherbrooke.
Taylor and Francis Journals: http://www.tandf.co.uk/journals/sublist.asp
Blackwell Journals: http://www.blackwellpublishing.com/listofj.asp
Elsevier Journals: http://www.elsevier.com/wps/find/journal_browse.cws_home
Biblio-GIS – Journals: http://perso.wanadoo.fr/dupuy.allignet/bibliosig/revues.htm
An initial sorting, produced by the jury2, reduced the corpus of 121 journals identified
at the beginning in the sources mentioned above, to 84 journals. After the first iteration of
the Delphi study, scientific journals that were region-based or unknown to most experts
were removed, as well as those published in languages unfamiliar to the experts, or those
whose content was not relevant to the definition of GIScience adopted in this study.
After the experts’ suggestions and the jury deliberation, the list of scientific journals,
used from the second iteration of the Delphi study with regard to the classification,
consisted of 54 titles3. Among these 54 journals, some are Francophone, but the vast
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
301
majority is Anglophone, i.e. most of them come from North America and Western
Europe.
3.3 Comparison with Other Measurement Approaches
To validate the classification resulting from the Delphi, we have compared it with the
JCR impact factor. Only some of the journals listed in the Delphi appear in the JCR,
which limits the comparison to only some of the classified journals. The ISI (Institute
for Scientific Information) JCR multidisciplinary database offers statistical data used to
assess how significant publications are in various fields. Two versions are available:
•
•
JCR Web Science Edition: 5,000 leading international scientific journals
JCR Web Social Sciences Edition: 1,600 key international journals
The impact factor is a quantitative measure of the frequency with which the articles
in a particular journal of a given field have been cited, compared to the other journals
of the same field. The impact factor is defined as follows:
The average numbers of times recent articles in a specific journal were cited in
the JCR cover year. For JCR impact factors recent articles and those published
in the two years preceding the JCR cover year. There are other ways of
calculating journal impact (Garfield 1972).
The impact factor is calculated by dividing the total number of citations published
during the previous two years by the total number of articles published in those two
years. For example, the impact factor of the American Political Science Review is 2.448,
resulting from the following calculation:
–
–
–
–
–
Number of citations in 2002: 123
Number of citations in 2001: 152
Number of articles published in 2001: 50
Number of articles published in 2000: 46
(123 + 152)/(50 + 46) = 235/96 = 2.448
The comparative approach used here consists of determining whether the sorting
in descending order of importance resulting from the Delphi also corresponds to a
descending impact factor.
4 Methodology Used in This Study
4.1 Questionnaires
A questionnaire was developed for each round of the Delphi (Appendices 1–4), and was
then distributed iteratively to the experts. The questionnaire was produced in English4.
To ensure a good understanding of the questionnaire, pre-tests were conducted before
each iteration on the jury of the Delphi and researchers from the GeoBusiness Group
(Université de Sherbrooke) and the Center for Research in Geomatics (Laval University).
Furthermore, a definition of “GIScience” was included with the questionnaire, in order
to make clear the boundaries of this particular field. To facilitate the experts’ work, the
questionnaires were developed using a four-level classification of journals (Faculté
d’administration 2002):
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
302
–
–
–
–
Level
Level
Level
Level
C Caron, S Roche, D Goyer and A Jaton
1:
2:
3:
4:
journals
journals
journals
journals
ranking in the top 15% of their field
ranking between 16% and 50% of their field
ranking between 51% and 100% of their field
that are not part of the first three categories5
When an expert had difficulties classifying some of the journals in one of the four
categories, this person could indicate “unknown” rather than a number from 1 to 4
(Hufnagel and Conca 1994).
An identical analytical method was used between each survey. The jury studied the
results and made certain decisions, such as discarding some redundant journals, and
then developed a questionnaire for the next survey. Each questionnaire listed the journals in alphabetical order to prevent the experts from being influenced by a presentation
of the journals in descending order (which may have led them to concentrate on the
evaluation of each journal individually).
4.2 First and Second Rounds
In August 2003, we sent an e-mail to the 174 experts of the original group. These
e-mails included a customized introductory letter, as well as the first questionnaire
(Appendix 1). A first reminder was sent three weeks after the initial message, but only
to 140 experts, for some e-mail addresses were invalid and certain experts had already
accepted or refused to participate in the study. Most of the refusals were on the ground
of lack of time, or because some of those contacted were deemed insufficiently qualified
with regards to GIScience. Thirty-seven experts (21%) answered the first round.
With the first questionnaire, the experts’ mandate was two-fold: to comment on the
relevance of each of the 84 journals appearing on the list, and to suggest adding missing
journals (Appendix 1). The following choices were offered: to decide whether a journal
should remain or be removed from the list for the second round, or to make no comment if the expert believed they did not have the necessary knowledge to judge the
journals or to include additional journals. At the end of the first round, the experts’
suggestions and comments enabled the jury to correct and enrich the original list.
Twenty-eight additional journals were proposed by the experts during the first
round (Table 1). Once the analysis and screening (number of experts who suggested a
journal, relevance of the journal regarding GIScience and breadth of the journal), the
six following journals were added by the jury: Cybergeo, Directions Magazine, GIM
International, ISPRS Highlights, Marine Geodesy and The Professional Geographer.
Moreover, updates were proposed, because some journals had changed their names
in recent years. It is the case for the Journal of the North American Cartographic
Information Society, known today as Cartographic Perspectives, but also Mapping
Awareness that is now GI News, or even the Revue de géographie de Lyon which today
is named the Géocarrefour. Some experts also emphasized certain redundancies –
the same journal was listed twice, but under two different names: Cartography and
Geographic Information Systems (same as Cartography and Geographic Information
Science), Geographical Systems (same as Journal of Geographical Systems).
The second round took place from November 2003 through January 2004. Its
process was the same as the first one, both in terms of questionnaire and outcome
measures. It included two reminders over a period of a month. Out of the 37 experts
contacted during the second round, only 28 answered the second questionnaire
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
Table 1
303
List of additional journals proposed by the experts (round 1)
Journals proposed
Number of suggestions
Administrative Sciences Quarterly
Arcuser
Statistics in Medecine
Catena
Computer and Operations Research
Cybergeo
Directions Magazine
Environmental Modeling
Geoderma
Geo-spatial Information Science (Wuhan)
GIM International
Hydrological Processes
Imaging Notes
ISPRS Highlights
Journal Mathematical Geology
Journal of Geographical Information Science
Journal of Hydrology
Journal of the Association for Computing Machinery
Marine Geodesy
Mondo GIS
Network and Spatial Economics
Ocean and Coastal Management
Policy Analysis
Professional Geographer
Public Administration Review
Sistema Terra
Journal of Geography in Higher Education
XYZ
1
1
1
1
1
3
1
1
1
1
1
1
1
1
2
1
1
1
1
1
1
1
1
4
1
1
1
1
(Appendix 2). This questionnaire required the experts to proceed in the same way as for
the first round, but then, they also had to propose a first classification for each journal.
To conclude this second round, the jury had once more, and using the same method, to
decide about the new journals added, as well as about the journals that the experts could
not make up their mind, whether they should remain on the list or not.
4.3 Third and Fourth Rounds
The third and fourth rounds took place, respectively from February 2004 to March
2004, and from April 2004 to May 2004. Both rounds followed the same patterns and
included two reminders over a period of a month. The questionnaires distributed to the
experts during both rounds are very similar (see the questionnaire of the third round,
Appendix 3). Both iterations have the same goal, namely to request each expert to
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
304
C Caron, S Roche, D Goyer and A Jaton
classify each journal according to the categories ranging from 1 to 4. Unlike the first
two rounds, these last two were no longer offering the possibility to add or remove
journals from the list. Each expert had two pieces of key information at their disposal
to proceed to a classification, that is to say: (1) their own evaluation of each journal
during the previous round and (2) the group of experts’ average rating of each journal.
The fact that each expert could have access to the average rating given by the group
represented an opportunity to be influenced. The expert could then maintain his/her
evaluation for each journal, or follow the overall opinion, so that a group consensus
could be achieved progressively and anonymously.
In fact, 27 experts answered the third iteration, whereas at the end of the fourth
and last iteration they were down to 26, which corresponds to a net response rate of
14.9% (26/174). This rate is very good and in line with expectations, since the literature
forecasts a 15% response rate for Delphi studies (Linstone and Turoff 1975). The
erosion of the group of experts after each round is a consistent characteristic of Delphi
studies, the average desertion rate usually reaching 10% to 40% (32% in our study).
The distribution of experts by country at the beginning and end of the study is
summarized in Table 2. The large majority came from Northern America (50%) and
Western Europe (27%).
The various iterations of a Delphi are complete only when a final consensus is
reached. Therefore, for the third and fourth rounds, the jury had to decide whether the
iterations should stop or continue. With regard to our study, four rounds were carried
out, which is in line with the development standards achieved by a Delphi study, which
usually consists of three to six rounds (Bradley and Stewart 2003, Linstone and Turoff
1975).
In order to determine accurately the achievement of a consensus, the jury measured
the stability factor (Scheibe et al. 1975) during the third and fourth rounds. For a given
iteration, the stability factor is the ratio between the total of units changed (in this case the
Table 2 Distribution of experts by countries
Countries
Original number of experts
Final number of experts
United-States
France
England
Austria
Canada
Germany
Australia
Italy
Netherlands
Switzerland
Tunisia
Japan
Total
11
8
5
3
3
1
1
1
1
1
1
1
37
10
3
3
2
3
1
0
1
0
1
1
1
26
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
305
differences between the answers given for each journal in two successive questionnaires),
and the number of participants (the total number of experts in a given iteration). This
calculation gives a percentage change, which should remain below 15% to have the
consensus achieved.
Table 3 shows the classification obtained at the end of the fourth iteration of the
Delphi. This ranking reveals a consensus among the experts contacted (for comprehensive
results, see Appendix 4).
5 Analysis and Triangulation of Results
Once the final results of the Delphi were obtained, we proceeded with the second part
of the study, namely the comparison with the JCR impact factor. To do so, we produced
a table comparing all the journals of the Delphi, ranking them from 1 to 3 that were
mentioned in the JCR (Table 4).
Table 4 indicates the JCR Rank in its main field for each journal analysed, which
enables the calculation of the ranking percentage for each of those journals. For
instance, the Applied Geography journal ranks 25th among the 35 journals listed in the
field “Geography”, that is to say 71st out of 100. Within a categorization ranking from
1 to 4, a ranking of 71% based on the JCR data corresponds to a level 3. Out the 44
journals ultimately classified from the Delphi, the same journal ranks 34th, which also
corresponds to a level 3. It is, thus, easy to make comparisons between the Delphi and
JCR approaches.
We have been able to compare nearly 50% of the journals from the Delphi (21/44).
Based on the analysis of these 21 journals, we can say that for:
57%: both classifications coincide
43%: both classifications do not coincide, but out of these 43%:
20%: the ranking resulting from the Delphi is just above the one calculated from
the JCR;
23%: the ranking resulting from the Delphi is just below the one calculated from
the JCR.
Consequently we can conclude that there is a match between the classification
resulting from the Delphi and the one compiled from the JCR (57%). When there is no match
(43%), the ranking given by the Delphi is equally just above or below the one resulting
from the JCR. On that account, the triangulation of the Delphi results and the JCR data
confirms the classification obtained. Therefore, we believe that, in regard to the 44 journals
the international experts consensually classified, the Delphi constitutes an innovative
and sound methodology to determine the importance of the journals in GIScience.
The final ranking, built on the basis of the Delphi and confirmed through a triangulation with the JCR, is presented in Table 5.
6 Conclusions
The classification, developed in a rigorous qualitative manner (Delphi method), has been
corroborated by a quantitative approach (JCR). Consequently, the resulting overall
ranking helps to correct a deficiency in GIScience.
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
306
C Caron, S Roche, D Goyer and A Jaton
Table 3 Final ranking obtained from the experts who participated to the Delphi
Journal
Delphi
International Journal of Geographic Information Science
International Journal of Remote Sensing
Photogrammetric Engineering & Remote Sensing (PE&RS)
Computers and Geosciences
Transactions in GIS
GeoInformatica
Geomatica
Cartography and Geographic Information Sciences
Environment and Planning B
IEEE Transactions Geoscience and Remote Sensing
Remote Sensing of Environment
Computers, Environments and Urban Systems
Annals of the Association of American Geographers
URISA Journal
Environment and Planning A
Landscape Ecology
Cartographica
Journal of Geographic Information and Decision Analysis
Marine Geodesy
Revue Internationale de Géomatique
Geographical System
Professional Geographer
Transactions of the Institute of British Geographers
Earth Surface Processus and Landforms
Progress in Human Geography
Spatial Cognition and Computation
Mapping Sciences and Remote Sensing
Canadian Journal of Remote Sensing
Cartographic Journal
Canadian Geographer
Geoscience Canada
Cybergeo
International Journal of Urban and Regional Research
Applied Geography
Surveying and Land Information Science
Earth Observation and Remote Sensing
Cartographic Perspectives (Journal North American Cartographic Information
Society)
Acta Cartographica
Remote Sensing Reviews
Espace Géographique
Geofocus International Review of GI Science and Technology
Mappemonde
Geomatique Suisse
Geographical Analysis
Géospatial Solutions
Geocarto International
1.12
1.27
1.45
1.52
1.58
1.60
1.60
1.64
1.64
1.65
1.67
1.70
1.72
1.77
1.82
1.92
1.95
2.12
2.15
2.15
2.20
2.20
2.20
2.25
2.28
2.29
2.38
2.39
2.39
2.45
2.47
2.50
2.50
2.53
2.53
2.67
2.71
2.73
2.80
2.90
3.00
3.07
3.11
3.20
3.43
3.71
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
Whole Title
JCR
Rank
Field (JCR)
Applied Geography
25/35 Geography
Canadian Geographer
21/35 Geography
Cartographic Journal
34/35 Geography
Environment and Planning A
8/35 Geography
Environment and Planning B
9/50 Environmental Studies
Geographical Analysis
19/35 Geography
Geoscience Canada
110/128 Geosciences, Interdisciplinary Computer Science
International Journal of Geographic Information
31/78 Computer Science, Information Systems
Science
International Journal of Remote Sensing
3/13 Imaging Science & Photographic Technology
International Journal of Urban and Regional Research
8/28 Urban Studies
Photogrammetric Engineering & Remote Sensing
21/35 Geography
Remote Sensing of Environment
1/11 Remote sensing
Professional Geographer
10/35 Geography
Progress in Human Geography
1/35 Geography
Landscape Ecology
50/128 Geosciences, Multidiciplinary
IEEE Transactions Geoscience and Remote Sensing
2/11 Remote Sensing
GeoInformatica
22/31 Geography, Physical
Earth Surface Processus and Landforms
42/128 Geosciences, Multidisciplinary
Earth Observation and Remote Sensing
119/128 Geosciences, Multidisciplinary
Canadian Journal of Remote Sensing
7/11 Remote Sensing
Computers and Geosciences
77/128 Geosciences, Multidisciplinary
JCR
Calculated DELPHI
%
Rank
Field Rank
71%
60%
97%
23%
18%
54%
86%
40%
3
3
3
2
2
3
3
2
34/44
30/44
29/44
15/44
9/44
43/44
31/44
1/44
23%
29%
60%
9%
29%
3%
39%
18%
71%
33%
93%
64%
60%
2
2
3
1
2
1
2
2
3
2
3
3
3
2/44
34/44
3/44
11/44
22/44
25/44
16/44
10/44
6/44
24/44
36/44
28/44
4/44
GIScience Journals Ranking and Evaluation
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
Table 4 Ranks obtained using Delphi and JCR approaches
307
308
C Caron, S Roche, D Goyer and A Jaton
Table 5 Final ranking of GIScience journals
Journal
Rank
International Journal of Geographic Information Science
International Journal of Remote Sensing
Photogrammetric Engineering & Remote Sensing (PE&RS)
Computers and Geosciences
Transactions in GIS
GeoInformatica
Geomatica
Cartography and Geographic Information Sciences
Environment and Planning B
IEEE Transactions Geoscience and Remote Sensing
Remote Sensing of Environment
Computers, Environments and Urban Systems
Annals of the Association of American Geographers
URISA Journal
Environment and Planning A
Landscape Ecology
Cartographica
Journal of Geographic Information and Decision Analysis
Marine Geodesy
Revue Internationale de Géomatique
Geographical System
Professional Geographer
Transactions of the Institute of British Geographers
Earth Surface Processus and Landforms
Progress in Human Geography
Spatial Cognition and Computation
Mapping Sciences and Remote Sensing
Canadian Journal of Remote Sensing
Cartographic Journal
Canadian Geographer
Geoscience Canada
Cybergeo
International Journal of Urban and Regional Research
Applied Geography
Surveying and Land Information Science
Earth Observation and Remote Sensing
Cartographic Perspectives (Journal North American Cartographic Information
Society)
Acta Cartographica
Remote Sensing Reviews
Espace Géographique
Geofocus International Review of GI Science and Technology
Mappemonde
Geographical Analysis
Geocarto International
Geomatique Suisse
Géospatial Solutions
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
309
Indeed, there are limits to these results. First of all, the group of experts contacted
is over-represented by residents from North America and Western Europe. Generalization
to the whole international community is, thus, precarious. For instance, how relevant
these results would be to Asia, especially China? Moreover, we did not carry out an
analysis aimed at determining in detail the experts’ specialized fields of expertise.
GIScience is characterized by a large variety of disciplinary backgrounds and expertise.
These specializations could, if the case arises, challenge the representativeness of the
group. For example, what could be the impact on the classification of certain journals
that are more specialized in photogrammetry or remote sensing? It is difficult to say.
The scale proposed here to carry out the Delphi (ranks from 1 to 4) entailed
difficulties in the experts’ decision-making process regarding the evaluation of some
scientific journals. For instance, some of them gave half scores, such as 2.5 or 3.5. Then,
we had to contact the experts to ask them to make a choice. Other experts informed us
that they had difficulties to carry out the exercise, mainly due to the very limited rating
scale. The rating exercise may have been easier if the experts had a wider choice than
the initial ranking from 1 to 4. For example, how might a five or seven-point Likert
scale have impacted the results?
Finally, the list of scientific journals covered by the Delphi does not pretend to be
exhaustive, even though it was compiled on the basis of various sources of information,
as well as on suggestions made by the participating experts. It is clear that the method
used to develop this ranking will have to evolve with time and according to the experts’
commentaries and further inquiries.
Sometimes though, this classification will necessitate updates. Journals evolve and
their rankings change over time as well, GIScience will evolve too, just like disciplines
continually do. The constant redefinition and the steady integration of new concepts
and new information technologies (IT) imply the need to maintain and improve our
global knowledge in this field. In order to make guarantees about how current, and
thus, useful this classification is, it is important to keep on improving and enhancing
this first version.
Generally speaking, it would be interesting to transpose this type of exercise to
conference rankings in GIScience (e.g. COSIT, GIScience conference), with respect to
thematic content and prestige, for example. Furthermore, we are convinced that the
methodology of this work could be generalized to research fields other than GIScience.
The Delphi method appears to be an innovative and efficient way to achieve this purpose.
We consider the innovative approach used in the present study as complementary
to the JCR approach. It efficiently fills some of the gaps of the JCR, such as the lack of
consideration for some key journals, and because few other rankings of journals from
the emerging field of GIScience have been attempted. This complementarity might imply
some changes relating to the criteria used to constitute the JCR, i.e. to complete these
criteria with others based on consensus building, such as the Delphi method.
Moreover, since the approach used in the present study aims at reaching consensus
among subject-field experts, we consider it as a sensible means for publishers, people in
charge of tenure-track and promotion decisions, and granting agencies to evaluate the
contribution and reputation of journals. We do hope that the present research will lead
to the development of a new journal ranking approach in the field of GIScience, and
possibly in other research fields. In this regard, we are ready to get involved in any
initiative aiming at structuring some kind of process to periodically update the results
of the present study.
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
310
C Caron, S Roche, D Goyer and A Jaton
7 Acknowledgements
The authors thank all the anonymous experts who participated to this research. The
quality of the results is directly connected to their generosity. Geoffrey Edwards, David Bachy,
and Mokhtar Saada should also be thanked for their review, as well as the Natural
Sciences and Engineering Research Council of Canada (NSERC) for their financial support.
Notes
1 The method for structuring a group communication process so that the process is effective in
allowing a group of individuals, as a whole, to deal with a complex problem confidentially
(Linstone and Turoff 1975).
2 The jury of the Delphi study consists of the authors of this paper.
3 One mistake was made during the Delphi study. The title: “Geographical Information Systems
for Urban and Regional Planning” was part of the final list compiling the 54 journals, even
though it is not a journal but a book. One of the experts had suggested adding this title during
the first iteration of the Delphi. But, neither the experts nor the jury of the Delphi ever detected
this mistake afterwards. Yet, it hardly affected the final ranking. We ultimately removed this
title. This mistake does not invalidate the remainder of the classification obtained.
4 A French version of the questionnaire was available upon request.
5 The fourth level corresponds to journals without editorial boards, that is to say professional
or transfer journals.
References
Amin M and Mabe M 2000 Impact factors: Use and abuse. Perspectives in Publishing 1: 1– 6
Bergeron M 1993 Vocabulaire de la Géomatique. Québec, Cahiers de l’Office de la langue
française, Publications du Québec
Blow A J and Sprenkle D H 2001 Common factors across theories of marriage and family therapy:
A modified Delphi study. Journal of Marital and Family Therapy 27: 385– 402
Bradley L and Stewart K 2003 A Delphi of Internet banking. Marketing Intelligence and Planning
21: 272– 81
Brancheau J C, Janz B D, and Wetherbe J C 1996 Key issues in information systems management:
1994–95 SIM Delphi results. MIS Quarterly 20: 225– 42
Caron C 2000 Les projets de géomatisation: contexte social et organisationnel. Revue Internationale
de Géomatique 10: 1–26
Chrisman N R 1999 What does “GIS” mean? Transactions in GIS 3: 175– 86
Deitz T 1987 Methods for analysing data from Delphi panels. Technological Forecasting and
Social Change 31: 79–85
Demeyrick J 2003 The Delphi method and health research. Health Education 103: 7–16
Dootson S 1995 An in-depth study of triangulation. Journal of Advanced Nursing 22: 183 –7
Dubois F L and Reeb D 2000 Ranking the international business journals. Journal of International
Business Studies 31: 689–704
Duffield C 1988 The Delphi technique. Australian Journal of Advanced Nursing 6(2): 41–5
Faculté d’administration 2002 Guide de la recherche 2002–2003. Sherbrooke, Université de
Sherbrooke Faculté d’administration
Frank A 1992 Spatial concept geometric data models, and geometric data structures. Computer
and Geosciences 18: 409–17
Fryrear R, Prill E, and Worzala M 2001 The use of Geographic Information Systems by corporate
real estate executives. Journal of Real Estate Research 22: 153–64
Gagnon P, and Bédard Y 1994 From surveying to geomatics: Evolution of education needs to
adapt to a new paradigm (a Canadian perspective). In Proceedings of the Third International
Think Tank of the Atlantic Institute, Vienna, Austria
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
311
Gagnon P and Coleman D J 1990 La géomatique: une approche systémique intégrée pour répondre
aux besoins d’information sur le territoire. CISM Journal 44: 383– 9
Garfield E 1972 Citation analysis as a tool in journal evaluation. Science 1: 527– 44
Gatrell T 2003 GIS and public health. Economic Geography 79: 341–2
Gatrell T and Smith A 1984 Networks of relations among a set of geographical journals. Professional Geographer 36: 300 –7
Gillenson M L and Stutz J D 1991 Academic issues in MIS: Journals and books. MIS Quarterly
15: 447–52
Goodchild M F 1992 Geographical Information Science. International Journal of Geographical
Information Systems 6: 31– 45
Goodchild M F, Egenhofer M J, Kemp K K, Mark D M, and Sheppard E 1999 Introduction to
the Varenius project. International Journal of Geographical Information Science 13: 731– 45
Goodman C M 1987 The Delphi technique: A critique. Journal of Advanced Nursing 12: 729– 34
Hamilton S and Ives B 1980 Communication of MIS research: An analysis of journal stratification.
In Proceedings oh the International Conference on Information Systems: 220 –32
Hardgrave B C and Walstrom K A 1997 Forums for MIS scholars. Communication of the ACM
40(11): 119–24
Hendriks P H J 1998 Information strategies for Geographical Information Systems. International
Journal of Geographical Information Science 12: 621–39
Hufnagel E M and Conca C 1994 User response data: The potential for errors and biases.
Information Systems Research 5: 48–73
Jairath N and Weinstein J 1994 The Delphi methodology: Administrative applications. Canadian
Journal of Nursing Administration 7(4): 29 – 42
Katerattanakul P, Han B, and Hong S 2003 Objective quality ranking of computing journals.
Communications of the ACM 46(10): 111–8
Kaynak K and Macaulay J A 1984 The Delphi technique in the measurement of tourism market
potential. Tourism Management: 87–101
Koong K S and Weistroffer H R 1989 Faculty usage of management information systems journals:
A survey. Journal of Computer Info Systems 30(1): 1– 4
Lanegran 1992 Communication. In Abler R F, Marcus M G, and Olson J M (eds) Geography’s
Inner Worlds. New Brunswick, NJ, Rutgers University Press: 187–211
Latour B 1987 Science in Action: How to Follow Scientists and Engineers through Society.
Cambridge, MA, Harvard University Press
Lee D and Evans A 1984 American geographers’ rankings of American geography journals.
Professional Geographer 36: 292–300
Lee D and Evans A 1985 Geographers’ rankings of foreign geography and non-geography journals.
Professional Geographer 37: 396 – 402
Linstone L and Turoff M 1975 The Delphi Method: Techniques and Applications. Reading, MA,
Addison-Wesley
Loo R 1997 Managing workplace stress in Canadian healthcare organizations. Journal of Management Development 16: 680–9
Loo R 2002 The Delphi method: A powerful tool for strategic management. International Journal
of Police Strategies and Management 25: 762–9
Mark D M 2000 Geographic Information Science: Critical issues in an emerging cross-disciplinary
research domain. Journal of Urban and Regional Information Systems Association 12(1): 45–
54
Martino J P 1983 An Introduction to Technological Forecasting. London, Gordon and Breach
Mitchell V W and McGoldrick P J 1994 The role of geodemographics in segmenting and targeting
consumer markets: A Delphi study. European Journal of Marketing 28(5): 54 –75
Moed H F, Van Leeuwen T N, and Reedijk J 1998 A new classification system to describe the
aging of scientific journals and their impact factors. Journal of Documentation 54: 387– 419
Murayama Y 2001 Geography with GIS. GeoJournal 52: 165–71
Ottens H 1999 Geographic Information Systems: An introduction. GeoJournal 48: 341– 4
Paradis M 1981 De l’arpentage à la géomatique. The Canadian Surveyor 35: 262–8
Péribois K, Roche S, and Caron C 2005 Etudes des variables conditionnant l’utilisation de
l’information géographique pour la participation publique locale. In Actes du Colloque International de Geomatique et d’Analyse Spatiale – SAGEO, Avignon, France
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
312
C Caron, S Roche, D Goyer and A Jaton
Pickles J (ed) 1995 Ground Truth: The Social Implications of Geographic Information Systems.
New York, Guilford Press
Pickles J 1997 Tool or science? GIS, technoscience and the theoretical turn. Annals of the Association
of American Geographers 87: 363 –72
Roche S, Sureau K, and Caron C 2003 How to improve the social-utility value of geographic
information technologies for the French local governments? A Delphi study. Environment
and Planning B 30: 429 – 47
Rousseau R 2002 Journal evaluation: Technical and practical issues. Library Trends 50: 418 –39
Sabrin M 2002 A ranking of the most productive business ethics scholars: A five-year study.
Journal of Business Ethics 36: 355–80
Sackman H 1975 Delphi Critique. London, Lexington Books
Sacks A M 2000 Research Measurement, and Evaluation of Human Resources. Scarborough,
Ontario, Nelson Thomson Learning
Scheibe M, Skutsch M, and Schofer J 1975 Experiments in Delphi methodology. In Linstone L and
Turoff M (eds) The Delphi Method: Techniques and Applications. Reading, MA, AddisonWesley 1–12
Schmidt R, Lyytinen K, Keil M, and Cule P 2001 Identifying software project risks: An
international Delphi study. Journal of Management Information 17: 5–36
Schuurman N 1999 Speaking with the enemy? An interview with Michael Goodchild. Environment
and Planning D 17: 1–15
Schuurman N 2000 Trouble in the heartland: GIS and its critics in the 1990s. Progress in Human
Geography 24: 569–90
Singleton A 1976 Journal ranking and selection: A review in physics. Journal of Documentation
32: 258–89
Sutter M and Kocher M G 2001 Tools for evaluating research output: Are citation-based rankings
of economics journals stable? Evaluation Review 25: 555–66
Tijssen R J W and Van Raan A E J 1990 A net citation balance: A measure of influence between
scientific journals. Journal of the American Society for Information Science 41: 298–304
Todorov R and Glaenzel W 1990 Compureur bibliometrics for journal classification. Information
Processing, and Management 26: 673–80
Walstrom K A, Hardgrave B C, and Wilson R L 1995 Forums for management information
systems scholars. Communications of the ACM 38(3): 93–107
Wilson A G 1985 Raising the levels of ambition in research: Some lessons from the journal.
Environment and Planning A 17: 465–70
Woudenberg F 1991 An evaluation of Delphi. Technological Forecasting and Social Change 40:
131–50
Wright D J, Goodchild M F, and Proctor J D 1997 Demystifying the persistent ambiguity of GIS
as “tool” versus “science”. Annals of the Association of American Geographers 87: 346 – 62
Appendix 1: Questionnaire #1
Please, answer the following 3 questions that represent your opinion.
1)
2)
3)
Look at the scientific and professional journals above and indicate those which,
according to your expertise, do not deserve to be evaluated because they do not
belong in the field GIS. In the second column entitled “RELEVANT”, please
answer by Yes or No.
Always according to the field of GIS, indicate the scientific or professional journals
that we omitted and those which deserve to be considered in the reserved area:
“Omitted journals”.
Please reply to us as soon as possible with the questionnaire completed at:
• Email adress: daniel.goyer@usherbrooke.ca
• Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
Journal
Acta Cartographica
Annales de Géographie
Annals of the Association of American Geographers
Applied Geography
Australian Geographer
Canadian Geographer
Canadian Journal of Remote Sensing
Cartographic Journal
Cartographica
Cartographie Mondiale
Cartography and Geographic Information Sciences
Cartography and Geographic Information Society
Cartography and Geographic Information Systems
Casual Cartographer
Computers and Geosciences
Computers, Environments and Urban Systems
Contour
Earth Observation and Remote Sensing
Earth Observation Magazine
Earth Surface Processes and Landforms
Economic Geography
Environment and Planning A
Environment and Planning B
Environment and Planning C
Environment and Planning D
Eurasian Geography and Economics
European Journal of Geographie
European Urban and Regional Studies
Geo Info
Geo Info systems
Geocarto International
Geofocus International Review of Geographical Information Science
and Technology
Geoforum
Geographical Analysis
Geographical and Environmental Modeling
Geographical Information Systems for Urban and Regional Planning
Geographical Journal
Geographical Review
Geographical System
Géographie Physique et Quaternaire
Geography
GeoInformatica
Geoinfosystems
Geomatica
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
313
Relevant? (Yes/No)
314
C Caron, S Roche, D Goyer and A Jaton
Geoscience Canada
Géospatial Solutions
Géoworld
GIS Europe
GIS World
GIS World Reports
IEEE Transactions Geoscience and Remote Sensing
International Journal of Geographical Information Science
International Journal of Remote Sensing
International Journal of Urban and Regional Research
Journal of Geographic Information and Decision Analysis
Journal of Geographical Systems
Journal of Geography
Journal of the North American Cartographic Information Society
Journal of Urban Planning and Development – ASCE
Journal of Urban Technology
Karlrusher Geoinformatik Report
Landscape Ecology
Les Cahiers de Géographies du Québec
L’espace Géographique
Map World Magazine
Mappemonde
Mapping (Spanish)
Mapping Awareness
Mapping Sciences and Remote Sensing
Mensuration, Photogrammétrie et Génie Rural
Meridian
Photogrammetric Engineering & Remote Sensing (PE&RS)
Progress in Human Geography
Remote Sensing of Environment
Remote Sensing Reviews
Revue Belge de Géographie
Revue Géographique de Lyon
Revue Internationale de Géomatique
Spatial Cognition and Computation
Surveying and Land Information Systems
Transactions in GIS
Transactions of the Institute of British Geographers
Urban Geography
URISA Journal
Omitted journals
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
315
Appendix 2: Questionnaire #2
Dear Sir/Madam,
We are pleased to note that you are among the 38 experts that answered our first survey
concerning our research project for classifying the scientific journals in the field of
Geographic Information Science (GIS). For a second time, we request your professionnal
skills to complete Survey # 2.
Following your suggestions from the first round, we eliminated nearly 20 journals.
However, we need your opinion, a second time, to evaluate some other journals from
the first round. Also, some new suggestions of journals provided by some of your
colleagues who answered the first survey need to be evaluated.
To do so, we provide you a definition of Geographic information science:
“GIScience has recently emerged as the field where problems of data capture,
encoding, storage, analysis, retrieval, synthesis, and dissemination of geographic information are studied. These problems have increased in significance
as new computer based technologies such as geographic information systems
(GIS), digital remote sensing (RS) and Global Positioning Systems (GPS) evolve.
GI Science is the foundation upon which these GI technologies are built.”
(Source: http://www.gisc.berkeley.edu/about/whatisgis.html)
Please, answer the following two questions that represent your opinion.
1)
2)
Look at the scientific journals above and indicate those which, according to your
expertise, do not deserve to be evaluated because they do not belong in the field of
Geographic Information Science. Please answer the question in the column entitled
“Yes, No or ?” question. (If you are unfamiliar with the journal, put a question
mark (?))
For each journal in both lists, identify its ranking according to the scale below
(If you are unfamiliar with the journal, put a question mark (?))
–
–
–
–
Level
Level
Level
Level
1:
2:
3:
4:
Peer journal classified among the first 15% of their field
Peer journal classified between 16% and 50% of their field
Peer journal classified between 51% and 100% of their field
Journal not part of the first three categories
(For example: A journal that does’t have a referee process.)
Please reply by December 12th, 2003 with the survey completed at:
•
•
Email address: daniel.goyer@usherbrooke.ca
Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)
Journal
(Yes, No or ?)
Canadian Geographer
Cartographie Mondiale
Casual Cartographer
Cybergeo
Directions magazine
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
Level
316
C Caron, S Roche, D Goyer and A Jaton
Earth Surface Processes and Landforms
Espace Géographique
Geo Info
Geoscience Canada
GI News (Mapping Awareness)
GIM international
International Journal of Urban and Regional Research
ISPRS highlights
Journal of Urban Technology
Landscape Ecology
Marine Geodesy
Mensuration, Photogrammétrie et Génie Rural
Professional Geographer
Progress in Human Geography
Transactions of the Institute of British Geographers
Part 2 of the survey
Level
Acta Cartographica
Annals of the Association of American Geographers
Applied Geography
Canadian Journal of Remote Sensing
Cartographic Journal
Cartographica
Cartography and Geographic Information Sciences
Computers and Geosciences
Computers, Environments and Urban Systems
Earth Observation and Remote Sensing
Earth Observation Magazine
Environment and Planning A
Environment and Planning B
Journals
Level
Geocarto International
Geofocus International Review of Geographical Information Science and
Technology
Geographical Analysis
Geographical Information Systems for Urban and Regional Planning
Geographical System
GeoInformatica
Geomatica
Géospatial Solutions (GeoInfo systems)
Géoworld (GIS World)
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
317
IEEE Transactions Geoscience and Remote Sensing
International Journal of Geographical Information Science
International Journal of Remote Sensing
Journal of Geographic Information and Decision Analysis
Cartographic Perspectives (Journal North American Cartographic Information Society)
Mappemonde
Mapping Sciences and Remote Sensing
Photogrammetric Engineering & Remote Sensing (PE&RS)
Remote Sensing of Environment
Remote Sensing Reviews
Revue Internationale de Géomatique
Spatial Cognition and Computation
Surveying and Land Information Systems
Transactions in GIS
URISA Journal
We warmly thank you for your participation in this research.
The research team:
Claude CARON, research responsible
University of Sherbrooke (Québec, Canada)
Annick JATON, research adviser
Laval University (Québec, Canada)
Stéphane ROCHE, researcher
Laval University (Québec, Canada)
And University of Angers (France)
Daniel GOYER, research assistant
University of Sherbrooke (Québec,
Canada)
Appendix 3: Questionnaire #3
Dear Sir/Madam,
We are pleased to note that you are among the experts that answered our second Delphi
Survey concerning our research project for classifying the scientific journals in the field
of Geographic Information Science (GIS). We are providing you with the results of
Survey #2 and we require your professionnal skills to complete Survey # 3.
Referring to the definition of Geographic Information Science *, please answer the
two questions below:
*“GIScience has recently emerged as the field where problems of data capture,
encoding, storage, analysis, retrieval, synthesis, and dissemination of
geographic information are studied. These problems have increased in
significance as new computer based technologies such as geographic
information systems (GIS), digital remote sensing (RS) and Global Positioning
Systems (GPS) evolve. GI Science is the foundation upon which these GI
technologies are built.”
Source: http://www.gisc.berkeley.edu/about/whatisgis.html
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
318
C Caron, S Roche, D Goyer and A Jaton
1)
Specify in which domain you consider yourself as an expert. (Cartography,
Geography, Geodesy, . . .)
2a) In the listing below, take notice of your first evaluation in the column “You” and
then compare it with the column “Group” (mean value of the group).
2b) For each journal that you evaluated, you have two options:
A. You can propose a new ranking* in the column “Now”.
B. You can keep your old ranking by leaving the column “Now” empty.
* (Don’t forget that you must rank the journals with an integer)
2c)
Also, if you wish to rank a journal that you have not evaluated in the first survey,
you are welcome to do so.
Please reply by March 5th 2004 with the survey completed at:
•
•
Email address: daniel.goyer@usherbrooke.ca
Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)
(If you are unfamiliar with the journal, put a question mark (?) or leave it empty)
–
–
–
–
Level
Level
Level
Level
1:
2:
3:
4:
Peer journal classified among the first 15% of their field
Peer journal classified between 16% and 50% of their field
Peer journal classified between 51% and 100% of their field
Journal not part of the first three categories
Journal
You Group Now
Acta Cartographica
Annals of the Association of American Geographers
Applied Geography
Canadian Geographer
Canadian Journal of Remote Sensing
Cartographic Journal
Cartographic Perspectives (Journal North American Cartographic
Information Society)
Cartographica
Cartography and Geographic Information Sciences
Computers and Geosciences
Computers, Environments and Urban Systems
Cybergeo
Directions magazine
Earth Observation and Remote Sensing
Earth Observation Magazine
Earth Surface Processus and Landforms
Environment and Planning A
Environment and Planning B
Espace Géographique
Géo Europe (GI News – Mapping Awareness)
Geo Info
2.7
1.8
2.3
2.4
2.3
2.2
2.6
1.9
1.6
1.5
1.8
2.7
3.8
2.6
3.5
2.4
1.8
1.6
2.4
3.7
3.7
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
GIScience Journals Ranking and Evaluation
Geocarto International
Geofocus International Review of Geographical Information Science
and Technology
Geographical Analysis
Geographical Information Systems for Urban and Regional Planning
Geographical System
GeoInformatica
Geomatica
Geoscience Canada
Géospatial Solutions
Géoworld
GIM international
IEEE Transactions Geoscience and Remote Sensing
International Journal of Geographical Information Science
International Journal of Remote Sensing
International Journal of Urban and Regional Research
ISPRS highlights
Journal of Geographic Information and Decision Analysis
Landscape Ecology
Mappemonde
Mapping Sciences and Remote Sensing
Marine Geodesy
Mensuration, Photogrammétrie et Génie Rural
Photogrammetric Engineering & Remote Sensing (PE&RS)
Professional Geographer
Progress in Human Geography
Remote Sensing of Environment
Remote Sensing Reviews
Revue Internationale de Géomatique
Spatial Cognition and Computation
Surveying and Land Information Systems
Transactions in GIS
Transactions of the Institute of British Geographers
URISA Journal
319
2.8
3.1
1.5
2.1
1.8
1.7
2.3
3.0
3.6
3.7
3.6
1.6
1.1
1.4
2.3
3.3
2.1
1.8
2.9
2.3
2.5
3.0
1.5
2.1
1.9
1.7
2.7
2.2
2.4
2.5
1.7
1.9
1.8
We warmly thank you for your participation in this research.
The research team:
Claude CARON, research responsible
University of Sherbrooke (Québec, Canada)
Annick JATON, research adviser
Laval University (Québec, Canada)
Stéphane ROCHE, researcher
Laval University (Québec, Canada)
And University of Angers (France)
Daniel GOYER, research assistant
University of Sherbrooke (Québec,
Canada)
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
320
Appendix 4: Delphi Final Results
Average (Rank)
Std. Deviation
Stability
Journal
Rnd. #2
Rnd. #4
Rnd. #2
Rnd. #4
Rnd. #3
Rnd. #4
33
34
43
10
51
25
26
9
18
32
46
11
2
53
17
38
8
37
41
48
24
44
52
16
45
49
International Journal of Geographical Information Science
International Journal of Remote Sensing
Photogrammetric Engineering & Remote Sensing (PE&RS)
Computers and Geosciences
Transactions in GIS
GeoInformatica
Geomatica
Cartography and Geographic Information Sciences
Environment and Planning B
IEEE Transactions Geoscience and Remote Sensing
Remote Sensing of Environment
Computers, Environments and Urban Systems
Annals of the Association of American Geographers
URISA Journal
Environment and Planning A
Landscape Ecology
Cartographica
Journal of Geographic Information and Decision Analysis
Marine Geodesy
Revue Internationale de Géomatique
Geographical System
Professional Geographer
Transactions of the Institute of British Geographers
Earth Surface Processes and Landforms
Progress in Human Geography
Spatial Cognition and Computation
1.08
1.38
1.48
1.52
1.71
1.74
2.31
1.59
1.59
1.63
1.65
1.78
1.78
1.77
1.77
1.80
1.91
2.12
2.50
2.15
1.78
2.13
1.92
2.43
1.92
2.36
1.12
1.27
1.45
1.52
1.58
1.60
1.60
1.64
1.64
1.65
1.67
1.70
1.72
1.77
1.82
1.92
1.95
2.12
2.15
2.15
2.20
2.20
2.20
2.25
2.28
2.29
0.27
0.59
0.59
0.58
0.84
0.91
0.86
0.71
0.72
0.49
0.75
0.51
0.93
0.74
0.80
0.79
0.61
0.62
1.00
0.80
0.73
0.75
0.86
0.79
0.92
0.65
0.33
0.46
0.67
0.51
0.65
0.68
0.60
0.66
0.73
0.49
0.59
0.47
0.74
0.61
0.80
0.86
0.59
0.49
0.55
0.80
0.62
0.83
0.77
0.71
0.83
0.61
0.12
0.13
0.09
0.13
0.12
0.11
0.07
0.09
0.13
0.06
0.11
0.17
0.08
0.09
0.23
0.17
0.10
0.17
0.25
0.29
0.06
0.06
0.17
0.00
0.08
0.08
0.04
0.00
0.00
0.04
0.13
0.15
0.05
0.09
0.09
0.05
0.11
0.00
0.08
0.14
0.05
0.15
0.05
0.12
0.00
0.15
0.05
0.00
0.00
0.25
0.11
0.00
C Caron, S Roche, D Goyer and A Jaton
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
#
Appendix 4: Continued
Std. Deviation
Stability
Journal
Rnd. #2
Rnd. #4
Rnd. #2
Rnd. #4
Rnd. #3
Rnd. #4
40
5
6
4
27
12
35
3
50
14
7
1
47
19
22
39
42
23
28
29
36
15
20
31
13
21
30
Mapping Sciences and Remote Sensing
Canadian Journal of Remote Sensing
Cartographic Journal
Canadian Geographer
Geoscience Canada
Cybergeo
International Journal of Urban and Regional Research
Applied Geography
Surveying and Land Information Systems
Earth Observation and Remote Sensing
Cartographic Perspectives (Journal North American Cart. Inf. Society)
Acta Cartographica
Remote Sensing Reviews
Espace Géographique
Geofocus International Review of GI Science and Technology
Mappemonde
Mensuration, Photogrammétrie et Génie Rural
Geographical Analysis
Géospatial Solutions
Geoworld
ISPRS highlights
Earth Observation Magazine
Geo Info
GIM international
Directions magazine
Geocarto International
GI News (Mapping Awareness)
Average
2.33
2.31
2.22
2.38
3.00
2.71
2.25
2.33
2.53
2.57
2.62
2.69
2.67
2.38
3.11
2.92
3.00
1.54
3.58
3.73
3.25
3.50
3.71
3.64
3.83
2.80
3.69
2.36
2.38
2.39
2.39
2.45
2.47
2.50
2.50
2.53
2.53
2.67
2.71
2.73
2.80
2.90
3.00
3.07
3.11
3.20
3.43
3.50
3.63
3.64
3.65
3.69
3.71
3.71
3.71
2.42
0.75
0.59
0.65
0.50
0.71
1.20
1.04
0.80
0.74
0.74
0.74
0.63
0.48
0.97
1.10
0.66
0.76
0.87
1.08
0.45
1.04
0.74
0.48
0.65
0.41
0.70
0.61
0.73
0.51
0.50
0.50
0.52
0.64
0.97
0.71
0.70
0.64
0.62
0.61
0.59
0.42
0.74
0.60
0.47
0.33
0.79
0.53
0.76
0.52
0.50
0.49
0.48
0.49
0.47
0.46
0.61
0.15
0.06
0.06
0.00
0.13
0.25
0.38
0.20
0.13
0.14
0.07
0.00
0.10
0.22
0.09
0.14
0.25
0.20
0.20
0.25
0.29
0.14
0.06
0.09
0.00
0.24
0.00
0.132
0.00
0.06
0.06
0.09
0.13
0.20
0.10
0.05
0.00
0.00
0.14
0.07
0.10
0.30
0.08
0.21
0.22
0.30
0.14
0.15
0.25
0.14
0.06
0.00
0.00
0.00
0.08
0.091
321
#
GIScience Journals Ranking and Evaluation
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Transactions in GIS, 2008, 12(3)
Average (Rank)