An integrated approach for the analysis of factors affecting journal
citation impact in clinical neurology
Weiping Yue*
The University of New South Wales, School of Information Systems, Technology and Management,
Sydney, NSW, 2052 Australia. Email: wp.yue@unsw.edu.au.
Concepción S. Wilson
The University of New South Wales, School of Information Systems, Technology and Management,
Sydney, NSW, 2052 Australia. Email: c.wilson@unsw.edu.au.
This study continues to develop and test an
integrated conceptual model of journal citation
impact. The main objective is to obtain a more
complete understanding of how and to what
extent the external factors affect journal citation
impact. Four external factors have been
identified from previous studies: journal
properties,
journal
accessibility,
journal
internationality and perceived journal quality. A
conceptual model of journal citation impact is
developed and a web-based survey is conducted
for obtaining information on the perceived
journal quality. Structural equation modeling
(SEM) with partial least squares (PLS) is used to
test the conceptual models with empirical data
from 41 research journals in clinical neurology.
Interrelationships among journal citation impact
and the four external factors have been
successfully explored. The results also suggest
that, for an integrated conceptual model of
journal citation impact, various citation
indicators should be used.
Introduction
Based on the standard perspective of citation, which views
citing as a merit-granting process, it is plausible to assume
that the information value of a given publication is
reflected by the frequency of citations obtained from other
publications. Therefore, citation counts can be used as
indicators or measurements of the level of quality,
importance, impact, influence or performance of
individual publications or aggregations of publications,
such as journals (Wilson, 1999). In the past decades,
citation analysis has been extensively utilized as a
practical evaluation tool to assess journal performance in
various disciplines. Journal citation indicators from the
Institute for Scientific Information (ISI) have become
popular measures for assessing journal quality, in
*
particular, the journal impact factor and the number of
total citations.
Nevertheless, researchers have realized that the citation
process is complex and it is affected by certain external
factors (Cronin, 1984; Gilbert, 1977). For example,
Garfield (1979, p. 148), the founder of the Science
Citation Index (SCI) and the journal impact factor, has
indicated that “scientific merit is not always the sole
reason an author will cite a paper published in a
particular journal”. Garfield (1979) further stated that an
author’s citation behavior might be influenced to varying
degrees by the visibility, prestige, and accessibility of the
cited journal. Likewise, Cronin (1984) suggested that
social and psychological factors as well as other external
factors, such as the target audience or the character and
status of a cited journal, affect the citing process.
In order to use citation analysis for evaluating journals
more objectively and comprehensively, we need a better
understanding of what and how external factors influence
the journal citation impact. To date, little effort has been
made to explore the complex interrelationships among the
various external factors with respect to the journal citation
impact. Thus, how and to what extent do the external
factors influence the journal citation impact in the context
of journal evaluation become crucial issues.
This study continues to develop and test an integrated
conceptual model of journal citation impact (Yue &
Wilson, 2003). The main objective of this study is to
obtain a more complete understanding of how the factors
affect journal citation impact. Although similar issues
have been discussed in previous literatures, there is no
study attempting to address the research problem at the
conceptual level and to integrate all the operational
external factors of journal citation impact into one model.
In this study, four external factors have been identified
from previous studies: journal properties, journal
accessibility, journal internationality, and perceived
For correspondence.
1
journal quality. We argue that such a conceptual model
involves latent variables or constructs that cannot be
observed directly. More specifically, the journal citation
impact itself and its external factors are regarded as
constructs that only can be inferred by other observed
indicators. Therefore, our study not only identifies the
external factors that affect journal citation impact but also
identifies the observed indicators of the external factors
and of the journal citation impact. Based on our prior
research findings we developed a web-based survey to
obtain the information on the perceived journal quality
and incorporated the results in our model. The proposed
model addresses the complex interrelationships among
these external factors and journal citation impact in one
case study, and further explores the extent to which the
journal citation impact is influenced by external factors.
citation analysis and previous empirical findings of
journal evaluation. Five constructs have been chosen for
the structural model and seven propositions were
developed from the structural model. Figure 1 graphically
depicts the propositions to be tested.
Methodology
Research Question
Journal Citation Impact: Impact is defined as a forceful
consequence or strong influence. In this study, journal
citation impact is used to refer to the degree of the journal
influence as gauged by the uses made of the journals
through citations. It is reflected through various journal
citation indicators obtained or calculated from ISI
indicators.
This study identifies five constructs: journal properties,
journal accessibility, journal internationality, perceived
journal quality, and journal citation impact. It addresses
the following interrelated research question: What is the
strength of the relationship among the five constructs each
in turn, and most importantly the relationship between the
journal citation impact and the other four constructs?
Models Developed
This study uses structural equation modeling (SEM) with
partial least squares (PLS) – a relatively new approach for
testing multivariate models with empirical data. SEM
incorporates both observed and latent variables; usually
these variables can be separated into a structural model
and several measurement models. The measurement
models address the reliability and validity of the indicators
in measuring latent variables or hypothetical constructs,
while the structural model specifies the direct and indirect
relations among the latent variables and describes the
extent of explained and unexplained variances in the
model.
Instead of using co-variance based SEM techniques, PLS
was used as the primary strategy for examining the
relative effects of external factors on the journal citation
impact, as it is suitable for “causal-predictive analysis in
situations of high complexity but low theoretical
information” (Jöreskog & Wold, 1982; p270). In addition,
PLS makes no prior distributional assumptions and is
applicable to small sample sizes (Sambamurthy & Chin,
1994).
Structural Model
A structural model is regression-based and consists of the
unobservable constructs and the theoretical relationships
among them. In this research, the structural model has
been developed on the basis of theoretical works of
J. Properties
Perceived
J. Quality
J. Accessibility
Journal
Citation
Impact
J. Internationality
Figure 1: Structural Model of Journal Citation Impact
Journal Properties: This construct is partly an
“editor-driven” concept, which describes the basic and
historical attributes of a particular journal. Journal
properties include, inter alia: age, size, frequency,
multidisciplinarity, the number of review articles for a
particular year, and the number of societal affiliations.
Journal Accessibility: Journal accessibility represents the
extent of the availability of a journal when needed by users.
It is a multi-dimensional concept encompassing physical
and content access of journals. Journal circulation (or
subscription), language, and the number of Abstracting
and Indexing (A&I) services or databases that choose to
include a particular journal are indicators of accessibility,
and thus, a mix of “publisher-controlled”, “editor-driven”
and “reader- or user-choice” construct.
Journal Internationality: Journal internationality refers to
the extent of a journal being international or global in
scope. It includes the indicators of the nationality (or
country affiliation) of the authors, the editor-in-chief and
members of the editorial board.
Perceived Journal Quality: “Quality” refers to a degree of
excellence or worth. In this study, perceived journal
quality is a concept with respect to the general perception
of a journal from peer assessment, obtained through a
survey questionnaire, and thus purely a “reader- or
user-choice” construct.
2
Measurement Models
Correspondingly, five measurement models have been
developed for the five constructs in the structural model.
Each measurement model consists of the relationships
between the observed variables and the constructs
measured. Indicators for each construct are derived from
those used in previous studies. When using SEM with PLS,
the indicators in the measurement model can be modeled
as reflective or formative. Reflective indicators are viewed
as affected by the same underlying concept (i.e., the Latent
Variable or LV). The change in the underlying LV will
result in similar changes in its reflective indicators. Thus,
reflective indicators should be correlated. In contrast,
formative indicators are measures that form or cause the
creation or change in a LV and they are not assumed to be
correlated (Chin, 1998). Wold (1980), the developer of the
PLS technique, suggested that in a situation of low
theoretical knowledge, all independent LVs should be
modelled in a formative mode and all dependent LVs in a
reflective mode This recommendation is adopted in
creating the model of journal citation impact for this study.
In short, journal properties, journal accessibility, journal
internationality and perceived journal quality are modeled
formatively; whereas, journal citation impact is modeled
reflectively. The five measurement models are shown with
all their indicators in Figures 2 to 6.
Measurement Model of Journal Citation Impact: The
indicators of citation impact used in this study are either
obtained or calculated from ISI data.
Impact factor
Immediacy index
Journal
Citation
Impact
Total citations: The indicator for total citations refers to
the total number of citations to articles in a journal for the
year 2000.
Cited/Citing rate: In previous studies, cited/citing rate was
regarded as a quality index of scientific journals (Price,
1981; Zinkhan & Leigh, 1999) or the input/output ratio
reflecting the status of the journal in a given network (Kim,
1992). In our study, this indicator is calculated by dividing
the total number of citations that a journal received by the
total number of references that the same journal has given;
both numbers are for the year 2000.
Citing disciplines: Citing disciplines refer to the extent of
the inter- or multi- disciplinary impact of a journal. It can
be traced through the analysis of subject areas from where
citations originate (Wormell, 1998). The measure is the
number of citing disciplines; this is obtained by using the
Dialog information system to search all the citations to
each journal, in our case study for the year 2000, and then
ranking the journal subject category (SC) field.
Citing Author Geographic Location: Geographic locations
(GL) or country affiliations of all citing authors in a
particular journal for the year 2000 can be obtained by
searching Science Citation Index; further, the GL field can
be ranked to produce a list of countries in decreasing order
of productivity. Hence, journal citation impact with
respect to the citing authors can be determined by the
number of country affiliations of the citing authors.
Measurement Model of Journal Properties: Journal
properties are formed by the journal age, size, publication
frequency, multidisciplinarity, number of review articles,
and number of societal affiliations.
Total citations
Journal age
Cited/Citing rate
Journal size
Citing disciplines
Citing author geographic location
Figure 2: Measurement model of Journal Citation Impact
Impact factor: The journal impact factor is a measure of
the frequency with which the "average article" in a journal
has been cited in a particular year. The ISI impact factor
for the year 2000 was used.
Immediacy index: The journal immediacy index is a
measure of how quickly the "average article" in a journal
is cited and for this study, the 2000 immediacy index was
used.
Journal
Properties
Frequency
Multidisciplinarity
No. of Reviews
No. of Affiliations
Figure 3: Measurement Model of Journal Properties
Journal age: Journal age was calculated from the year the
journal was first published regardless of how many times
the journal changed its title or altered its scope.
Journal size: Journal size was defined as the total number
of articles the journal published in the year 2000. It is
3
collected from the Journal Citation Reports for 2000, and
mainly includes the document types, articles and reviews.
Frequency: Frequency of a journal is the number of issues
the journal publishes each year.
Multidisciplinarity: Each journal in the ISI database is
assigned at least one subject category, indicating a specific
area of knowledge. Journals may be included in more than
one subject category; therefore, the extent or number of
subject areas a journal covers can be an indicator of the
journal’s multidisciplinarity.
Number of Reviews: The total number of review articles in
the journal for the year 2000 was regarded as an indicator
of journal properties.
Number of Affiliations: The number of affiliations a
journal has with academic societies is another indicator of
a journal’s property.
Measurement Model of Journal Internationality: Journal
internationality is formed by the number of country
affiliations of authors and editorial board members. In
previous studies, this construct has been discussed from
various aspects: the international scope of the readers
(Wormell, 1998) and the editorial board (Zsindely et al.,
1982); the geographical source of citations (Ren &
Rousseau, 2002; Wormell, 1998); and affiliation data of
the authors (Gutiérrez & López-Nieva, 2001). In this
paper, due to the difficulty of obtaining readership
information from the publishers, we had to delete reader
geographic location from the operational model.
Alternatively, we could regard the actual users’ (that is,
citing authors’) geographic location distribution as a
measure to replace the readership internationality.
However, due to the objectivity of our study, the citing
authors’ geographic location is regarded as more related to
the journal citation impact, and therefore, was considered
as a reflective indicator of journal citation impact.
Author GL
Measurement Model of Journal Accessibility: The
formative indicators of journal accessibility include:
circulation, publication language of the journal, and the
number of A&I services.
Journal
Internationality
Editorial board GL
Figure 5: Measurement model of Journal Internationality
Circulation
Journal
Accessibility
Language
A&I services
Figure 4: Measurement model of Journal Accessibility
Circulation: Although circulation size may over-represent
a popular generalist journal and under-represent a
scholarly specialist journal, it can still be used as a
measure of a journal’s accessibility in a discipline. In this
study circulation size from the Ulrich’s database is
regarded as an indicator of the journal’s accessibility.
Language: More than 95% of journal articles in the SCI
for 1998 are in English and the role of English as the major
language of scientific communication is ‘striking’ (Van
Leeuwen, et al., 2001). Thus, the language of publication
becomes an important indicator. It is treated as a ‘dummy’
variable; journals published in English were labeled ‘1’
while ‘2’ was assigned to the journals published in
multi-languages.
A&I services: This measure is defined as the total number
of A&I services or databases indexing a journal. The more
A&I services indexing a journal, the wider the potential
audience of readers (Ali et al., 1996).
Author Geographic Location: As with the country
affiliations of the citing authors, the GLs of authors of
documents in each journal for the year 2000 can be
obtained. The method of assigning values to this measure
for each journal is the same as that of assigning values to
the citing authors.
Editorial Board Geographic Location: The number of
country affiliations of the editors-in-chief and of the
editorial board members was also determined for each
journal for the year 2000.
Measurement Model of Perceived Journal Quality: In this
study, perceived journal quality is formed by one indicator
only – the results from a web-based survey.
Perceived J.
Quality
Peer Perception
Figure 6: Measurement model of Perceived Journal
Quality
Propositions
(1) Journal properties affect journal citation impact:
Numerous studies have found that journal properties
influence different citation indicators. For instance, the
frequency and the size of a journal have a positive effect
on the value of the journal impact factor (Rousseau &
4
Hooydonk, 1996). Journal age has been regarded as the
main factor for the number of total citations (Cooper et al,
1993; Dombrowski, 1988). Moreover, the coverage or
scope of a journal is also regarded as a measure of citation
impact (Dombrowski, 1988). The argument is that the
more disciplines the journal content covers, the more
people would use it and thus the more citations the journal
would attract. The type of articles in a journal often
influences the journal citation impact. For example,
review articles generally attract more citations due to its
special publication type. Further, Smart and Elton (1981)
also pointed out that the citation frequency could be
explained by journal structural characteristics, such as
journal size, references per article, and disciplinary focus.
In short, as indicated by Cozzens (1989), there is a need to
analyze journal characteristics in terms of citation
‘inflators’ or ‘deflators’. Thus, journal properties are
proposed to affect the journal citation impact.
(2) Journal accessibility affects journal citation impact:
Previous studies indicate that one reason for not citing
relevant works was the inaccessibility of the journal
(Dubios & Reeb, 2000; Gorman, 2000; Van Dalen &
Henkens, 1999; Cooper et al., 1993). The unavailability of
the journal might be due to physical inaccessibility or
other factors that hinder potential citers from using the
work. For example, the language of publication can affect
the number of citations (Dombrowski, 1988). Moreover,
coverage by A&I services may also influence the citation
rates of the journal (Cooper et al., 1993). The argument is
that the more A&I services indexing the journal, the wider
the potential readers (Ali et al., 1996) and the more
opportunity for the journal to be cited. As journal
accessibility may influence citation behavior, it is
proposed in this study that journal accessibility affect
journal citation impact.
(3) Journal internationality affects journal citation impact:
Although the issue of journal internationality has been
discussed, there are few studies investigating the
relationship between journal internationality and journal
citation impact. Theoretically, if a journal has a larger
cohort of international ‘source’ authors and editors, then it
is more likely to attract more citations and perhaps be
more ‘influential’. However, with respect to the editorial
board, Nisonger (2002) found no correlation between the
international composition of the editorial board and two
citation measures: the impact factor and total citations.
Nevertheless, it is necessary to explore this issue further.
Thus, it is proposed that journal internationality could
affect journal citation impact.
Dieks and Chang (1976) stated that the ‘impact’ of a paper
was not only to be determined by its scientific significance,
but also by the prestige of the journal in which the paper is
published. Likewise, Johnson (1997) pointed out that a
‘good’ journal increased the rate of citations. Anderson
and Goldstein (1981) also indicated that the perceived
reputation of a journal could be related to the citation
counts to that journal from other journals. Thus, in this
study, perceived journal quality is proposed to affect
journal citation impact.
(5) Journal properties affect perceived journal quality:
These indicators forming journal properties were thought
to be potentially important determinants of journal quality
(Bioscience, 1973; Brooks et al, 1991; Zinkhan and Leigh,
1999). For instance, it is generally assumed that a journal
would have an established reputation if it has been
published for a substantial number of years (Ali et al.,
1996). In addition, affiliation with professional
associations was also proposed to be one of the factors in
assessing journal quality (McCracken & Coffey, 1996).
Normally a journal affiliated with a major society would
have higher visibility than a journal not sponsored by a
society (Bioscience, 1973). Hence, journal properties are
proposed to have an effect on the perceived journal
quality.
(6) Journal accessibility affects perceived journal quality:
A review of the relevant literature suggests that perceived
journal quality is affected directly by journal accessibility
(BioScience, 1973; McCracken & Coffey, 1996; Van
Dalen & Henkens, 1999; Gorman 2000). Dubois and Reeb
(2000) also indicate that there might be a relation between
journal circulation and journal perception ratings. Further,
the abstracting and indexing of a journal by secondary
publication services has been regarded as a possible
journal quality criterion (Zwemer, 1970; BioScience,
1973; Gorman; 2000). Based on prior research, it is
proposed in this study that journal accessibility affect
perceived journal quality.
(7) Journal internationality affects perceived journal
quality: Previous studies discussed journal internationality
with respect to author geographic location and readership
scope (Wormell, 1998; Vastag & Montabon, 2002).
However, the relationship between journal internationality
and perceived journal quality was not explored; if this
relationship had been explored, then, we propose that
journal internationality would have a positive relationship
to perceived journal quality.
Data Collection
(4) Perceived journal quality affects journal citation
impact: According to previous studies, perception of
journal quality is also an external factor, which influence
the number of citations the journal receives. For example,
All the observed variables were collected from the
following sources: Journal Citation Reports (JCR);
Ulrich’s International Periodicals Directory; Science
Citation Index; National Library of Medicine’s
5
LOCATORplus; the World Wide Web; printed versions of
journals, and a survey of perceived journal quality.
In order to produce a set of core journals in clinical
neurology, we invited a clinical neurologist (also a journal
editor of the European Journal of Neurology) to identify a
set of journals for clinical neurology used by specialists
like himself. Among 137 journals in the subject category
of clinical neurology in ISI’s Journal Citation Reports for
the year 2000, 41 journals were identified and confirmed
to be journals largely on clinical neurology rather than on
neurosurgery, neurosciences, or psychiatry – closely
related disciplines.
Data collection is based on the above 41 journals from the
clinical neurology journal subject category for the year
2000. In order to obtain a complete profile for each journal,
those that have changed titles, merged with another
journal or split into two or more journals are regarded as
the same journal. Different data on journal age from
different sources have been resolved by tracing the journal
itself back in time.
A web-based survey of the perceived quality of journals in
clinical neurology was designed to obtain information on
peer
assessment
(see
URL:
http://www.sistm.unsw.edu.au/people/wpyue).
This
survey is the first attempt, to our knowledge, to identify
and evaluate the perceived journal quality in clinical
neurology. In the survey, respondents were asked to rate
the journals on a five-point Likert scale that ranged from
‘Excellent’ to ‘Poor’.
If they were unfamiliar or
otherwise cannot rate the journal, they were asked to
choose ‘Cannot Rate’. Respondents were also asked to
provide demographic or general information about
themselves, such as: research areas, current positions, the
number of recent publications, etc. There were also some
open-ended questions for respondents to write comments
or to add and rate journals that they thought important in
clinical neurology but not included in the list. Although
respondents were encouraged to use the web to complete
and submit the survey, we also provided other options:
completing a Word-version of the survey and returning the
result via E-mail, fax, or postal mail. With the support of
the World Federation of Neurology (WFN), an e-mail
message was sent to members of WFN (approximately
1500 members) inviting them to participate in the survey.
The average values of the ratings for all journals were used
as indicators of perceived journal quality.
Results and Discussion
Survey Results
Within three months of the survey distribution, 252
validated and non-duplicate responses were collected for a
response rate of ca. 17%. Most (238) of the responses were
via the web, 9 via e-mail as Word attachments, 4 via fax,
and 1 via postal mail. The responses were globally
distributed and came from 49 countries; respondents from
the USA ranked first with 19%. The 49 countries were
grouped into major geographical areas and the distribution
of respondents is shown in Table 1.
Table 1. Distribution of Geographic Locations
Geographic
Location
Europe
Latin America
North America
Asia
Middle East
Oceania
Africa
No response
No. (n=252)
%
95
56
50
17
8
7
3
16
37.7
22.2
19.8
6.7
3.2
2.8
1.2
6.3
Over three-quarters of respondents indicated that they are
academics, engaged in teaching, research, and clinical
practice and their current positions are shown in Table 2.
In addition, of the 196 (77.8%) respondents who have
published at least one paper in a peer-reviewed journal
between 1998 and 2003, 42 (21.4%) indicated that they
have published more than 20 papers over the same period
of time.
Table 2. Current Positions
Current Position
No. (n=252)
%
Clinician
79
31.3
Professor
68
27.0
Associate Professor
34
13.5
Assistant Professor
25
9.9
Senior Lecturer
7
2.8
Consultant
5
2.0
Director
5
2.0
Research Fellow
4
1.6
Retried
4
1.6
Sr. Research Fellow
3
1.2
Emeritus Professor
3
1.2
Lecturer
2
0.8
Associate Lecturer
2
0.8
Other
2
0.8
No response
9
3.6
As expected, most (91.3%) respondents indicated that
their principal focus of research is in clinical neurology;
however, most were also engaged in other research areas
such as neuroimmunology, neuropsychology, and
neuroimaging, etc. The distribution of the research areas
of the respondents is shown in Table 3. Respondents were
able to nominate as many research areas as were
applicable. The category ‘Other’ in Table 3 refers to other
disciplines in neurosciences and more specific research
6
areas in clinical neurology (eg. stroke, epilepsy,
movement disorders).
Table 3. Distribution of Research Disciplines
Research Discipline
Clinical Neurology
Neurophysiology
Neurobiology
Neuroimaging
Behavioral Neurology
Neuroimmunology
Neuroradiology
Neuropsychology
Neuropediatrics
Neuropathology
Neurosurgery
Psychiatry
Neuroepidemiology
Neurorehabilitation
Neurogenetics
Other
No. (n=252)
230
116
64
62
48
42
38
31
26
18
14
12
10
5
4
101
%
91.3
46.0
25.4
24.6
19.0
16.7
15.1
12.3
10.3
7.1
5.6
4.8
4.0
2.0
1.6
40.1
In the survey, we asked respondents to ‘position’
themselves into one of the ten-year age groups.
Interestingly, about 42.1% of respondents were in the age
group, 46-55. Respondents in this age group might well be
in their most productive phase. Not surprisingly, the next
two age groups are up to ten years younger or up to ten
years older. Table 4 shows the age group distribution of
the respondents in ranked order.
Table 4. Age Distribution
Age Group
46 - 55
36 - 45
56 - 65
26 - 35
Over 65
Under 26
No response
No. (n=252)
106
65
51
14
13
1
2
%
42.1
25.8
20.2
5.6
5.2
0.4
0.8
Table 5. Distribution of Highest Degree
No. (n=252)
130
81
%
51.6
32.1
10
9
6
2
10
4
4.0
3.6
2.4
0.8
4.0
1.6
Structural Model
PLS-Graph (version 3.00) was used to analyse the data.
PLS has no requirement of the distributional assumption,
and “evaluation of the PLS models should apply
prediction-oriented
measures
that
are
also
nonparametric” (Chin, 1998, p316). Hence, the multiple
R2 values given for the dependent construct is used to
assess the predictiveness. It is equivalent to the value
derived from the regression, which indicates the fraction
of total variance in the dependent construct accounted for
by those independent constructs (Chin, 1998; Mathieson
et al., 2001). In our study, the R2 of the journal citation
impact is 0.798, showing that the four external factors
explained 79.8% of the variance of the journal citation
impact.
J. Properties
0.464**
Perceived
J. Quality
J. Accessibility
J. Internationality
Table 5 shows that the majority (88.5%) of respondents
hold at least one doctoral degree. 83.7% have M.D.
degrees and of these, about two-fifths also have Ph.D
degrees. The category ‘Other’ in Table 5 refers to various
fellowships, e.g., FRACP, FRCP, FRCPI, and MRCP.
Degree
M.D.
M.D.+Ph.D.
Ph.D.
M.B.B.S
M.A./M.Sc.
DM
Other
No response
0.243*
0.499**
Journal
Citation
Impact
0.311**
(*Significant at 0.05 level; **Significant at 0.01 level)
Figure 7: Results of Structural Model
Moreover, we chose “bootstrapping” to estimate the
precision of our PLS estimates. Overall, four path
coefficients were found to be significant at the 0.05 level;
T-test for these four path coefficients are all >1.65, thus
providing support for the corresponding research
propositions 2, 3, 4, and 5. However, no significant
influence was found for journal properties to journal
citation impact, nor for journal accessibility and journal
internationality to perceived journal quality. All the path
coefficients between the research constructs are expressed
in a standardized form to permit comparison of the relative
strengths. The results of this study support the assumption
that certain external factors influence the journal citation
impact. In particular, journal accessibility (path
coefficient of 0.499) has an important influence on the
journal citation impact while journal internationality
(0.311) and perceived journal quality (0.243) have only
moderate influences on the journal citation impact.
However, our study shows that journal properties have no
7
direct influence (0.013 and not significant) on journal
citation impact. Nevertheless, journal properties with a
path coefficient of 0.464 demonstrate a strong influence
on perceived journal quality.
Measurement Models
As to the measurement models, PLS also enables the
assessment of measurement components by providing
loadings and weights of indicators. In general, weights are
appropriate for interpreting the effects for formative
indicators while loadings are more suitable for
examination of reflective indicators. Additionally,
bootstrap resampling also examines the significance of the
weights and the loadings (Chin, 1998).
Formative indicators are weighted according to their
relative importance in forming the construct. Thus, the
weights allow us to determine the extent to which each
indicator contributed to the development of the construct.
The general approach is to compare the weights of
different indicators and to see if the weights are significant
(Sambamurthy & Chin, 1994). Table 6 shows the weights
for all formative indicators in the model, thus, showing
extents of the reliability of the indicators for all the
formative constructs. However, when some weights
include the necessary constant to complete the PLS
estimation procedure, weights can be >1. In our case,
author internationality has a stronger weight.
Table 6. Formative constructs, indicators, and weights
Construct
Journal
Properties
Journal
Accessibility
Journal
Internationality
Perceived
Journal Quality
Indicator
Journal size
Journal age
Frequency
Number of affiliations
Number of reviews
Multidisciplinary
Language
Circulation
No. of A&I services
Author internationality
Editorial
internationality
Peer Assessment
Weight
0.727**
0.088
0.290
0.020
0.013
0.247*
0.108
0.777**
0.532**
1.046**
0.367**
1.000
* Significant at the 0.05 level (T-Statistic is above 1.65).
** Significant at the 0.01 level (T-Statistic is above 2).
On the other hand, each reflective construct is viewed as
an existing entity. The extent to which each indicator
reflects the underlying construct can be determined by
examining the loadings. The rule of thumb is that the
standardized loadings should be >0.707 (Chin, 1998).
Moreover, for reflective constructs, we need to assess the
internal consistency and discriminant validity. Composite
reliability calculated by PLS is suitable for assessing the
internal consistency while Average Variance Extracted
(AVE) is appropriate for checking discriminant validity. It
is suggested that the AVEs of LVs should be >0.500 and
the composite reliability should be >0.700 (Chin, 1998).
Overall, Table 7 shows that all loadings of the indicators
for journal citation impact are >0.800 with a composite
reliability of 0.947 and an AVE of 0.749, thus confirming
our construct design of journal citation impact. The results
also suggest that, for an integrated conceptual model of
journal citation impact, various citation indicators should
be used.
Table 7. Reflective construct, indicators and loadings
Construct
Journal
Citation
Impact
Indicator
Loading
Impact factor
Immediacy index
Total citations
Cited/Citing ratio
Citing disciplines
Citing Author GL
0.822**
0.850**
0.885**
0.903**
0.853**
0.876**
** Significant at the 0.01 level (T-Statistic is >2).
Overall Model Discussion
Our finding that journal accessibility has a significant
influence on journal citation impact is consistent with
discussions in prior research indicating that physical
inaccessibility to journals would hinder their use and
subsequent citings to them (Dubios & Reeb, 2000;
Gorman, 2000; Van Dalen & Henkens, 1999; Cooper et al.,
1993; Dombrowski, 1988). It also supports statements that
the number of A&I services indexing a journal could be an
indicator for an increase number of citations (Cooper et al.,
1993). Among the formative indicators of journal
accessibility, data in Table 6 confirm that circulation and
number of A&I services are influential factors in forming
journal accessibility. However, language contributes little
to the formation of journal accessibility. The reason for
this result might be due to the fact that most journals
covered in our research are in English.
Journal internationality had a medium influence on journal
citation impact, thus confirming our proposition 3; in
particular, the author internationality contributed greatly
(1.046) to the journal internationality construct. This
finding confirms a previous study showing that
internationalization indicators correlated moderately with
journal impact (Zitt & Bassecoulard, 1998). Likewise,
editorial internationality with a weight of 0.367 also
contributed significantly in forming the construct of
journal internationality. Overall, in terms of geographic
location, our study suggests that the broader the
8
distribution of authors and editorial board members, the
higher the citation impact of the journal.
Another significant relationship in our model is between
perceived journal quality and journal citation impact; this
is consistent with discussions in prior research indicating
that perceived reputation or prestige of a journal positively
affects the journal citation impact (Dieks & Chang, 1976;
Anderson & Goldstein, 1981; Johnson, 1997). However, it
is surprising to see that perceived journal quality is not the
most significant factor influencing journal citation impact.
In this situation, there may exist two possible scenarios.
Firstly, it is possible that a few high ‘impact’ articles in
some less prestigious journals attracted a large number of
citations and subsequently inflated their journal citation
impact; however, the peer perception of these journals
remained the same. Secondly, we found that respondents
from different geographical regions rated certain journals
very differently. Thus, the variation among respondents
due to their geographical location and the subjectivity of
the survey results may account for the relatively low
influence from perceived journal quality to journal citation
impact.
Although, there is no statistical significance between
journal properties and journal citation impact, we found
that journal properties have a strong influence on
perceived journal quality. Thus, through perceived journal
quality, journal properties might have an indirect effect on
journal citation impact. Among all the indicators of
journal properties, journal size (0.727) certainly
contributed greatly to the formation of the construct. The
number of disciplines a journal covers also significantly
contributed to the journal properties construct. However,
as shown in Table 6, journal age, frequency, the number of
affiliations and the number of review articles did not
contribute significantly to the development of the journal
properties construct. Therefore, the most significant
indicators influencing perceived journal quality are
journal size and journal multidisplinarity.
As we move into an electronic environment without
‘walls’ or ‘boundaries’, it is not surprising to find that
journal accessibility and internationality do not affect the
perception of journal quality. Our findings are not
consistent with prior speculations that accessibility could
influence perception (McCracken & Coffey, 1996;
Todorov & Glanzel, 1988; Van Dalen & Henkens, 1999;
Dubois & Reeb, 2000; Gorman 2000). However, our
research goes beyond speculation; we have used empirical
data to investigate the relationships. Additionally, our
research also found that the international scope of authors
and editors does not affect perception of journal quality.
Conclusion
In conclusion, inter-relationships among journal citation
impact and four external journal factors have been
successfully explored. Nevertheless, there are some
limitations related to measurement reliability and research
validity. Firstly, there are missing values in the circulation
variable. Also, we need to bear in mind that circulation
figures obtained from Ulrichs may not completely reflect
the actual number of readers or users, particularly in the
case of organizational subscriptions; nor are the figures
always updated (upwards or downwards) from year to
year. Secondly, the survey itself has limitation in having
subjectivity and measurement errors. Further, like any
statistical tool, PLS itself also has its bias in obtaining
parameter estimates. The bias can be minimized by
increasing the sample size and at the same time the
number of indicators (Chin, 1998). Finally, other potential
equivalent models are possible. Hence, it is suggested that
issues concerning cultural and disciplinary biases need to
be considered before generalizing our research findings.
We suggest that future research be extended to other
disciplines and to more journal sets.
ACKNOWLEDGMENTS
We thank Dr. François Boller for his valuable comments
on our survey design. We would also like to thank the
World Federation of Neurology and its members for their
support and cooperation. Finally we thank the two
anonymous reviewers for their constructive comments.
REFERENCES
Ali, S.N., Young, H.C., & Ali, N.M. (1996). Determining the
quality of publications and research for tenure or promotion
decisions: a preliminary checklist to assist. Library Review, 45,
39-53.
Anderson, P.J., & Goldstein, R.K. (1981). Criteria of journal
quality. Journal of Research Communication Studies, 3,
99-110.
BioScience Information Service. (1973). Biological Abstracts,
56.
Brooks, C.H., Walker, L.R., & Szorady, R. (1991). Rating
journals in health-care administration - the perceptions of
program chairpersons. Medical Care, 29, 755-765.
Chin, W. W. (1998). The partial least squares approach for
structural equation modelling. In George A. Marcoulides (Ed.),
Modern Methods for Business Research (pp. 295-336). New
Jersey: Lawrence Erlbaum Associates.
Cooper, R.B., Blair, D., & Pao, M. (1993). Communicating MIS
research - a citation study of journal influence. Information
Processing & Management, 29, 113-127.
Cozzens, S.E. (1989). What do citations count? the rhetoric first
model. Scientometrics, 15, 437-447.
Cronin, B. (1984). The citation process: the role and significance
of citations in scientific communication. London: Taylor
Graham.
Dieks, D., & Chang, H. (1976). Differences in impact of
scientific publications - some indexes derived from a citation
analysis. Social Studies of Science, 6, 247-267.
Dombrowski, T. (1988). Journal evaluation using Journal
9
Citation Reports as a collection development tool. Collection
Management, 10, 175-180.
Dubois, F.L., & Reeb, D. (2000). Ranking the international
business journals. Journal of International Business Studies, 31,
689-704.
Garfield, E. (1979). Citation indexing - its theory and application
in science, technology, and humanities. New York: Wiley.
Gilbert, G.N. (1977). Referencing as persuasion. Social Studies
of Science, 7, 113-122.
Gorman, G.E. (2000). Journal quality in the Asian region:
Results of a pilot study for the IFLA round table of library and
information science journals. Retrieved January 15, 2004 from
http://www.ifla.org/IV/ifla66/papers/156-125e.htm.
Gutiérrez, J., & López-Nieva, P. (2001). Are international
journals of human geography really international? Progress in
Human Geography, 25, 53-69.
Johnson, D. (1997). Getting noticed in economics: The
determinants of academic citations. American Economist, 41,
43-52.
Jöreskog, K.G., & Wold, H. (1982). The ML and PLS techniques
for modeling with latent variables - Historical and comparative
aspects. In: Jöreskog, K.G. and Wold, H., (Eds.), Systems under
indirect observation: causality, structure, prediction Part I (pp.
263-270). Amsterdam; New York; Oxford: North-Holland
publishing company.
Kim, M.T. (1992). A comparison of three measures of journal
status: influence weight, importance index, and measure of
standing. Library and Information Science Research, 14,
75-96.
Mathieson, K., Peacock, E., & Chin, W.W. (2001). Extending the
technology acceptance model: The influence of perceived user
resources. Data Base for Advances in Information Systems, 32,
86-112.
McCracken, M.J., & Coffey, B.S. (1996). An empirical
assessment of health care management journals - a business
perspective. Medical Care Research and Review, 53, 48-70.
Nisonger, T.E. (2002), The relationship between international
editorial board composition and citation measures in political
science, business, and genetics journals. Scientometrics, 54 :
257-268.
Price, D.J.D.S. (1981). The analysis of square matrices of
scientometric transactions. Scientometrics, 3, 55-63.
Ren, S.L., & Rousseau, R. (2002). International visibility of
Chinese scientific journals. Scientometrics, 53, 389-405.
Rousseau, R., & Hooydonk, G.V. (1996). Journal production and
journal impact factors. Journal of the American Society for
Information Science, 47, 775-80.
Sambamurthy, V., & Chin, W.W. (1994). The effects of group
attitudes toward alternative GDSS designs on the
decision-making performance of computer-supported groups.
Decision Sciences, 25, 215-241.
Smart, J.C., & Elton, C.F. (1981). Structural characteristics and
citation rates of education journals. American Educational
Research Journal, 18, 399-413.
Todorov, R. and Glanzel, W. (1988). Journal citation measures a concise review. Journal of Information Science, 14, 47-56.
Van Dalen, H.P., & Henkens, K. (1999). How influential are
demography journals? Population and Development Review,
25, 229-251.
Van Leeuwen, T.N., Moed, H.F., Tijssen, R.J.W., Visser, M.S., &
Van Raan A.F.J. (2001). Language biases in the coverage of the
Science Citation Iindex and its consequences for international
comparisons of national research performance. Scientometrics,
51, 335-346.
Vastag G., & Montabon F. (2002). Journal characteristics,
rankings and social acculturation in operations management.
Omega - International Journal of Management Science, 30,
109-126.
Wilson, C.S. (1999). Informetrics. Annual Review of
Information Science and Technology, 34, 107-247.
Wold, H. (1980). Model construction and evaluation when
theoretical knowledge is scarce - Theory and applications of
Partial Least Squares. In: Kmenta, J. & Ramsey, J.B., (Eds.),
Evaluation of econometric models (pp. 47-74). New York:
Academic Press.
Wormell, I. (1998). Informetric analysis of the international
impact of scientific journals: how 'international' are the
international journals?
Journal of Documentation, 54,
584-605.
Yue, W.P., & Wilson, C.S. (2003). Measuring the citation impact
of research journals in clinical neurology: a structural equation
model analysis. In G. Jiang, R. Rousseau, & Y. Wu (Eds.),
Proceedings of the 9th International Conference on
Scientometrics and Informetrics (pp. 362-368). Dalian: Dalian
University of Technology Press.
Zinkhan, G.M., & Leigh, T.W. (1999). Assessing the quality
ranking of the journal of advertising, 1986-1997. Journal of
Advertising, 28, 51-70.
Zitt, M., & Bassecoulard, E. (1998). Internationalization of
scientific journals: a measurement based on publication and
citation scope. Scientometrics, 41, 255-271.
Zsindely, S., Schubert, A., & Braun, T. (1982). Editorial
gatekeeping patterns in international science journals. A new
science indicator. Scientometrics, 4, 57-68.
Zwemer, R.L. (1970). Identification of journal characteristics
useful in improving input and output of a retrieval system.
Federation Proceedings, 29, 1595-1604.
10