Evaluating Journals and the Institutions that Publish in Them Cabell’s Adds a New Dimension These ranking systems will assist users when: • Evaluating the importance of a journal in furthering future research • Determining the degree of difficulty an author encounters when seeking to publish in a journal Classifications of Methods to Evaluate the Importance of Journals Classifications of Methods to Evaluate the Importance of Journals • Journal Citation Count - number of times an article has been cited in a publication • Online Usage – citations, tweets, blog posts, news stories that mention the article • Peer Review - panel members make judgments about research published in journals Criteria for Evaluating these Approaches • Reliability • Validity • Usefulness Why Peer Review is Not Enough • Inherent subjectivity lessens reliability • Reviewers may have limited knowledge • May lack framework for comparing the journals Why Online Usage is Not Enough • Does not meet the validity criteria • Tweets, blog posts and news stories only indicate awareness • Number of downloads does not indicate usefulness Methodological Criticisms of Citation Counts • Emphasis on the journal cited not the article • Approach does not apply to more recent digital scholarship sources • Citation counts slow to accumulate and update Further Methodological Criticisms • Reasons to cite other than develop a framework: – Refute previous research – Political reasons – Curry favor with editors/reviewers • Vary among disciplines • Algorithms and statistical methods are difficult to understand Why is the Cabell’s Approach Better? • Classification • Inclusion • Simplicity Cabell’s Classification Index (CCI) Cabell's Classification Index (CCI) • Used to determine the influence of a journal – Frequently cited journals are considered to have more influence on future research – Journals that have been published for five years or more are considered to influence future research Cabell's Classification Index (CCI) • Using citation counts from the most recent 3 years: – Average citation count for each journal was calculated – Reduces the importance any one article had on the journal’s citation count – Gives more importance to recent published research relative to articles published more than 3 years ago – The total weighted score of each journal was normalized using its Z score Cabell's Classification Index (CCI) Classifies the influence of a journal into 5 categories: Cabell’s Classification Index Z Score Classification 1.282 or higher Premiere 0.842 - 1.281 Significant Influence less than or equal to 0.841 High Influence No citation count/Published for a period of 5+ years Qualified No citation count/Published for a period of less than 5 years Novice Number of Journals & Average Citation Count Journals with Citation Counts Accounting C/S Business Info System Economics & Finance Management Marketing Average Significant High Premiere Influence Influence Qualified Novice 126 948 4.6455 6.8343 2.6312 4.1645 0.8182 1.5009 n/a n/a n/a n/a 687 4.7181 2.7731 0.9329 n/a n/a 1,095 256 5.4852 5.2062 3.1188 3.1724 1.0039 1.1155 n/a n/a n/a n/a 5.37786 3.17196 1.07428 n/a n/a Sample Journal Entry with CCI Sample Journal Entry with CCI Number of Journals Listed in Cabell’s and/or Scopus Total Listed in Scopus Total Listed in Cabell’s Accounting 126 593 Computer Science – Business Information System Economics & Finance 948 1,593 687 1,494 Management 1,095 2,206 Marketing 256 758 Total 3,112 6,644 Journal Longevity • CCI calculation includes journals that have been published for at least 5 years • Longevity of publication indicates the journals have influence on research • Publication history makes the journals familiar to researchers • Permits researchers to use previous issues to determine the journal’s usefulness Benefits of the CCI • Allows journal rankings to be compared across disciplines • Accounts for some variation in the calculations without altering perceived influence • Provides a more in-depth description of the journals Cabell’s Institutional Publishing Activity Index (IPA) Why Acceptance Rates Are Not Enough • 3 Issues – Methodology used to compute acceptance rates differ among journals – Publishers are reluctant to provide acceptance rates – Specialized focus of a journal may limit the number of people who can publish in that journal Cabell’s Institutional Publishing Activity Index (IPA) • Used to improve the process of evaluating institutions publishing in journals – Recognize the differences in research resource availability among institutions – Such factors result in differences in the publishing effectiveness of institutions Cabell’s Institutional Publishing Activity Index (IPA) • Using citation counts from the most recent 3 years: – The counts in each classification were weighted by the average citations per document (article) – Compensates for smaller percentages of journals in Premiere (10%) and Significant (10%) groups relative to the number of journals in High Influence (80%) – Recognizes increased importance due to frequency of citation of journals in Premiere and Significant groups – The total weighted score of each institution was normalized using its Z score Cabell’s Institutional Publishing Activity Index (IPA) Classifies the publishing activity of an institution into 4 categories: Cabell’s Institutional Publishing Activity Index Z Score Classification 1.282 or higher Recognized Publishing Activity 0.842 - 1.281 Significant Publishing Activity less than or equal to 0.841 High Publishing Activity Institutions Accredited by National Accreditation Associations, but whose faculty members publish in journals without citation counts Accredited Accrediting Agencies Directory Association Accounting AACSB International Computer Science-Business Information Systems AACSB International Economics and Finance AACSB International Education Curriculum and Methods NCATE Educational Psychology and Administration NCATE Educational Technology and Library NCATE Science Health Administration ALA Management AACSB International Marketing AACSB International Nursing AACN Psychology APA Cabell’s Difficulty of Acceptance Index (DA) Cabell’s Difficulty of Acceptance Index (DA) • Used to determine the varying degrees of difficulty authors encounter in seeking to have their manuscripts accepted by journals – Institutions whose faculties frequently publish are likely to have advantages in resources – It can be more difficult for authors to publish in journals which frequently accept articles from “Recognized” institutions (IPA) Cabell’s Difficulty of Acceptance Index (DA) • Using citation counts from the most recent 3 years: – Classifies the challenge of having manuscripts accepted by journals into three categories – Each journal’s Z score is calculated, and rankings are based on Z score Cabell’s Difficulty of Acceptance Index (DA) Cabell’s Difficulty of Acceptance Index Z Score Difficulty of Acceptance 1.282 or higher Rigorous 0.842 - 1.281 Significantly Difficult Less than or equal to 0.841 Difficult Cabell’s Difficulty of Acceptance Index (DA) Accounting Computer Science & Business Info System Economics & Finance 0.6753 0.7751 Significantly Difficult 0.4811 0.6581 0.7718 0.6649 0.4126 Management Marketing 0.8407 0.7520 0.7394 0.6244 0.4785 0.37634 Average 0.7629 0.6335 0.3823 Rigorous Difficult 0.2082 0.436 CCI Summary • Cabell’s Classification Index (CCI) – Includes a larger number of journals than other databases that emphasize citation counts – Makes citation counts user-friendly – Allows comparison of journals across disciplines – Recognizes the relative influence of journals IPA Summary • Cabell’s Institutional Publishing Activity Index (IPA) – Recognizes that the institution plays a major role in the faculty’s publishing activity – Accounts for disparities in the budgets available for research, equipment, facilities, faculty support, research assistance, the rewards for publishing, and the capabilities of individual faculty members DA Summary • Cabell’s Difficulty of Acceptance Index (DA) – A measure of an individual’s capability to publish – Those journals labeled as “Rigorous” accept more manuscripts submitted by authors at “Recognized” institutions than institutions not labeled as “Recognized” Cabell’s Login http://www.cabells.com Members Page Login User ID: Singapore Password: Cabell’s