Multi-Dimensional Factors in
Academic Research Evaluation
Xiaodong Zhang
Ohio State University
Impact of Strong Research to Universities

A foundation of improving undergraduate and
graduate programs


Timely updating contents in classrooms
Creating research environment and opportunities to
students
 Bringing competitiveness and recognition


New knowledge, discoveries, and innovations.
Continue to attract strong students, excellent faculty, and
research grants.
How is Research Measured?

Research Production is Typically Measured by





Publications in refereed conferences and journals
Software and prototypes
Invited lectures
Patents
Research Awards and honors
 Input of Research



Research proposal writing, and research project planing
Research grants
Recruiting and forming teams.
How is Research Impact Measured?

Research results are useful






Citations of the publications
Widely used software and prototypes
Technology transfer from original research results
Useful patents
Influence in the research community
Leading new directions in the field
 The quality of research is essential


Number of publications should not be a major metric
Our research should be Impact-driven
Where to Publish?

Why publish in top venues?


Each field has its own flagship and leading conferences/journals
defined by reputations
Most researchers only read papers there.
 Conferences versus Journals

In certain fields, such as systems, architecture, networking, and
databases, top conference papers are prestigious, highly visible,
with 4-5 or more reviews in depth, and low acceptance rate.
 SCI and EI should NOT be used as a guidance



An SCI/EI entry does not necessarily reflect its quality in the field
The quality should be judged by the peers in the field.
The impact factor sometimes is field-size dependent.
Establishing Identity and Reputation

Conducting research on focused topics in depth

Making names behind some important innovations
 US NSF Career Program for Junior Faculty

A 5 year research/education plan, aiming at conducting
research to solve a small set significant problems.
 A strong department should have several identities


Paying attention to building on existing strength.
Rewarding to strong research identity/reputation
What is an ``Acceptable” Research Production?

How to quantify research productions?




Number vs. quality of publications
Research grants/expenditure (mission vs. basic research)
Supervising Ph.D. students: productivity and quality of
placements.
Impact of research after many years.
 Conference/journal organizations

Not part of the research, but professional service.
 Each department has its formula

Resources dependent (teaching load, et. al.)
Effects of University Rankings





Rankings by society have to be a consideration
Rankings significantly affect student recruiting
University rankings influence parents
Faculty recruiting is also affected.
However, these rankings are a reference not an
absolute indicator.
Major Sources of University Ranking in US
 US News & World Reports (University ranking)



Published every August.
Based on 6 criteria to quantify comprehensive ability.
Quality of undergraduate students is an important consideration.
 US News & World Reports (Graduate school ranking)



Published every March.
Different school rankings are based on different disciplines.
Quality of graduate programs is the major factor.
 National Science Foundation (funding statistics)


Research expenditure of last year published every August
Ranking the research scale of universities.
 National Research Council (Department ranking)


Published every 10 to 15 years
Ranking all the Ph.D. granted departments in all majors
US News & World Reports’ 6 Criteria
 Undergraduate student selectivity


SAT scores, high school ranking, and other activities.
Acceptance rate.
 Faculty academic reputation

Research activities and accomplishments
 Faculty-student ratio

Reflecting the efforts and attention to students, e.g. class size.
 Student retention and graduation rate

Seriousness of the students and management of curriculum.
 Financial resources

University revenue and endowed funds.
 Alumni donation rate

Reflecting graduates’ proud, passion, and care to the university.
Major Themes of Graduate Program Ranking
 Scholarships and impacts


Quality of faculty publications and their influence to the field
Citations rates and technology transfers
 Ph.D. student production and placement

How many per year, and where they go after graduations.
 Ph.D. alumni achievements

The group distinguishing themselves in the fields.
 Faculty resources

Award winners, members of the academies, and others.
 Research scale

Measured by the amount of expenditure by external grants
Foundations of Academic Excellence
 Excellent undergraduate programs



Highly selected students with comprehensive talents.
Providing highly quality education and research environment.
All top U.S. universities have top undergraduate programs.
 Excellent research and education infrastructure

Students are able to effectively learn and research.
 Excellent faculty

World-class professors who are well established in their fields.
 High standards in research and education

Rigorous training to students and doing impact driven research.
 Ambitious students who will play leadership anywhere


Enter: grow the wisdom; Depart: serve the better country and the
kind.
Should not be too narrowly focused.
Mr. Hu’s 4 Conditions of an ``Established University”
 In 1947, Hu Shi urged the government to invest 5
existing universities to make them be ``established
and independent” in the world within 10 years.
 1. Providing first-class education environment.
 2. Being able to retain young researchers.
 3. Being able to solve country’s problems in
science, industries, health, and national defense.
 4. Being able to collaborate with foreign scientists
to solve open and hard problems.