Natural Research Assessment:
ability and new requirements for
CRIS
Sergey Parinov,
CEMI RAS, Russia, Moscow
Natural Research Assessment is …
– an inherent part of science communication,
since it is a basic consequence of existed labor
division and knowledge exchange between
scientists
– a practical “testing” of research outputs by
scientists when they are trying to use it for
producing new scientific knowledge
– a statistical portrait of total trials-and-errors for
some researchers, their results, etc.
• Can we practically use it? If yes, how?
Natural Assessment in
Social Systems
• All social systems, where independent actors are working
with labor division and products exchange between them,
naturally produce trial-and-error usage statistics
• If in the system the practical testing of is doing well and
trial-and-error data is available for actors, so the system
provides for actors useful natural assessment mechanism
and statistics
• By such assessments actors know how their activity are
currently valuable for the system and what/whose
activities are the most useful now
• Than better such signaling mechanism works, than the
system is more efficient
Scientific Social System Features
• Science is a social system with labor division and
knowledge exchange between scientists in a trialand-error form
• Trial-and-error usage statistics are mostly latent
since researchers are testing “units of thought”
hidden inside publications which are currently
visible form of scientific exchange
• Scientists can get trial-and-error data mostly in a
form of lists of publications selected by publishers
and citations, what is really not enough
• Current scientific practical testing (trial-and-error)
process is doing not very well and compare with
many other social systems the Science has weaker
signaling system and its efficiency can be improved
Current Role of CRIS
(e.g. socionet.ru CRIS)
• CRIS is an interface for scientists to give
their research outputs for common usage
and to take “products” from other scientists
for their personal usage
• CRIS is working as a scientific circulation
mechanism, since it delivers “products”
from “producers” to “consumers”
• CRIS is a tool of practical testing of
research outputs and virtual environment
for scientific trial-and-error acts
Specification of Tasks
• Strong natural assessment and professional signaling
system can be designed by:
– discovering of what is a matter (object) of scientific trial-and-error
process in producing a new knowledge
– efficient scientific circulation mechanism, which involves all
objects into representative trial-and-error process
– registering and collecting all existed trial-and-error data with
presenting them as publicly available/visible statistics
– the statistics should show both side of scientists’ activity within a
network of professional relations: as a “producer” and as a
“consumer” of research outputs
• We should exploit typical motivations and scientific
behavior models: a wish to get professional recognition
and existed norms to provide professional reviews
Trial-and-error process within CRIS
• An author registers research output as a ready
for testing scientific object-for-reuse (OfR) by
– specifying which research materials were used as
roots/basements for his/her output (citation links)
– specifying materials/scientist where/by whom the
output could be used/reviewed (links to possible
users)
• Authors of linked materials receive a notification
about created links, and
– they can protest on how the materials were used
– they can use suggested OfR and link it with their
materials, or can review it, or can ignore it
Scientific Communications
within CRIS
• On a producer side
– specify used OfR with
quality characteristics
– request on usingreviewing own OfR by
linking it with other OfR
or scientists
• On a consumer side
– protest against usage
characteristics and/or
provide comments on it,
or do nothing
– ban requests from some
authors, or specify
personal reviewing rate,
or rewrite own OfR by
using/citing suggested
OfR
Trial-and-error statistics within
CRIS
• Accumulated data about scientist-producer activity
– number of produced OfR for certain period of time
– number of OfR usage for certain period of time, including quality
characteristics distribution
– number of OfR reviews for certain period of time, including
quality characteristics distribution
• Accumulated data about scientist-consumer activity
– number of requests from scientists on using/reviewing their OfR
– number of made citations (utilized OfR), including quality
characteristics distribution
– number of made reviews, including quality characteristics
distribution
– number of banned authors, queue length of requests compare
with personal rate, number of rejections
Who will benefit?
• A scientist –
– by data on how people used his/her outputs and by
usage request as some can be valuable
• A research institution –
– by data on staff’s produced/used outputs, etc.
• A research funder –
– by data on produced/used outputs, performance
statistics about researchers and organizations
• A research community –
– by better scientific circulations, usage and professional
signaling system
Conclusion
• We are implementing natural research
assessment scheme within Socionet CRIS
(socionet.ru) as a pilot to allow a playing
with it
• We suggest for euroCRIS to include a
support of natural research assessment
into CRIS-CERIF model
Thanks!
Necessary Role of CRIS
• Electronic registration research
outputs/results focused on its usage
• Tools to request usage/review/evaluation
from a community and/or its members
• A form to make usage/review/evaluation in
easy way
• Taking statistics on research performance
of scientist in a role of a producer and a
consumer research outputs/results
Requirements for new CRIS
features
• To operate with OfR as citation and artifact
information objects
• To make semantic links (by usage/impact
metrics) with CRIS objects (material, person,
etc.)
• To run monitoring of linkages and notification
services
• To give scientist an ability to specify a personal
norm of reviewing efforts
• To gather data, collect and process trial-anderror statistics
Background
• Open Science as a conceptual platform
– 1 open access to research
– 2 open access to usage and impact data
– 3 open access to basic assessment data
• CRIS Socionet as technological platform
– open archives and IR
– scientific circulation mechanism
– monitoring and statistics services