Tenth International Workshop CIA 2006, Edinburgh
Evaluating Dynamic
Services in Bioinformatics
Maíra R. Rodrigues
Michael Luck
University of Southampton, UK
Outline








2
Bioinformatics
Agents and Bioinformatics
Model for Cooperative Interactions: Overview
Requirements for Service Evaluation
Evaluation Method
Example Scenario
Conclusion
Future Work
ECS - University of Southampton
CIA 2006
Bioinformatics

Bioinformatics


Bioinformatics Services




Heterogeneous
Locally and remotely used
Continuous update
Management and analysis of biological
data and tools

3
Application of computer technology to manage
and analyse biological data
Suitability of an agent-based approach
ECS - University of Southampton
CIA 2006
Bioinformatics


Interrelated data
Cooperative applications


Services free of charge


4
Participants request and
provide services to each
other
Non-economic exchange of
different types of tools and
data
Interactions are based on
reciprocal relations
ECS - University of Southampton
CIA 2006
Agents and Bioinformatics

The agent-based approach:




Therefore..

5
Agents provide and request bioinformatics
services
Existence of alternative providers
Services are provided with different levels of
quality (heterogeneity)
Agents need to select service providers
ECS - University of Southampton
CIA 2006
Agents for Interaction

Agent-based applications in
bioinformatics:


Our concern:



6
Concerned with high-level management tasks
Model non-economic cooperative interactions
Evaluation method for bioinformatics services
to determine an agent’s satisfaction
Guide agent’s decisions over service
providers
ECS - University of Southampton
CIA 2006
Model for Cooperative
Interactions

Model non-economic cooperative interactions
based on exchange values (Piaget 1973)
• effort
• credit
7
A1
A2
service
ECS - University of Southampton
• satisfaction
• debt
CIA 2006
Model for Cooperative
Interactions

Model non-economic cooperative interactions
based on exchange values (Piaget 1973)
• effort
A1
A2
• credit
• credit
• satisfaction
8
• satisfaction
• debt
• debt
A1
A2
ECS - University of Southampton
• effort
CIA 2006
Model for Cooperative
Interactions

Exchange values result from the agent’s
evaluation of the service
Service
Evaluation
Exchange Values
Partner Selection
(future interactions)
9
ECS - University of Southampton
CIA 2006
Model for Cooperative
Interactions

Exchange values result from the agent’s
evaluation of the service


Exchange values (Rodrigues, Luck 2005, 2006)
Current work focus on service evaluation
Service
Evaluation
Exchange Values
Partner Selection
(future interactions)
10
ECS - University of Southampton
CIA 2006
Service Evaluation

Bio-Services are dynamic:




Evaluation requires



11
Constant updates
Regular behaviour, but
Sensitive to different parameter configuration
Repeated evaluation
Attach context information
Evaluation of different aspects of the service
ECS - University of Southampton
CIA 2006
Service Evaluation

Evaluation method should address:




12
Generality: apply to different types of bioservices and aspects of these services
Continuity: repeat evaluation every time a
service is received
Consistency: compare evaluations made at
different points in time
Discriminated information: allow flexible
decision-making by using evaluation of
individual aspects or a global evaluation
ECS - University of Southampton
CIA 2006
Alternative Approaches

Quantitative approaches


Qualitative approaches:

13
Scoring or utility functions
• Objective values
• Precision, consistency, combination is
straightforward
Classification rules (e.g., poor, good, excellent)
• Subjective values
ECS - University of Southampton
CIA 2006
Evaluation Method

Choose evaluation attributes for service


examples: performance, quality, reliability,
etc.
For each attribute, associate result
measures


Pieces of information derived from service
result that can determine the service utility in
relation to an attribute (observed value).
Static or dynamic measures (e.g., quality of
interface and response time)
14
ECS - University of Southampton
CIA 2006
Evaluation Method

General evaluation function for
evaluation attributes (utility):

For a set of attributes A = {a1,..,ai}
result measure for ai

Ui = bc
evaluation strictness
1
Ui
0
c
15
ECS - University of Southampton
CIA 2006
Evaluation Process

Before evaluation:


Repeat evaluation process every time a service
is received



16
Identify evaluation attributes for services and result
measures for each attribute
Input is the service result and configuration used
For each evaluation attribute ai
• Compute result measures
• Calculate evaluation Ui
• Store evaluation
Output is a set of evaluations (evaluation tuple)
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Proteomics research





17
Protein identification services
• Input: file (list of unknown peptides)
• Process: database + matching algorithm
• Output: list of proteins, peptides per protein
Services: OMSSA, MASCOT, Tandem Local and Remote
Heterogeneous results for same input data
Sensitive to different input configurations
Evaluation can be used as criterion for future
selection
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Evaluation attributes:



18
Sensitivity
• Capacity of matching related proteins
Accuracy
• Capacity of identifying true matches
Performance
• Time taken from input submission until
result is received
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Result measures (rm):

Sensitivity
• Number of proteins
• Peptide ratio - peptides per protein
• Influence of input size
• Increasing utility
rm =
19
input_size
peptide_ratio x protein_number
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Accuracy
• Number of false positives
• Decreasing utility
rm = false_positives

Performance:
• Response time
• Influence of input size
• Decreasing utility
response_time
rm =
input_size
20
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Evaluation functions:
Sensitivity (U1):

Ui = 0.5rm
• U1 increases with peptide_ratio and
protein_number
Accuracy (U2):
• U2 decreases with false_positives
Performance (U3):
• U3 decreases with response_time
21
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Practical evaluation:



Evaluation of sensitivity

22
Same input spectra
Two different configurations (C1 and C2)
Evaluation reflects different results for C1 and C2
ECS - University of Southampton
CIA 2006
Evaluating Bio-Services

Evaluation of performance

Again, evaluation reflects different
results for C1 and C2
23
ECS - University of Southampton
CIA 2006
Conclusions


Present an evaluation method to be
used by agents requesting dynamic
services in bioinformatics
Discussion of issues for efficient
evaluation of these services, including




24
Adoption of a repeated evaluation process
Absolute evaluations
Generation of individual and compatible evaluations
Single evaluation must be calculated during selection
ECS - University of Southampton
CIA 2006
Conclusions



25
Show the application of the evaluation
method for protein identification services
Importance of dynamic (repeated)
evaluation is shown through empirical
results
Provide more accurate information for
agents that need to select services with
dynamic characteristics
ECS - University of Southampton
CIA 2006
Future Work

Develop selection strategies that use
and combine service evaluations




Combination through objective and
subjective values
Probabilistic analysis of past evaluations
Consider similarity between different service
configurations
Validate evaluation results with those of
bioinformaticians
26
ECS - University of Southampton
CIA 2006
Thank you
27
ECS - University of Southampton
CIA 2006
References

J. Piaget. Sociological Studies. Routlege, London, 1973.

M. R. Rodrigues and M. Luck. Analysing partner selection
through exchange values. In Jaime Sichman and Luis
Antunes, editors, Multi-Agent-Based Simulation VI,
volume 3891 of Lecture Notes in Artificial Intelligence,
pages 24-40, Berlin Heidelberg, 2006a. Springer-Verlag.

M. R. Rodrigues and M. Luck. Cooperative interactions:
An exchange values model. In Coordination,
Organization, Institutions and Norms in Agent Systems
(COIN), ECAI Conference, Riva del Garda, Italy, August
2006b.
28
ECS - University of Southampton
CIA 2006