Intelligent Agents -User Agent
Interaction and Learning
Katia Sycara
The Robotics Institute
email: katia@cs.cmu.edu
www.cs.cmu.edu/~softagents
Why do we need to worry about
security?
• We want to deploy our system in open networks;
– Agents come and go => agents interact with strangers;
– Can agents be trusted? Can their deployers be trusted?
• Agents are expected to do more serious things. Ex:
– Getting info on your banking account;
– Carry out sales transactions.
Security issues for mobile agents
• Origin authentication;
– Include public key certificate of the originator as part of the agent
– In practice this is not enough; the entire agent must be signed and
integrity-protected
• Data Integrity
– The body of the agent should be integrity protected, thus allowing
for after-the-fact tampering detection
Security issues for mobile agents
(cont)
• Access/itinerary control
– Restrict number and identity of the servers to be visited; however
access control by naming severely restricts agent’s freedom
– In practice, it may be better to manage agent access control via
attributes, e.g. cpu available, agent origins, willingness to pay
• Agent’s privacy
– It is difficult to keep an agent protected from tampering
– Tradeoff between the security advantages of fixed itineraries and
the flexibility of free roaming
Security issues for mobile agents
(cont)
• Privacy and integrity of gathered information
– In stateless information gathering, the agent intermittently sends
acquired information back to its user. Protecting information is
done by encrypting it.
– In stateful information gathering, the gathered information is kept
by the agent as it visits servers and is delivered upon the agent’s
eventual return. A malicious execution environment could
terminate the agent, thus preventing it from migrating further or
returning.
Trusting your agents
• Agents are delegated authority by their users
• Agents behavior may not be predictable
– While this is true of many application programs, they usually have
very well-defined domains of action and so can be thoroughly
tested.
• Agents have a potential for causing damage
Trusting your agents (cont)
• Testing and training the agent in safe mode (proposing but not
allowed to execute actions)
• Limited delegation
• Verification through an audit trail
• Reputation
– Receiving a request from a “trusted” agent client, makes the
request more trustworthy
– If an agent reports information from a reputable source, it is more
trustworthy
– Reputation servers can keep track of agent behaviors
Testing and validation of agents
•
•
•
•
•
•
Testing and training the agent in safe mode
--Testing is costly
--Incremental testing
--Agent should be configurable with realistic knowledge of its bounds
--Ability to log every action, and basis of decisions it makes
Validation
– Harder for agents, e.g. if agents are learning, they could have nondeterministic behavior
– Should be done in isolated environment
– In conventional systems, one knows a priori the boundaries of the
parameter space; this is not so with agents
– Establish separate environment with historical data and replay the
environment aganst the agents while monitoring the outcomes
Training your agent
• By analogy to human training
– Identify the goal
– Identify the process
– Explain the environment
– Identify the limits of the trainee’s authority
– Verify the results
Assumptions we make
• Neither agents nor their deployers are trustworthy (They can
misbehave):
– Agents may eavesdrop on communication between two other
agents;
– An agent may masquerade as some other agent;
– Agents may not behave as expected (e.g., an agent may not want to
pay for goods received).
• There must be a few, selected entities that can be trusted.
Adding Security to Retsina (MAS in
general)
•
Prevent misbehaviors from happening + have recovery mechanisms if
they happen:
– Identify different security issues that MAS face;
– Propose solutions for these problems;
– Design and implement a security infra-structure for Retsina.
•
Focus: application-independent issues:
– Communication security: yes;
– Fair exchange in electronic sales: no.
•
Approach: Standard security techniques used in distributed systems
– Authentication;
– Access control;
– etc.
Security Threats in MAS
• System-level threats: Those that subvert inter-agent interactions,
independently of the application a system is running.
– Untrustworthy ANSs and matchmakers;
– Untrustworthy application agents;
– Insecure communication channels.
• Application-level threats: Those that subvert the security of
applications. They may exist even if the underlying system of
agents is secure.
– Service providers that do not implement appropriate access control
policies;
– Untrustworthy application agents.
Our Solution (1)
• To guarantee the integrity of naming and matchmaking services:
– Include access control:
– <use> trusted ANSs and matchmakers!
• Make agents uniquely identifiable, and give them unforgeable
proofs of identity:
– Prevents spoofing;
• Make deployers of agents liable for the actions of their agents:
– Agents are given proofs of identity only when deployers allow
their own identities to be linked with those of their agents.
Our Solution (2)
• Protect communication channels;
• Add access control mechanisms (which usually rely on the delegators’
Ids);
• Make agents prove that they are delegates of whom they claim to be.
A Design of a Security Infra-structure
for Retsina
Assumptions:
• Deployers have public key certificates binding their physical identities
(SSN, company names, etc.) to their public keys;
• DCAs are assumed to exist (lie outside our security infra-structure);
• ANSs and matchmakers
– are trusted entities;
– their public keys are publicly known.
• The addresses of ANSs are publicly known.
Public Key Cryptography
• Key pairs:
– Private keys: a ;
– Public keys: A ; pub(a)
• Digital signatures: (m)a
• Signature verification:
– Use A to verify (m)a
• Public key certificates:
– (Name, Public key)ca
Giving Identities to Agents +
Establishing Liability (1)
1. Choose an Agent ID AID
2. Generate a public key pair {a, pub(a)}
3. m1=(certify AID,pub(a),t)d
4. Verify the validity of the request
5. Generate m2=ACA-signed certificate binding pub(a) to AID
6. Creates an entry [D’s public key certificate, m1] in the certification DB
8. Verify the signature in m2 with ACA’s public key.
Giving Identities to Agents +
Establishing Liability (2)
• Agents are given:
– a public key certificate, and
– a matching private key.
• The certification process:
– certification can be requested only by deployers who can prove
their own identities => make the deployer aware of his or her
liabilities.
Revoking an Agent’s Public Key
Registering at an ANS
Unregistering at an ANS
The Lookup Protocol
Agents are identified by their keys, and not their names!!
Matchmaker Protocols
• Very similar to ANS protocols;
• Differences:
– Physical addr may not be shared by more than one agent; capabilities may;
– Agents use ANSname.Agentname to register with the matchmaker;
– The lookup protocol:
1. CAP
2. [CAP, {[ANS-x1.AID1,CERT1],… [ANS-xn.AIDn,
CERTn]}, T]mm
Secure Communication Channels
• SSL (Netscape’s Secure Socket Layer protocol);
• Why?
– Keep communication security transparent from the
application;
– Off-the-shelf trustworthy technology (extensible too).
• Implementation:
– Local effort: at the Communicator.
Secure Delegation + Access Control
• Knowing who is the delegator may be necessary or
desirable;
• Original design: Have the agent know the secret key of
its deployers;
• Weaknesses:
– Agents should not know such important secrets;
– Sometimes they do not even have to know (ex: PIN);
– Should have a “weaker,” “temporary” solution.
Some Interesting Pages
• http://microsoft.com/security/tech/certificates/formats.asp
– Some introductory material on standards for
cryptographic object;
• http://security.dstc.edu.au/projects/java/release3.html
– Info on a real-world security package (can download
the code and play);
• Communications of the ACM - June 1996, volume 39,
Number 6
– An issue dedicated to EC; from which the article is
extracted.
Adaptability
• Adaptability of an individual agent
– Agent Communication
– Agent Coordination
– Agent Planning
– Agent Scheduling
– Agent Execution Monitoring
• Learning from interactions (user or other
agents)
• Organizational adaptability (middle agents)
• Performance adaptability (cloning)
_____________________
Sycara, “Levels of Adaptivity in Systems of Coordinating Information Agents”,
Lecture Notes in Artificial Intelligence1435, Klusch and Weiss (Eds.), July 1998 CIA’98
User Agent Interaction and Learning
• Agent can be pre-programmed
– efficient but could be inflexible
• User explicitly delegates goal to an agent through end
user programming
– flexible but difficult
• Agent learns from user interactions
– metaphor of agent as apprentice.
User Agent Interaction and Learning
(cont)
• Instruction
– a set of directions to the agent on how to carry out a
task
• Confirmation
– a request for user’s approval before the agent carries
out an instruction
• Observation
– a behavioral pattern learned by the agent
• Suggestion
– an agent recommendation to the user
User Agent Interaction and Learning
(cont)
• Instruction
– user can either instruct the agent directly or accept the
agent’s offer of automation for an observed work
pattern
• When the agent informs of an observation, the user
can:
– accept the observation and have the agent create an
instruction
– decline the observation
– edit the observation to fine-tune the instruction
– postpone a decision till later
Learning Approaches
• Machine learning plus Rule based, or expert system
– machine learning for knowledge acquisition and rule
based inference for knowledge maintenance
• Case-based reasoning
– based on past experiences/cases that contain
– the problem
– attributes of the situation/context
– the solution
– indication of success or failure of the solution
Learning Approaches (cont)
• Neural Networks
– self-organizing learning approach
• Various statistical clustering techniques
– perform classification of concepts
Fact Interpretation for Agent-User
Interaction
• User behavior
Agent interpretation
rule
• user declines agent offer to automate-- offer less general
observation
• user repeatedly undoes agent action -- offer to turn off
existing rule
• user sometimes undoes agent action -- offer to make rule
more specific to match condition
•
• agent interaction preference does not match user behavior
-offer to change preference
Agent Apprentice Assistance
•
•
•
•
•
•
•
In-context tips
Coaching when the user needs help
Proactive Assistance
Shortcuts for a sequence of steps
Customized offer based on learned user preferences
Automation offer for repetitive user tasks
Automation suggestions based on what the user is not
doing
• Notification of significant events
Control Delegation of Actions
• Confirm Once-- the agent displays a confirmation
message only before the first time it carries out the
instruction
• Confirm Always-- the agent displays a confirmation
message every time before it carries out an instruction
• Don’t Confirm -- the agent never confirms with the user
before carrying out the instruction
Information Search
• Ways to Find Information
– Browsing: Following hyper-links that seem of interest
– Searching: Sending a query to a search engine such as Lycos
– Categories: Following existing categories such as Yahoo
• Problems
– Spent a lot of time and effort to navigate. Can search be made more
efficient?
– Search but it is difficult to accurately express the user’s intention.
– Search engines are not personalized
Web Site Personalization
• Personalized newspaper-- a publisher can deliver
personalized news to each of its subscribers
• Personalized store fronts: An on-line merchant can make
customized recommendations based on user preferences and
transaction history
• Customized service providers: An on-line service provider can
make personalized recommendations (video, ads etc) based on
individual user preferences and preferences of other users
Personalization Content
• User Data
–
–
–
–
demographic information (e.g., age, gender)
dynamic user interests (e.g. music, travel)
user transaction history (e.g. seasonal purchases)
user behavior at a site (e.g. hyperlinks clicked)
• User data in user data base for use during recommendations
– indexing provided through data base tables
– indexing through engine such as Verity Search for web pages
Personalized Web site Operation
• Agent collects new content from web or other content data
bases
• Users register their interests with the agent using profiles
• Agent serves new information to users according to their
preferences
• Using machine learning, agent learns more about users from
what they click and where they go
• Agent suggests changes to the user and learns from feedback
• After each successive visit, the agent gives new information and
ideas matching user interests and preferences
Level of Personalization
• Customization--user fills out profile and agent delivers content
according to it. But, this approach requires user to change
preference profile
• Learning user interests. --agent watches over user shoulder
• Learning community behavior. -- agent compares user
preferences to those of others with similar interests. This
approach encourages user exploration
User Experience
• Explicit vs implicit ranking--some collaborative
filtering tools require that users explicitly rank an
initial amount of pages. This imposes additional
burden on the user
• User identification-- user name and password; use of
cookies during revisits.
Collaborative Filtering
A collaborative filtering system makes
recommendations based on the preferences
of similar users.
People: Yenta, Referral Web
Products: Firefly, Tunes, Syskill & Webert
Readings: Wisewire, Phoaks
Content vs. Collaboration
• Content-based retrieval returns documents
that are similar to a query (search) or a user
profile (preference)
• Collaborative recommendation retrieves
documents liked by others with similar
profiles
Problems in Collaborative
Filtering
Incentives & Startup
• Need a critical mass of users/recommenders to
make meaningful predictions
• Need mechanisms to maintain participation
Reliability
• Spoofing- will content providers inflate their
ratings
• Technical problems with clustering & similarity
measures
Privacy
• Once you share your profile who else may want
it?
Functionality of WebMate
• Learning user’s interests for information filtering
– Multiple TF-IDF vectors representation
– Incremental and adaptive Learning
– Compile personal newspaper
• Support for efficiently finding information
– Automatic refinement using Trigger Pairs
– Relevance feedback
_____________________________
Chen, Sycara, “WebMate: A Personal Agent for Browsing and Searching”,
Proceedings of the Second International Conference on Autonomous Agents, Minneapolis, MN, May 1998
Learning to extract information
Maulsby, David & Ian Witten (1997). Teaching agents to learn: From user study to
implementation. IEEE Computer, Nov, 36-43.
Problem: Features of materials prepared for human
viewing such webpages or printed text are
designed to make it easy for us to extract
information but can be very difficult for a
computer. Telling a computer “how” to extract
information rather than just which URL to open
can be a hard problem.
How We Extract Information
Repetitive information such as bibliographic records,
on-line classifieds, or catalogs usually use some
combination of:
• visual demarcations (such as ruled lines or tables)
• visual distinctions (such as italics, color, or
indentation)
• punctuation (such as commas, colons, or periods)
• precedence/succession relations among
informational elements
But How do we tell a computer
what these visual cues are?
We don’t want to write a parser
We don’t want to write & debug a production
system
We may not even know what the markup
looks like
So… How do we tell an agent how to do it
(from our point of reference) so it can do it
(from its point of reference)..
Mausby’s Study
Problem: Train a program to recognize and
translate between bibliographic entries in a variety
of formats
Approach:
• Wizard of Oz study to find out how people would
like to instruct a computer in doing the task
• Implement/test a system that takes instruction of
this sort
Instructing as Machine Learning
• Learns incrementally and adds exceptions as they
are encountered
• Uses demonstration (point/select) and “verbal”
instruction such as “precedes”
• Proposes “guesses” when classification is
ambiguous
• Learns Perl style patterns for things such as phone
numbers and uses punctuation and capitalization
as referents in rules
CIMA/TURVEY Rules
• Learns DNF rules.. Greedy with exceptions tacked
on..
• As a result of pointing, clicking, indicating
precedes/follows, etc. CIMA can generate rules
which do things such as “recognize” phone
numbers with or without area code,
identify the surname and forename/initials of the
first author of a paper and similar feats..
Adaptive Negotiation (the Bazaar Model)
• Aims at modeling multi-issue negotiation processesª
• Combines the strategic modeling aspects of game-theoretic
models and single agent sequential decision making
models
• Supports an open world model
• Addresses heterogeneous multi-agent learning utilizing the
iterative nature of sequential decision making and the
explicit representation of beliefs about other agents
______________________________
ª D.Zeng and K.Sycara. “Bayesian Learning in Negotiation.” International Journal of HumanComputer Studies (1998), 48, pp. 125-141.
Utility of Learning: Experimental Design
• The set of players N is comprised of one buyer and one
supplier who make alternative proposals.
• For simplicity, the range of possible prices is from 0 to 100
units and this is public information
• The set of possible actions (proposed prices by either the
buyer or the supplier) A equals to {0, 1, 2,…, 100}
• Reservation prices are private information.
• Each player's utility is linear to the final price ( a number
between 0 and 100) accepted by both players
• Normalized Nash product as joint utility (the optimal joint
utility when full information is available is 0.25)
Average Performance of Three Experimental
Configurations in Bazaar
• A non-learning agent makes decisions based solely on his own
reservation price
• A learning agents makes decisions based on both the agent's own
and the opponent's reservation price
Configuration
Buyer’s Supplier’ # of Proposals
Utility
s Utility
Exchanged
0.49
.051
24
Both Learn
Joint
Utility
0.22
Neither Learn
0.18
0.49
0.51
34
Only Buyer Learns
0.15
0.59
0.41
28