Information Elicitation
in Scheduling Problems
Ulaş Bardak
Ph.D. Thesis Defense
Committee
Jaime Carbonell (chair), Eugene Fink, Stephen Smith,
and Sven Koenig (University of Southern California)
Outline
Introduction
 Related work
 Domain
 Optimization
 Elicitation
 Evaluation
 Conclusions

7/17/2016
Ulas Bardak - Thesis Defense
2
What is information elicitation?
For example…
7/17/2016
Ulas Bardak - Thesis Defense
3
Why elicitation?
Scheduling problems include information
about resources, constraints, and
preferences
 Uncertain information can lower the
quality of schedules
 We need to select and ask questions that
help to reduce uncertainty

7/17/2016
Ulas Bardak - Thesis Defense
4
Example problem
We are organizing a small conference,
using three available rooms
 We have incomplete information about
speaker needs

7/17/2016
Ulas Bardak - Thesis Defense
5
Initial schedule
Available rooms:
Initial schedule:
Room
num.
Area
Projector
1
2
3
Large
Med.
Small
Yes
No
Yes
Requests (by importance):
1. Invited talk, 9–10am:
Needs a large room
2. Poster session, 9-11am:
Needs a room
7/17/2016
1
Talk
2
Posters
3
Missing
info:
Assumptions:
•
• Invited
Invited talk:
talk:
–– Projector
need
Needs a projector
•
session:
• Poster
Poster session:
–– Room
size
Smaller
room is OK
–– Projector
need
Needs no projector
Ulas Bardak - Thesis Defense
6
Choice of questions
Initial schedule:
1
Talk
2
Posters
3
Candidate questions:
Requests:
• Invited talk,
talk: 9–10am: Useless info: There are no
large rooms w/o a projector
× Needs a large
projector?
room
• Poster session,
session: 9–11am: Potentially useful info
How big
a room?
a room
√ Needs
Potentially useful info
√ Needs a projector?
7/17/2016
Ulas Bardak - Thesis Defense
7
Improved schedule
Requests:
Initial schedule:
• Invited talk, 9–10am:
1
2
Needs a large room
Posters
• Poster session, 9–11am:
Needs a room
3
Talk
Info elicitation:
New schedule:
System:
Does the poster session
2
1
need a projector? How
Posters
big a room does it need?
User:
A projector may be useful.
Talk 3
A small room is OK.
7/17/2016
Ulas Bardak - Thesis Defense
8
Motivation
Improve optimization results by
reducing uncertainty of the
available knowledge.
7/17/2016
Ulas Bardak - Thesis Defense
9
Related work

Example critiquing (Burke)
 Have

Collaborative filtering (Resnick and Hill)
 Have

the user rank related items
Similarity-based heuristics (Burke)
 Look

users tweak result set
at past similar user ratings
Focusing on targeted use (Stolze)
7/17/2016
Ulas Bardak - Thesis Defense
10
Related work
Clustering utility functions (Chajewska)
 Decision tree (Stolze and Ströbel)
 Min-max regret (Boutilier)

 Choose

question that reduces max regret
Auctions (Smith, Boutilier, and Sandholm)
7/17/2016
Ulas Bardak - Thesis Defense
11
What is different?
No bootstrapping
 Both continuous and discrete variables
 Large number of uncertain variables
 Tight integration with the optimizer
 Synergy of multiple approaches

7/17/2016
Ulas Bardak - Thesis Defense
12
Explored domains

An academic conference
 Assigning

Placing vendor orders
 Assigning

rooms to sessions
orders to sessions
Social networking
 Matching
7/17/2016
users to other users
Ulas Bardak - Thesis Defense
13
Selected domain
Scheduling a conference
 Rooms are our resources
 We need to assign rooms to sessions

7/17/2016
Ulas Bardak - Thesis Defense
14
Collaborative scheduling
Automatic operations
Manual operations
invoke the
auto scheduling
7/17/2016
• Edit resources and constraints return the control
to the user
• Modify the schedule
• Provide advice to the system
Ulas Bardak - Thesis Defense
15
Collaborative scheduling
Automatic operations
Process new data
and advice
Optimize
schedule
Generate and send
questions
to the user
Manual operations
invoke the
auto scheduling
7/17/2016
• Edit resources and constraints return the control
to the user
• Modify the schedule
• Provide advice to the system
Ulas Bardak - Thesis Defense
16
Architecture
Top-level control
and learning
Representation
Optimizer
Info elicitor
Process
new info
Optimize the
schedule
Choose
questions
Graphical
user interface
7/17/2016
Ulas Bardak - Thesis Defense
Administrator
17
1
Rooms
2
2000 ft2
3

Rooms have a set of properties
 Size,
seating capacity,...
 Microphones, projectors,...
Room 1 is 2000 square feet and has one projector.

We also know distances between rooms
Room 1 is 400 feet away from Room 3.
7/17/2016
Ulas Bardak - Thesis Defense
18
Sessions
Session description includes
 Importance
 Hard constraints, such as the
minimal acceptable room size
 Soft preferences, such as the
desired room size
The invited talk is more important than the poster session.
The assigned room has to be at least 500 square feet, and
preferably 1000 square feet.
7/17/2016
Ulas Bardak - Thesis Defense
19
Sessions
We represent preferences by
piecewise-linear functions.
1.0
Quality
0.5
0
Unacceptable
250
500
750 1000
Room size
The invited talk is more important than the poster session.
The assigned room has to be at least 500 square feet, and
preferably 1000 square feet.
7/17/2016
Ulas Bardak - Thesis Defense
20
Uncertainty
We usually have incomplete
knowledge of room properties,
session importances, and
constraints and preferences.
7/17/2016
Ulas Bardak - Thesis Defense
21
Uncertain properties
We represent an uncertain value as either
 a completely unknown value, or
 a probability density function,
approximated by a set of uniform
distributions.
7/17/2016
Ulas Bardak - Thesis Defense
22
Uncertain properties
Example: An auditorium has about 600 seats.
0.2 chance: [450..549]
0.6 chance: [550..650]
0.2 chance: [651..750]
Probability
0.006
0.004
.6
0.002
0
7/17/2016
.2
0
200
400
Capacity
.2
600
Ulas Bardak - Thesis Defense
800
23
Uncertain preferences
We represent an uncertain preference as
 completely unknown function,
 piecewise-linear function with uncertain
y-coordinates of endpoints, or
 set of possible piecewise-linear functions
with related probabilities.
7/17/2016
Ulas Bardak - Thesis Defense
24
Uncertain preferences
The description of a demo session does
not include a room-size preference.
1.0
Quality
0.5
0
Unacceptable
.95 chance
250
500
.05 chance
750 1000
Room size
Demo sessions usually require at least 250 square feet, and preferably
750 square feet; however, there is a 5% chance that a big sponsor
shows up unexpectedly and asks for additional 250 square feet.
7/17/2016
Ulas Bardak - Thesis Defense
25
Optimization
The optimizer assigns rooms to sessions.
 Input: Rooms and sessions
 Output: Room and time for each session
7/17/2016
Ulas Bardak - Thesis Defense
26
Session quality
Quality value of a session is based on how
much each preference is satisfied
 Uncertainty is taken into account when
calculating quality

7/17/2016
Ulas Bardak - Thesis Defense
27
Schedule quality
Overall schedule quality value is a
weighted sum of session quality values
 If any session violates hard constraints,
the whole schedule is unacceptable

7/17/2016
Ulas Bardak - Thesis Defense
28
Optimizer
Simple version is based on hill-climbing
 Advanced version uses randomized hillclimbing, similar to simulated annealing

7/17/2016
Ulas Bardak - Thesis Defense
29
Elicitation
We use elicitation to reduce uncertainty
 User can selectively answer any questions

7/17/2016
Ulas Bardak - Thesis Defense
30
Elicitation
Synergetic Elicitor
Heuristic
Elicitor
7/17/2016
Rule-based
Elicitor
Ulas Bardak - Thesis Defense
Search
Elicitor
31
Heuristic elicitor
Synergetic Elicitor
Heuristic
Elicitor
Rule-based
Elicitor
Search
Elicitor
Selection of questions based on the
standard deviation of schedule quality
 Fast calculation, once per variable
 Domain-independent

7/17/2016
Ulas Bardak - Thesis Defense
32
Heuristic elicitor
Get list of questions
Each uncertain variable is a potential question
For each question,
determine impact on
schedule quality
of possible answers
Plug in possible answers to the quality
function to get change in schedule quality
For each question,
calc. question score
Score q    quality , q  cost  q 
Return top questions
7/17/2016
Ulas Bardak - Thesis Defense
33
Rule-based elicitor
Synergetic Elicitor
Heuristic
Elicitor
Rule-based
Elicitor
Search
Elicitor
Selection of additional questions, based on
domain-specific heuristics, such as “Room
capacity is more important than ceiling
height.”
7/17/2016
Ulas Bardak - Thesis Defense
34
Search elicitor
Synergetic Elicitor
Heuristic
Elicitor
Rule-based
Elicitor
Search
Elicitor
Ranks selected questions using B* search
 Relies on the optimizer for evaluating
nodes in the search space
 Domain-independent and optimizerindependent

7/17/2016
Ulas Bardak - Thesis Defense
35
Example
Uncertain room size versus uncertain projector number
0 0.1 0.2 0.3 0.4 0.5
100-150:40% Min qual.:0.1
151-200:60% Max qual.:0.5
160-200:50%
100-160:50%
100-120:25%
0-1:50%
2-3:50%
Min qual.:0.15
Max qual.:0.35
120-160:25%
0-1:50%
Min qual.:0
Max qual.:0.4
2-3:50%
Min qual.:0.1
Max qual.:0.25
Min qual.:0.28
Max qual.:0.33
The minimal possible utility of asking about the room size
is greater than the maximal possible utility of asking about
the number of projectors.
7/17/2016
Ulas Bardak - Thesis Defense
36
Evaluation
The synergetic elicitor is far more
effective than each of its individual
components, simple heuristics, and
random selection of questions.
7/17/2016
Ulas Bardak - Thesis Defense
37
Evaluation
Four scenarios with 88 sessions
 10
rooms,
 20 rooms,
 50 rooms,
 84 rooms,
7/17/2016
100 uncertain values
500 uncertain values
1000 uncertain values
3300 uncertain values
Ulas Bardak - Thesis Defense
Less
complex
More
complex
38
Evaluation
For each setting, we use five
different elicitation systems
 Synergetic
Elicitor
 Heuristic & rule-based
 Search & rule-based
 Rule-based
 Random
7/17/2016
Synergetic Elicitor
Heuristic
Elicitor
Ulas Bardak - Thesis Defense
Rule-based
Elicitor
Search
Elicitor
39
Evaluation
We plot:
 Change
in the schedule quality
 Change in the quality loss due
to uncertainty (100%  0%)
FullyCertainQuality  CurrentQuality
FullyCertainQuality  InitialUncertainQuality
7/17/2016
Ulas Bardak - Thesis Defense
40
Sche
0.7
0.65
0
Evaluation
500 1000 1500 2000 2500 3000
Questions answered
% of questions needed for 85% of full quality
80%
100 variables
33%
Heuristic
Search
70%
Rule based
Full quality
100%
Remaining loss .
0.74
Qualityy
Schedule Quality
15%
Random
12.5%
Full
0.735
0.73
0.725
80%
60%
40%
20%
0%
0
25
50
75
75
100
answered
Questions answered
7/17/2016
0
25
50
75
100
Questions answered
Ulas Bardak - Thesis Defense
41
Sche
0.7
0.65
0
Evaluation
500 1000 1500 2000 2500 3000
Questions answered
% of questions needed for 85% of full quality
45%
3300 variables
26%
Heuristic
Search
44%
Rule based
Full quality
100%
0.8
Remaining
Remaining loss
loss ..
Schedule Qualityy
33%
Random
17.5%
Full
0.75
0.7
80%
60%
40%
20%
0%
0.65
0
500 1000 1500 2000 2500 3000
500 1000 1500 2000 2500 3000
Questions answered
Questions answered
7/17/2016
0
Ulas Bardak - Thesis Defense
42
Sche
0.7
0.65
0
Evaluation
500 1000 1500 2000 2500 3000
Questions answered
% of questions needed for 95% of full quality
98%
73%
100 q.
Random
18% 3400 q.
Full
Problem size
42%
500 q.
1000 q.
Heuristic
Search
Rule based
Full quality
Remaining loss .
100%
80%
60%
40%
20%
0%
0%
0%
25%
31%
50%61%
75% 91%100%
Questions answered
7/17/2016
Ulas Bardak - Thesis Defense
43
Summary
We have applied the elicitor to
conference scheduling
 Synergetic elicitor outperforms its
components and simple heuristics
 Improvement is more prominent
for larger problems

7/17/2016
Ulas Bardak - Thesis Defense
44
Contributions
We have investigated a novel approach to information
elicitation, which has led to three main contributions.
Fast heuristic computation of the
expected utility of potential questions
 Use of B* search for determining more
accurate question utilities
 Synergy of domain-independent and
domain-specific elicitation techniques

7/17/2016
Ulas Bardak - Thesis Defense
45
Future work
Learning question costs
 Learning elicitation strategies

7/17/2016
Ulas Bardak - Thesis Defense
46
7/17/2016
Ulas Bardak - Thesis Defense
47
Additional Slides
7/17/2016
Ulas Bardak - Thesis Defense
48
Vendor elicitation
Domain
 Sessions
can require services that external
vendors provide

e.g. mobile equipment, food deliveries
 Each

item can satisfy multiple services
e.g. Laptop  Computer, Portable computer
 Penalty
for spending money
 A vendor optimizer finds a near optimal
placement of vendor orders
 Uncertainty can exist in prices, availability of
Ulas Bardak - Thesis Defense
49
7/17/2016 items
Vendor elicitation
Elicitation Algorithm
 Enumerate
all of the services
 Order based on affecting the overall cost
penalty
7/17/2016
Ulas Bardak - Thesis Defense
50
Vendor elicitation
Evaluation
100%
0.8
Remaining Loss
Schedule Quality
1
0.6
0.4
0.2
80%
60%
40%
20%
0%
0
0
500
1000
500
1000
Questions Answered
Questions Answered
7/17/2016
0
Ulas Bardak - Thesis Defense
51