User Profiling
by Social Curation
By
GENG Xue
Supervised by Prof. Chua Tat-Seng
A Crowd of Social Network Platforms
Changes in
Concerts & Pope Inauguration
1990
2010
Exponential Multimedia
The Visual nature
of the web increases
exponentially
2013 Internet Trends：
http://www.kpcb.com/insights/2013-internet-trends
One of Kind
Big Multimedia
How to Deliver Meaningful Contents
to the Right Person ?
User Profiling
User Profiling
• Definition
– A process to establish user profiles by
extracting & representing the characteristics
and preferences of users.
Better Service
Better Experience
Recommendation
A
B
Similar
Recommendation
A
B
Similar
• Basic info & Social relationships
So,
User Profiling by Multimedia Analysis
However,
• Multimedia data are very diverse &
unorganized.
Traditional approaches fail.
Solutions
• Structured multimedia & Social Intelligence.
Flickr Galleries
Facebook Like
Social Curation
Modern Funnel: Social Curation.
People
select
organize
keep track of
items they like.
Social Curation Services (Pinterest)
Social Curation Services (Pinterest)
Boards to which image
are re-pinned
Board name and holder
Why SCSs good?
• Organized Contents
Better Content Models
• Content-centric network
Refine Content Models
Why SCSs good?
• Organized Contents
Better Content Models
Why SCSs good?
• Content-centric network
Social intelligence to refine content models
Framework
Profile Structure
(Ontology)
Learning
Profile Structure
Refinement
User Profiles
Ontology-based user profiles
(e.g., Fashion Domain)
Profile Ontology Construction
• Idea: Pruning Wikipedia ontology.
– Remove out-dated words which are not often
used currently.
– Insert new words.
• Find the most probable position in the existing
ontology.
• Each new word ℎ is represented by
𝒂𝒓𝒈𝒎𝒊𝒏 | 𝒉 − 𝑽𝜶 |𝟐𝟐 + 𝝀||𝜶||𝟏
𝒂
𝜶 is the sparse coefficient.
An example (‘pegged pants’)
Profile Ontology Learning
• Idea: sibling samples are more visually
similar, classifiers should be more distinct.
‘V dresses’, ‘Strapless dresses’, ‘Halter dress’, ‘One Shoulder dress’
Failure Cases
• Features may be Wrong
Refinement of user profiles
• Organized contents without social
intelligence (content-centric network).
• Social intelligence to refine user profiles.
User/Board-level Connection
• If images are shared by more
users/boards simultaneously, they more
likely belong to the same preference.
Observation: Shopping Sites
Recommendation
T shirt
People also see these
Observation:
Movie Recommendation
Recommendation
User/Board-level Connection
• If images are shared by more
users/boards simultaneously, they more
likely belong to the same preference.
Content-level Connection
• Similar images share similar visual cues
and semantics.
More Similar
Mathematical Social Intelligence
• User-level:
–
𝑢
𝑆𝑖𝑗
𝑛, if there are 𝑛 users sharing images 𝑖 and 𝑗.
=
0, if there are no users sharing them.
• Bundle-level:
𝑏
– 𝑆𝑖𝑗
=
𝑛, if there are 𝑛 boards sharing images 𝑖 and 𝑗.
0, if there are no boards sharing them.
• Content-level:
𝑣
– 𝑆𝑖𝑗
= exp(−
𝑋𝑖 −𝑋𝑗
𝜌2
ℎ
– 𝑆𝑖𝑗
= 𝐩𝑇𝑖 𝐇𝐩𝑗 =
and j.
2
2
) visual similarities of images i and j.
𝑘
𝑙
𝑘,𝑙 𝑝𝑖 𝐻𝑘𝑙 𝑝𝑗
hierarchical similarities of image i
Refinement of User Profiles
• Multi-level connections are incorporated
into the low-rank method
After
refinement
Bundle-level Semantic-level
connection
connection
𝐦𝐢𝐧 𝑱 𝑹 = | 𝑷 − 𝑹 |𝟐𝑭 + 𝜶| 𝑹 |∗ + 𝜷𝒕𝒓𝒂𝒄𝒆{𝑹𝑻 𝑳𝒖 + 𝑳𝒃 + 𝑳𝒗 + 𝑳𝒉 𝑹}
𝑹
Before
refinement
User-level Visual-level
connection connection
𝑷, 𝑹 : hierarchical representations of all training images before and after refinement.
𝑳𝒖 ，𝑳𝒃 ，𝑳𝒗 ，𝑳𝒉 : the graph Laplacians of corresponding graphs of social curation.
Visualization of User Profiles
It is a vector: (…, 0.13, …, 0.23, …, 0.3, …)
Experimental Data
• Data collection
– 1,239 users, 1,538,658 images.
• Profile learning and refinement
– We split labeled images equally into training/testing
sets as the ground truths.
• Image recommendation
– We split the dataset by pin-time for training/testing
– We added half noisy data out of fashion domain to
simulate real world recommendation system.
Evaluation of Profile Learning
Failure Cases:
a) Some image samples are too fine-grained.
b) Some concepts tend to co-occur in the same image frequently.
Evaluation of Profile Refinement
Failure Cases:
a) Sparse & noisy connections from some outdated items.
b) Some items are co-repined leading to similar multi-level
connections
Evaluation of Image
Recommendation
Conclusion
• Social Curation is NEW!
• It has
–Well-organized Contents
–Social Intelligence
• We test it on Pinterest (fashion domain).
Thank You