Data Quality
Class 10
Agenda
• Review of Last week
• Cleansing Applications
• Guest Speaker
Review of Similarity
• We use measures of similarity to establish
measures and thresholds for linkage
• Edit Distance
• Phonetic Similarity
• Ngrams
Linkage
• Critical component of data quality applications
• Linkage involves finding a link between a pair of
records, either through an exact match or through
an approximate match
• Linkage is useful for
–
–
–
–
Deduplification
Merge/Purge
Enhancement
Householding
Linkage 2
• Two records are linked when they match with
enough weighted similarity
• Matching can be a combination of exact matching
on particular fields, to approximat ematching with
some degree of similarity
• For example: two customer records can be linked
via an account number, if account numbers are
uniquely assigned to customers
Thresholding
• We have functions to determine similarity
• Tune these functions to return a value
between 0 and 1 (i.e., 0% and 100%)
• 0 = absolutely no match
• 1 = absolute match
Threhsolding 2
• For any pair of values, we can apply the
similarity function and get a score
• For different kinds of data, we can set a
minimum threshold, above which the two
values are said to match
• For example, for an n-gram match, we can
set the threshold at 75%
Thresholding 3
• When comparing 2 records, we apply the
similairty functions in a pairwise manner to each
of the important attribute value pairs
• We assign some weight to each attribute value pair
score
• Total similarity score for each pair of records is
the sum of individual attribute value pairs scores,
adjusted by weight
• We can assign an overall threshold indicating a
match
Thresholds and Matches
• We actually assign two thresholds, to partition
scores into three groups:
– Definite matches
– User review
– Definite no-matches
• Scores above the high threshold are definite
matches
• Scores between low and high thresholds are user
review matches
• All others are not matches
Deduplification
• Duplicates are sets of records that represent the
same entity
• Duplicate elimination is the process of finding
records that belong to the same equivalence class
based on similarity above a specified threshold
• When duplicates are found, one record is created
to replace all the suplicates
• That record is composed of the “best” data
gleaned from all equivalence class members
Merge/Purge
• Similar to duplicate elimination
• Application used when merging two or
more different database
• Goal: find all the records associated with
the same entity
• Example: when two banks merge, find all
accounts owned by the same person in both
banks
Enhancement
• We can enhance data by merging it with
other data sets
• Linkage may be based on profile
information extracted from each record
Householding
• We link on address as well as some
permutation of the entity name
• Look to establish a location match and some
relation between entities at that location
Naive Algorithm
• Goal: find all possible matches between any
pair of records
• Approach: Perform a pairwise similarity
score for all record pairs
• Downside: O(n2)
Improvements
• Desire to reduce the number of candidates
for pairwise similarity testing
• We can use fast matching for fixed pivot
values when merging data sets
• We use a concept called blocking to reduce
the search set
Fast Matching for Merging
• We have looked at Bloom filters for fast
matching
• We can load all (recordID,attribute value)
tuples from the first data set into the Bloom
filter O(n)
• We can then test each of the values from the
second set to see if the pivot matches in the
first set
Blocking
• Goal: reduce number of match candidates
by using some form of “compression” on
the records to be linked
• Example: phonetic encodings
• Example: limit by fixing one of the
attributes
• Example: find a pivot attribute and use that
for affinity
Blocking 2
• Example: if we want to perform
householding on a mailing list:
– Block by ZIP code, since we don’t expect to
find members of the same household living in
different locations
Linkage Algorithms
• All linkage algorithms make use of these ideas:
– A blocking mechanism
• We must choose based on the data available and what makes
sense for the application
– Similarity functions
• Every data type and data domain should have an associated
similarity function, even if it is a 0/1 exact match test
– Weights for the similarity functions
• This requires more insight into the problem, to see how each
attribute’s scores should weigh for the overall score