Data Quality Services
+
What’s new in SSIS
in
SQL Server 2012
James Beresford
www.bimonkey.com
@BI_Monkey
But first….
Data Quality Services
SQL 2012 RC0 Edition
The Data Quality problem
!=
Hands up if you’ve been 2 years old
Your first data quality problem
Data Quality = Shape Sorting
There is Good Data
There is Bad Data
There is Repairable Data
Data Quality Services Terms
A Shape = Domain
Data Quality Services Terms
Collection of Shapes = Knowledge Base
Data Quality Services Terms
Shape Sorting = Domain Rule
The Data Quality Client
Allows you to:
• Create or Maintain
• Knowledge Bases
• Data Quality Projects
• Manage and Configure DQS
The Data Quality Client
Create a Knowledge Base
The Data Quality Client
Create a Knowledge Base
What did we see?
• Create a Knowledge Base from Data using
Knowledge Discovery
• Set values as Correct, Error or Invalid
• Teach the KB some simple auto correction using
Domain Rules
The Data Quality Client
Pop Quiz: Who owns a Knowledge Base?
Maintaining KB’s is done by:
a)
b)
c)
d)
Qualified Data Professionals
A BI Developer
Users
Tony Abbott
The Data Quality Client
Pop Quiz: Who owns a Knowledge Base?
Maintaining KB’s is done by:
a)
b)
c)
d)
Qualified Data Professionals
A BI Developer
Users
Tony Abbott
The Data Quality Client
Create a Composite Domain
+
n
The Data Quality Client
Create a Composite Domain
What did we see?
• Use values from one domain to affect another
The Data Quality Client
Other Domain Functions
Things that I won’t demonstrate
• Term Based Relations – autocorrect
substrings (e.g. Inc. > Incorporated)
• Reference Data – Validation against external
sources on Azure DataMarket, e.g. Melissa
Data
The Data Quality Client
Create a Data Quality Project
The Data Quality Client
Create a Data Quality Project
What did we see?
• Interactive Cleansing of data
• Export of cleansed data set
The Data Quality Client
Matching Policies
Fuzzy & Domain matching
• Matching Polices
• Exact – Must match to count in score
• Similar – Approximate match counts
• Prerequisite – Must match or match fails
• De-Duplication
Automation with SSIS
Automation with SSIS
The DQS Cleansing Task
What’s new in SSIS
Denali CTP3 Edition
What’s new in SSIS
Projects & Parameters
What’s new in SSIS
The SSIS Catalog
What’s new in SSIS
Developer Experience
• Visual Studio 2010 & .Net 4 support
• Undo / Redo
• Auto-save project recovery
What’s new in SSIS
Developer Experience
New Toolbox
What’s new in SSIS
Developer Experience
Expressions
• New Expressions
• LEFT
• TOKEN
• TOKENCOUNT
• Removal of 4000 character limit
What’s new in SSIS
Developer Experience
New Data Flow Column Mapper
What’s new in SSIS
Developer Experience
Expression Markers on icons
What’s new in SSIS
Developer Experience
Variable Window improvements
1. Package Scope by Default
2. Change Scope button
What’s new in SSIS
Developer Experience
Data Quality Services Integration – DQS Cleansing Task
What’s new in SSIS
Administration Experience
• Simplified deployment & configuration
• Automatic logging
• Management dashboard
• Built-in reports
• Improved SSMS management experience
• Advanced troubleshooting features
• Package performance analysis
What’s new in SSIS
SQL 2012 RC0 Edition
What’s new in SSIS
Developer Experience
Change Data Capture
What’s new in SSIS
Developer Experience
ODBC Support
• Source and Destination Adapters
What’s new in SSIS
Developer Experience
Connection Manager Changes
What’s new in SSIS
Developer Experience
Flat File Source Improvements
What’s new in SSIS
Developer Experience
•
•
•
•
Package Format Changes
Visual Studio Configurations
Scripting Improvements
Troubleshooting & Logging
What’s new in SSIS
Administration Experience
• Data Taps
• Server Management with PowerShell
What’s new in SSIS
Developer Experience
Other Changes
• Updated look for the Control Flow and Data Flow
• Pivot UI
• Row Count UI
• New Expression:
◦REPLACENULL
• BIDS is now SQL Server Data Tools
• Many small fixes and improvements based on CTP
feedback