CSV files import automation
Kostya Khomyakov
kostya@varigence.com.au
Copyright © 2013 Varigence, Inc.
A little bit about me
• SQL Server MVP from 2013
• MCSA SQL Server 2012
• Over 6 years of experience in Database Design and
Development in SQL Server
• Over 8 years of experience in enterprise application
development using C# language
• Speaker at SQLSaturday, 24HOP, SQLRally
Copyright © 2013 Varigence, Inc.
What’s This All About?
Learn how to develop Microsoft BI Projects faster, more consistently, and more
reliably than you ever thought possible…
… and have fun doing it.
3
Copyright © 2013 Varigence, Inc.
What will we cover today?
• Introduction
• Main blocks of CSV integration package
• Logical BIML steps for CSV integration package
• Live Demo. BIML from scratch
4
Copyright © 2013 Varigence, Inc.
Introduction
• a lot of SSIS packages are very similar
• packages importing data from different sources
• packages exporting data to other sources
• quite often .Net is used inside packages
• packages used for dimension updates
•…
• … but development takes a lot of time to create similar packages
Why not consider ETL processes as a set of models/patterns which
can be easily added or removed?
5
Copyright © 2013 Varigence, Inc.
Main blocks of CSV integration package
•
•
•
•
•
6
Connection to a DB
Create Staging Tables
Create Flat File Connection
Config File Format
Transformation from CSV to Staging Tables
Copyright © 2013 Varigence, Inc.
• Connection to a DB
• FOREACH (csvFile in FilesCollection)
{
- Create Staging Tables
- Create Flat File Connection
- Config File Format
- Transformation from CSV to Staging Tables
}
Logical BIML steps for CSV integration package
1.00 CreateStagingTables.biml
2.00 CreateImportCSVPackages.biml
3.00 CreatePackageProject.biml
7
Copyright © 2013 Varigence, Inc.
Foreach (file in fileCollection)
{
Read 1-st line and get fields
}
Foreach (file in fileCollection)
{
a. Create Flat File Connections
b. Create File Formats
c. Transformation
}
Logical BIML steps for CSV integration package
Challenges with CSV files:
• Header Row Delimiter
• Column Delimiter
• Row Delimiter
8
Should be defined as variables in .Net
+ add variables: Files Folder
Files Extension
Copyright © 2013 Varigence, Inc.
Create SSIS packages
automatic VS manual
9
Copyright © 2013 Varigence, Inc.
Live Demo
9
Copyright © 2013 Varigence, Inc.
Resources
•
Twitter
–
–
•
LinkedIn Biml User Group
–
–
•
http://www.bimlscript.com
CodePlex
–
•
http://www.varigence.com/mist
BimlScript
–
•
http://www.linkedin.com/groups?home=&gid=4640985&trk=anet_ug_hm
http://www.linkedin.com/groups/Biml-User-Group-Australia-5190127?home=&gid=5190127
Varigence Mist
–
•
@BimlScript
@BimlDownunder
http://bidshelper.codeplex.com/
Biml Documentation
–
http://www.varigence.com/documentation/biml/
Copyright © 2013 Varigence, Inc.