iWay Data Profiler
Product Highlights
Anand Maheshwari
5.0 GA
1
What is iWay Data Profiler
iWay Data Profiler provides the ability to look at data quality from a
business user perspective with dashboard templates to view, monitor,
compare, and report by leveraging the turnkey business intelligence
technology through semantic analysis, complex business rules and data
drill-down.
2
Features
1.
Ability to profile data against a set of standard (pre-defined) data quality
rules and metrics
2. Ability to create custom organization specific data quality indicators
3. Advanced comparison of data profiling and quality indicators
4. Ability to create and execute data quality projects against varying payloads
5. Ability to rollup data with drill through
6. Advanced administration capability
7. Advanced auditing capability
8. Easier migration from earlier versions
9. Ability to track background processes for faster troubleshooting and status
10. Use of current standards like JQuery for enhanced user experience
11. Support for all leading database platforms
12. Support for wide variety of OS platforms
13. Support for German, French, Spanish and English locales
14. Increased performance through several tuning efforts
3
1. Ability to profile data against a set of standard (predefined) data quality rules and metrics - Comparison
Features
4.1
5.0


AHTML


PDF


XLS
×



Data Analyses


Business Rules


Primary Key


Foreign Key


Summary


Hierarchical folders
×

Uploading profile in background
×

View process result
×

Rollup profile with drill-through
×

Create profile with drill-through
Export profile report to: AHTML and PDF format
Email profile using SSL
Create profile report for:
Profile file size
100MB
300MB
4
1. Ability to profile data against a set of standard (predefined) data quality rules and metrics – New Features
•
•
Uploading profile in background
View result to display uploading progress
• Export profile report in XLS format
5
2. Ability to create custom organization specific data
quality indicators – Comparison
Features
4.1
5.0
Create and view DQIs report


Export PDF report


Folder Hierarchical division
×

Comparison Across Expression
×

Visualization for success and failure counts
×

More user interaction using AHTML Format
×

6
2. Ability to create custom organization specific data
quality indicators – New Features
 Easy to understand the data quality indicators through multiple


reports
Flexibility to compare the expressions
View Result to display progress for the background processes
7
2. Ability to create custom organization specific data
quality indicators – New Features
 Easy to understand the data quality indicators through multiple reports
8
2. Ability to create custom organization specific data
quality indicators – New Features
 Flexibility to compare the expressions
9
2. Ability to create custom organization specific data
quality indicators – New Features
 View Result to display uploading progress
10
3. Advanced comparison of data profiling and quality
indicators - Comparison
Features
4.1
5.0
Quick compare of profiles/DQIs


Create Dataset & Template


Graphical representation of profile/DQI compare


Default Graph
Line
3D Bar
Displaying table report with visualization
×

Hierarchical folders division
×

11
3. Advanced comparison of data profiling and quality
indicators – New Features
 Additional table report and chart type
• Display of both success and failure count and percentages
• Informative table report display with charts
12
4. Advanced comparison of data profiling and quality
indicators – New Features
 New design of data displayed in dataset table, for Profile & DQI
 Modified default report as 3D Bar
13
4. Ability to create and execute data quality projects
against varying payloads – New Features
•
•
•
Data Quality Projects for Profiles/DQIs enables running a DQC plan for
generating a Data Profile/DQI and loading into database repository
• Create data quality project
• Manage data quality project
Reusability of plan - The same plan can be executed against different inputs
View Result for different level of progress
14
4. Ability to create and execute data quality projects
against varying payloads – New Features
 Create project – option to upload Profiling plans and DQI plans
 Plan can be with drill-through and also with Rollups
15
4. Ability to create and execute data quality projects
against varying payloads – New Features
 Manage project
16
4. Ability to create and execute data quality projects
against varying payloads – New Features
 Reusability of plan : The same plan can be executed against different inputs
17
4. Ability to create and execute data quality projects
against varying payloads – New Features
Note:
 Plan is supported with TXT and CSV Input, Other Input in DQC like XML,


Database is not supported in the current version
Output can only be profile steps (or DQI steps depending on the type of DQ
project)
The following input settings of the input file can be modified without DQC GUI:
• Number Of Lines In Header
• Field Separator
• Line Separator
• String Qualifier
• Encoding
18
4. Ability to create and execute data quality projects
against varying payloads – New Features
 View Result for different level of progress
19
5. Ability to rollup data with drill through – New Features
Roll up is a part of iWay Data Profiler that enables a user to profile view a specific
subset of a data.
Rollups for:
 Data Analysis
 Primary Key
 Business Rules.
20
5. Ability to rollup data with drill through – New Features
 Enabled drill-through for the subset of data
21
6. Advanced administration capability – Comparison
Features
4.1
5.0
Create Roles, Groups and Users


LDAP User Authentication and LDAP Role Authorization


Option to create, delete and manage folders
×

Group and Role details can be viewed while creating a user
×

Groups have access to one or more Virtual Folders
×

Base Role : Admin, Power, Technical and Analytical with different
privileges
×

iSM Tracking privilege
×

DQ Project privilege
×

Rollup privilege
×

22
6. Advanced administration capability – New Features
• Hierarchical folder structure to manage profiles, DQIs, datasets, templates
 Grouping related things together based on the user choice
 Sharing and easy to compare
 Created a group called “Admin” and linked all folders to it
 Folder structure for a user
Private
o My Group
o Custom
o Public
Created default base roles: admin, power, technical and analytical with
varying authorizations
o
•
23
6. Advanced administration capability – New Features
 Manage Folders
24
6. Advanced administration capability – New Features
 Manage Roles
25
6. Advanced administration capability – New Features
 Manage Groups
26
6. Advanced administration capability – New Features
 Manage Users
27
7. Advanced auditing capability – New Features
 Audit (Tracking user name, IP address, login time and logout time) with
different databases
• Enhanced user security
• Filter audit information
• Track log in and log out time in iDP with IP Address
• Track invalid login & iDP server manager login
28
7. Advanced auditing capability – New Features
 Audit information for a user
29
7. Advanced auditing capability – New Features
 Filter audit information
30
8. Easier migration from earlier versions – New Features
Includes
Upgrade script is for migrating from the older version to newer version
 4.1  5.0
Available for:
 MySql
 MSSQL
 Oracle
There is no loss of data exists in database “idp_data”
Does not include
Synonym creation
Drill-through adapter creation in WebFOCUS
31
9. Ability to track background processes for faster
troubleshooting and status – New Feature
 Interface for the user to monitor the back-end process
 Multilingual support for view process result – Available for Profile, DQI, DQ
Project and Custom Reports
32
iWay Data Profiler : Architecture
33
iWay Data Profiler - Component Architecture
34
Benefits
 Web enabled thereby single deployment and access anywhere
 Aids in the identification of data quality issues
 Expertise and support of a mature business intelligence product –





WebFOCUS
Data quality comparison over time
Documents and audits quality-control processes for review
Streamlines offline maintenance activities to ensure optimum performance
Share the reports using a secured connection
Mobile support
35
iWay Data Profiler
Thank You !
Anand Maheshwari
5.0 GA
36
Appendix – Additional Features
37
Multilingual support
 iWay Data Profiler supports multi-languages
 English
 German
 Spanish
 French
 Note: ISM Tracking Edit page language will change corresponding to the
language choose in ISM configured not based on the IDP chosen language.
38
Performance tuning
 JAVA code has been restructured for better performance
• Complete recoding using XPATH and transaction based multiple coding
•
being undertaken as processing of complicated Rollups found to crash the
JVM
The maximum size of the Profile XML file, which can be uploaded into DP,
has been increased from 1MB to 300 MB
39
Revamped iWay Data Profiler with JQuery

JQuery Functionalities
• Enhance look and feel
• Styling
• Cross browser compatibility
• Support all mobile devices
• Grouping related things together
• Sharing
• Easy to compare
• Fast loading
• Alerts and alignment
• Expand All/ Collapse All functionality
• More/ Less functionality
• Pagination
• Homepage carousel – to show case different graphs of iDP
• Landing page accordion layout – to display different iDP modules
40
Modified both Windows and Linux installer
 Updated both Windows and Linux Installer scripts for DQ Project to get the
Drill-through DB credential from the installer
 Installer is available in English, German, French, Spanish
 Same Linux build is supported for AIX platform
41
Backward Compatibility with iWay Data Quality Center
 Modified iDP for
 Profiling
 DQI
 Support across versions
 DQC 7.0.1 & higher
 Input data
 Reports
42