A WEB-BASED DATA WAREHOUSE ANALYSIS TOOL FOR MARKET RESEARCH
A Project
Presented to the faculty of the Department of Computer Science
California State University, Sacramento
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF SCIENCE
in
Software Engineering
by
Adekunle Adefela
SPRING
2012
© 2012
Adekunle Adefela
ALL RIGHTS RESERVED
ii
A WEB-BASED DATA WAREHOUSE ANALYSIS TOOL FOR MARKET RESEARCH
A Project
by
Adekunle Adefela
Approved by:
__________________________________, Committee Chair
Dr. Meiliu Lu
__________________________________, Second Reader
Dr. Du Zhang
____________________________
Date
iii
Student: Adekunle Adefela
I certify that this student has met the requirements for format contained in the University format
manual, and that this project is suitable for shelving in the Library and credit is to be awarded for
the project.
__________________________, Graduate Coordinator
Dr. Nikrouz Faroughi
Department of Computer Science
iv
___________________
Date
Abstract
of
A WEB-BASED DATA WAREHOUSE ANALYSIS TOOL FOR MARKET RESEARCH
by
Adekunle Adefela
In the Mystery Shopping, Customer, and Market research field, data is collected about
shopping experiences, customer reactions, store evaluations, employee evaluations,
pricing evaluations, and brand/competitor evaluations. An efficient way of storing and
formatting the data is needed to facilitate data analysis, data interpretation, and the
generation of meaningful reports. This Masters project involves designing and
developing a data warehouse to store evaluation data. It also involves creating a dynamic
web-based tool to run analysis queries and generate meaningful reports from the data
warehouse.
The tasks of the project included client interviews, detailed analysis of an operational
database, and understanding the current processes and steps taken to manually create
summary reports. Important subject areas were then identified for creating data marts and
v
implemented using an incremental approach. The data warehouse and analysis tool was
developed using a Microsoft SQL Server 2008 database, and ASP and ASP.NET.
_______________________, Committee Chair
Dr. Meiliu Lu
_______________________
Date
vi
ACKNOWLEDGEMENTS
I would like to thank Dr. Meiliu Lu for being my project advisor, and guiding me through
the two semesters it took to complete this project. I learned a lot from taking her Data
Warehousing and Data Mining class, CSC 177, in Fall 2011. Her advice and direction
were invaluable in helping me complete this project.
I would also like to thank National Shopping Service for giving me the opportunity to
work on this project for them. They provided me with access to their applications,
operational database, and real-life data, which made it easier for me to develop an
effective solution.
Finally, I would like to thank my family and friends for their support and encouragement
throughout my masters program. I could not have done it without them. I would like to
thank my Mom, for her love, prayers, and encouragement. And I would like to thank my
Dad for his advice, encouragement, and believing in me. I wouldn’t be where I am today
without you two.
vii
TABLE OF CONTENTS
Page
Acknowledgments.................................................................................................................. vii
List of Tables ............................................................................................................................ x
List of Figures .......................................................................................................................... xi
Chapter
1. INTRODUCTION ……………..……………………………………………………….. 1
1.1 Motivation.............................................................................................................. 2
1.2 Research ................................................................................................................. 3
2. BACKGROUND ................................................................................................................ 4
2.1 Client …................................................................................................................. 4
2.2 Technology ............................................................................................................ 5
3. USER REQUIREMENTS .................................................................................................. 7
3.1 Functional Requirements ....................................................................................... 8
3.2 Non-functional Requirements .............................................................................. 11
4. SYSTEM DESIGN ........................................................................................................... 13
4.1 High level Component Design ............................................................................. 13
4.2 Data Warehouse Design....................................................................................... 14
4.3 Web-based Data Analysis Tool Design ............................................................... 19
4.4 ETL Process Design............................................................................................. 22
5. IMPLEMENTATION ....................................................................................................... 24
5.1 Data Warehouse Implementation ......................................................................... 24
5.2 Web-based Data Analysis Tool Implementation ................................................. 28
5.3 ETL Process Implementation ............................................................................... 42
6. TESTING .......................................................................................................................... 43
6.1 Data Warehouse Testing ...................................................................................... 43
6.2 Web-based Data Analysis Tool Testing .............................................................. 44
6.3 ETL Process Testing ............................................................................................ 47
7. CONCLUSION ................................................................................................................. 48
viii
Appendix A: Data Warehouse Schema Diagrams ............................................................ 49
Appendix B: Webpages Code ............................................................................................... 54
Bibliography ...... ……………………………………………………………………………..70
ix
LIST OF TABLES
Tables
Page
Table 1: List of Fact tables............................................................................................................. 14
Table 2: List of Dimension tables .................................................................................................. 15
x
LIST OF FIGURES
Figures
Page
Figure 1: Use Cases ......................................................................................................................... 8
Figure 2: Component design .......................................................................................................... 13
Figure 3: Question Answers Schema Diagram .............................................................................. 16
Figure 4: Aggregate Tables Schema Diagram ............................................................................... 18
Figure 5: Three-tier Architecture ................................................................................................... 19
Figure 6: Page Layout .................................................................................................................... 20
Figure 7: ETL Process ................................................................................................................... 23
Figure 8: RatingAnswerScores simplified schema ........................................................................ 25
Figure 9: RatingAnswerScores average question score report ...................................................... 27
Figure 10: Login page .................................................................................................................... 29
Figure 11: Summations Lobby....................................................................................................... 29
Figure 12: Category Scores filters ................................................................................................. 31
Figure 13: Category Scores Report ................................................................................................ 31
Figure 14: Category Scores Trend Report Filters .......................................................................... 32
Figure 15: Category Scores Trend Report ..................................................................................... 33
Figure 16: Day of Week Performance Report Filters .................................................................... 34
Figure 17: Day of Week Performance Report ............................................................................... 34
Figure 18: Time of Day Performance Report Filters ..................................................................... 35
Figure 19: Time of Day Performance Report ................................................................................ 36
Figure 20: Top and Bottom Performers Report ............................................................................. 37
Figure 21: Sites Category Ranking Report .................................................................................... 38
Figure 22: Sites Listing Report ...................................................................................................... 39
Figure 23: Manage Account Summary Suite ................................................................................. 40
Figure 24: Manage User Summary Access .................................................................................... 41
Figure 25: Summary Reports Validation ....................................................................................... 46
xi
1
Chapter 1
INTRODUCTION
The objective of this project is to design and develop a data warehouse and web-based
analysis tool for, National Shopping Service, a Mystery Shopping & Customer Research
company. National Shopping Service collects data for its clients through online surveys
and evaluations. The data collected, can vary from hundreds to millions of records
depending on the client. The data collected is then processed and aggregated into
meaningful reports to be presented to their clients. Currently, evaluation data is exported
from operational databases to MS access and Excel and used to manually generate
periodic summary reports. This project provides an efficient way to store customer
evaluation data such that it can easily be queried and used to generate meaningful reports
automatically and through a web interface. The customer evaluation records that will be
stored could number in the millions, so it was essential to design an efficient data
warehouse to accommodate such a large number of records.
The introduction of a data warehouse will add value to National Shopping Service by:
a.) Storing evaluation data in a uniform and easily accessible format:
This project will allow data to be stored at different levels of granularity such that
meaningful reports can be generated quickly and efficiently across all accounts.
b.) Automate current manual processes and facilitate the generation of new reports:
This project involves designing and developing an enterprise data warehouse with
data marts for the different subject areas that exist across accounts and creating a
2
web interface that will give users the ability to run queries and analysis on the
data.
c.) Providing an automated Extraction, Transformation, and Load (ETL) process:
The data warehouse and ETL process will make it possible to transfer data from
any operational Database as long as an adapter is developed to convert the data.
This is important in case National Shopping Service decides to change the
structure of their operational databases or the format in which their evaluation
data is stored. It would also allow them to accommodate clients who want to
migrate their historical data from external systems to the National Shopping
Service system to run historical analysis.
1.1.
Motivation
My interest in developing interactive web-based applications and solving complex problem is one
of the motivating factors for selecting this project. I also have a great deal of knowledge about the
Mystery Shopping and Customer Research field having worked in it for over 5 years. From my
experience in this field, I have learned that the presentation of meaningful data to the client is a
major factor in acquiring new clients and keeping existing ones. Therefore, the ability to generate
dynamic summary reports automatically is a requirement for any Mystery Shopping company to
compete in this growing industry. From building Online Transaction Processing Mystery
Shopping applications, I realized that the relational database model was not the best way to
generate aggregates and summary reports. A few attempts were made to run aggregates and
summaries directly from the operational database in real-time, but usually led to slow running
queries which also affected the speed of other applications. I had heard about Data warehousing
3
and Data Mining and knew that was the way to go to solve this problem. Because I had little
knowledge of Data warehousing and Data Mining, I decided to enroll in CSC 177, a Data
Warehousing and Data Mining class at CSUS, to learn more about this subject area. In addition to
the class, I decided to do a Masters project that involved developing an Enterprise Data
Warehouse that would make it possible to generate summary reports.
1.2. Research
A great deal of research was needed to effectively tackle the problem of storing evaluation data in
a uniform format and generating meaningful summary reports. I had to acquire a firm
understanding of the current structure of the operational databases the data was going to be
extracted from, identify the different formats in which the data was stored, and understand the
current processes that are followed to generate summary reports manually. I was able to acquire
this knowledge by analyzing existing documents and applications used to collect the data,
analyzing the existing data models and operational database structure, as well as inspecting
existing summary reports and interviewing the people that generate the summary reports. Taking
CSC 177 in the Fall of 2011 helped me acquire the knowledge and experience I needed to
understand and learn how to develop data warehouses. The Lessons, assignment, exams, and
projects equipped me with all the tools I needed to complete this project.
In this report, I will outline and describe the steps that I took to complete this project. In
Chapter 2, I will give the background of the project, followed by the User Requirements
and Systems Design in Chapters 3 and 4. Chapters 5 and 6 will cover the Implementation
and Testing of the project, followed by the Conclusion in Chapter 7.
4
Chapter 2
BACKGROUND
This Chapter will give you an introduction to the client, the problem that required a
solution, and the technology involved in developing a solution for this project.
2.1. Client
This project was developed for a company called National Shopping Service (NSS). NSS
is a leading Mystery Shopping provider that has been in the business since 1972. They do
business with both domestic and international companies and receive thousands of
surveys and evaluations each day from their 300, 000+ Mystery Shoppers. In the Mystery
Shopping field, data is collected about shopping experiences, customer reactions and
needs, store evaluations, employee evaluations, pricing evaluations, and brand
evaluations. The data collection is done in various ways including online customer
evaluations, surveys, call center data collection, audits, as well as web data scraping. All
this data is entered and stored into databases and used to generate individual reports as
well as periodic reports and summaries that are delivered and presented to clients.
Currently only a few individuals with knowledge and experience in statistics, MS Excel
and Access are able to analyze and interpret the data to create meaningful reports. There
is a need for the raw data that resides in multiple account tables to be formatted, cleaned
up, and transformed automatically into a form that would be easy and efficient to access,
analyze, and manipulate to create dynamic and interactive reports and summaries through
a web interface. The current process of generating monthly summary reports costs the
5
company time and money each month that could be applied to other tasks. The current
process also limits the availability of client summaries to static summary reports that are
generated periodically. National Shopping service would like the ability to offer their
clients a suite of dynamic summary reports that could be generated automatically and
through a web-based interface.
2.2. Technology
This section gives a brief overview of the technology used in the development of this
project.
2.2.1. ASP.NET 4.0
ASP.NET is a powerful Microsoft web framework used to develop dynamic websites and
web applications [4]. It provides a means for webpages to query, insert, and edit records
in a database, and makes it possible to develop interactive applications that give users
control of different features of the applications. ASP.NET consists of HTML, JavaScript,
as well as a scripting or programming language of your choice. The most popular
languages used in ASP.NET are VB.NET and C#. ASP.NET 4.0 is the most recent
version of ASP.NET which provides a host of addition features, functions, and controls
that were not available in previous versions. One of the recent additions is the chart
control, which provides the ability to generate graphs and charts dynamically from a data
source. This new addition eliminates the need to purchase a 3rd party graphing component
which is often expensive. APS.NET also provides an easy means to integrate Microsoft
SQL Server with web applications.
6
2.2.2. Microsoft SQL Server 2008
SQL Server is a Microsoft Database Management System (DBMS) that allows users to
create and manage Relational Database systems as well as Data Warehouses. This gives
users the ability to create tables and objects to store data in an organized and easily
retrievable manner. Using queries and store procedures, data can be inserted into, edited,
and retrieved from database tables with ease from different types of applications included
ASP.NET web applications. In addition, MS SQL Server also provides a host of other
tools like SQL Server Integration Services (SSIS), which can be used to perform other
intensive processes that could be scheduled and run at different times.
2.2.3. JavaScript/Ajax
JavaScript is a client-side scripting language that allows users to execute functions
without sending data to the server and waiting for a response. Ajax, also known as
Asynchronous JavaScript is a library of JavaScript functions that makes it possible to
develop web applications that have the look and feel of desktop applications.
7
Chapter 3
USER REQUIREMENTS
The user requirements represent the features and functionality the client is expecting from
the finished product at the end of the project. User requirements vary from functional
requirements, which involve tasks and functions that can be performed using the tool, to
non-functional requirements like performance and usability. To effectively develop a
solution that meets the client’s needs, it was important to conduct requirements gathering
and analysis to uncover and document the user requirements of this project. This involved
numerous meetings to get input from the client on what they wanted in the solution. First,
to get an overall description of the problem they wanted to solve in their own words, and
what they thought was needed to solve the problem. I was also granted access to the
operational database, applications, sample reports, and data to analyze and help with my
requirements gathering process. After weeks of meetings with the client and analyzing
the materials and data given to me, I was able to identify and document the functional
and non-functional user requirements for this project.
8
3.1. Functional Requirements
These are the features and functions that can be performed using the tool. The Use Case
diagram in Figure 1, gives an overview of the available features in the web-based
analysis tool and the users they are associated with. Three types of users can access the
web-based analysis tool. The client summary user who can login and generate reports, a
client admin user who can generate summary reports and manage user access permission,
and a NSS Admin user who has the ability to generate reports, and manage account and
user access permissions.
Web-based Analysis Tool
Login/Logout
Client Summary User
Generate Summary
Reports
Export Summary
Reports
Manage User
Account Setings
Client Admin User
NSS Administartive User
Manage Account
Settings
Figure 1: Use Cases
9
3.1.1. User login/logout
Access to the web-based data analysis tool must be restricted to select users granted
access to the tool. Users granted access to the web-based data-analysis tool should have
the ability to login to the tool and have access to the suite of reports they have been
assigned. The users should also have the ability to logout of the tool to end their session
once they are done using the tool.
3.1.2. Generate Summary Reports
Users that login to the data analysis tool should be presented with a suite of summary
reports that they can generate using the tool. A menu with links to the summary reports
should be on the lobby of the tool for the users to select from after they login. The
different summary reports should allow the user to enter and submit various search
criteria and filters into a form. Upon submission of the form, a summary report consisting
of charts and/or tabular data should be displayed to the user. Users may have access to
one or more of seven summary reports available through the data analysis tool.
The following seven summary reports were required for this project:
a.) Category Scores Report: This is a report to generate the Category/Section Scores
for client Sites, Regions, and entire company by month, period, or year.
b.) Category Scores Trend: This is a report to generate trending Category/Section
Scores for client Sites, Regions, and Entire company by month, period, or year.
10
c.) Day of Week Performance: This is a report to compare Evaluation scores and
store performance with the number of employees and customers present in the
store by day of week.
d.) Time of Day Performance: This is a report to compare Evaluation scores and store
performance with the number of employees and customers present in the store by
time of day.
e.) Top and Bottom Performers: This is a report to view a client’s top and bottom
performing sites by month, period, or year.
f.) Category Ranking: This is a report to rank a client’s sites by category/section
score and performance.
g.) Sites Listing: This is a report to generate a client’s sites listing with average
evaluation scores.
3.1.3. Export Summary Reports
Users should have the ability to export and save summary reports they generated to PDF
and Excel. Users should also have the ability to export and download tabular data for
generated reports in excel format that can later be used to run their own external custom
reports. Users should be able to select they export option when generating the summary
report.
11
3.1.4. Manage Account Summary Settings
NSS Administrative level users should have the ability to add and remove summary
reports from the suite of summary reports available for an account. This feature restricts
the summary reports that can be generated by all the users under specific accounts.
3.1.5. Manage Account User Settings
Administrative level users should have the ability to assign and restrict the access of
account users to specific summary reports in the suite of summaries available under that
account. This feature restricts the summary reports that can be accessed and generated by
specific users under any account.
3.2. Non-functional Requirements
These are requirements that can be used to judge the operation of a system rather than its
specific behaviors [5]. This section gives a brief overview of the non-functional
requirements of this project.
3.2.1. Performance
Users should be able to generate reports for the criteria entered instantly. The web-based
tool should be able to generate and display summary reports to the user in a few seconds.
A generation response time of more than 2 seconds in unacceptable for a user request.
12
3.2.2. Usability
Users should be able to navigate to summary reports easily with a few clicks without
problems. Links to summary reports should be display in an easy to read menu located on
the lobby of the analysis tool. Search fields and filters used to generate reports should be
organized and display in a simple and intuitive manner for the user. A help button or link
to access instructions or to contact an NSS account manager must be available to the
client and easy to locate in the tool.
13
Chapter 4
SYSTEM DESIGN
To ensure that the design and development of the system did not get drawn out and fail, I
decided to design, code and implement the different components of the system in an
incremental fashion. The system design involved the design of the various data marts in
the data warehouse in an incremental fashion and populating them with test data to ensure
that the data could be queried efficiently. After the design of the data warehouse was
completed, the web tool and ETL process were also designed and implemented in an
incremental fashion.
4.1. High level Component Design
This project consists of four main components; the NSS operational databases, an ETL
process to transform the data, the Enterprise Data Warehouse that would store evaluation
data and aggregates, and the Web-based Data Analysis tool used to generate reports.
1
3
2
5
4
F1:
EDW
ETL
Process
Operational Databases
10%
10%
10%
F2:
Option 1
10%
10%
Option 2
10%
10%
10%
10%
10%
Enterprise Data Warehouse
Web-based
Data Analysis Tools
Figure 2: Component design
14
4.2. Data Warehouse Design
Designing the data warehouse required a firm understanding of the tools and processes
involved in gathering the evaluation data in the operational database. The processes
involved in generating the summary reports manually also had to be analyzed and
documented. The data warehouse consists of fact tables that store evaluation data at
different levels of granularity, and dimension tables that store data that describes and
gives context to the data in the fact tables. Aggregate tables were also designed to store
various levels of aggregation of the data in the fact tables.
4.2.1. Fact Tables Design
Based on my analysis, I determined that the grain or lowest unit of measure was the
question answer. I was able to identify 6 fact tables that would be needed to store the
different question answer types. All the fact tables have similar designs with the only
difference being the field containing the measure or answer value.
Table 1: List of Fact tables
Table Name
Description
RatingAnswerScores
Contains answer values that have a numeric representation
or equivalent for an answer.
Contains answer value to indicate time duration in hours,
minutes, and seconds.
Contains answer values that indicate the number of
occurrences of a person, thing, or action.
Contains answer values that are monetary values.
Contains answer values that indicate the time of day certain
events occurred.
Contains answer values that are text values and do not have
a numeric representation and are not picked from a list of
options.
DurationValues
CountValues
PriceValues
TimeValues
TextAnswerValues
15
4.2.2. Dimension Tables Design
I was able to identify 17 possible dimension tables needed to describe and give context to
the data in the fact tables.
Table 2: List of Dimension tables
Table Name
Description
ShopDate
ShopTimes
ShopMonth
Contains information about the date evaluations were performed.
Contains information about the time evaluations were performed.
Contains information about the month evaluations were
performed.
Contains information about the year evaluations were performed.
Contains information about the client period evaluations were
performed.
Contains information about the client quarter evaluations were
performed.
Contains information about the client accounts that summary
reports are generated for.
Contains information about client sites.
Contains information about client regions.
Contains information about individual evaluation reports.
Contains information about the type of shop an evaluation is.
Contains information about evaluation scenarios. Comprises of a
combination of 2 shop types.
Contains information about evaluation report versions. Report
versions are iterations of a report scenario over time.
Contains information about sections or categories in an
evaluation report.
Contains information about the individual questions in an
evaluation report.
Contains information about the answer types of questions in an
evaluation report.
Contains information about individual answer options of
questions in an evaluation report.
ShopYear
ClientPeriods
ClientQuarters
ClientAccounts
Sites
SiteRegions
Reports
ReportShopTypes
ReportScenarios
ReportVersions
Sections
Questions
RatingTypes
RatingOptions
16
The schema diagram in Figure 3 shows the RatingAnswerScores,
OccurrenceCountValues, and DurationInSecondsValues fact tables and their associated
dimension tables. This schema diagram is the basis for generating Section and Evaluation
scores aggregates that populate the aggregate tables in the data warehouse.
ClientAccounts
PK
ShopCode
ClientName
Sections
PK
SectionKey
SectionId
V_ID
SectionTitle
SectionTitleFull
SectionRank
SectionPointsPossible
Sites
PK
SiteKey
FK1
clientId
ShopCode
SiteId
SiteName
SiteRegionId
SiteRegions
PK
RatingScoreId
FK8
FK2
FK6
FK5
FK3
FK1
FK4
FK7
ReportVersions
VersionKey
ClientId
V_ID
ScenarioId
VersionPattern
VersionTitle
StartDate
EndDate
PK
ShopDateId
FK1
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
OccurrenceCountValues
PK
FK8
FK2
FK6
FK5
FK1
FK4
FK3
FK7
CountValueId
QuestionId
SectionId
SectionKey
ReportId
V_ID
ClientId
SiteKey
ShopDateId
EvaluatorId
countValue
ShopMonth
PK
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ShopYear
PK
ShopYearId
ShopYearNum
DurationInSecondsValues
DurationValueId
FK8
PK
RatingOptionId
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
RatingPointValue
isSelected
SiteRegionId
ClientId
RegionName
RegionAbbr
ShopCode
ShopDate
RatingAnswerScores
ClientId
FK2
FK6
FK5
FK3
FK1
FK4
FK7
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
durationValue
Evaluators
PK
EvaluatorId
Gender
Ethnicity
DateOfBirth
ShopperNumber
Questions
PK
QuestionId
Reports
PK
V_ID
SectionId
QuestionLabel
Questiontext
PointsPossible
OptionCount
ReportId
ShopCode
ClientId
V_ID
Rpt_No
Figure 3: Question Answers Schema Diagram
17
4.2.3. Aggregate Tables Design
By analyzing the dimension tables, I was able to identify a number of potential aggregate
fields to group by to create aggregate tables. Aggregate tables are tables that contain precalculated data that is generated and stored for later use [2]. The advantages of aggregate
tables are felt in the form of increased performance and faster query execution. Potential
aggregates were determined by identifying fields in dimension tables where meaningful
groupings could be made to calculate aggregates.
Table 3: List of Potential Aggregate fields
Dimension
ReportVersions
ClientPeriods
ShopDate
Questions
RatingOptions
Sections
ClientAccounts
Sites
Potential Aggregate fields
VersionId, VersionTitle
Period, Year
Day, Month, Year, Weekday, Weekend,
Holiday
QuestionText, QuestionLabel
OptionText
SectionTitle
ClientName, ShopCode
SiteId, SiteName, City, State, Zip, Country,
Region, District, Zone
18
The Schema diagram in Figure 4 shows the Section Score aggregate tables and their
associate dimension tables. Reports such as the Category Scores and Category Scores
Trend reports are generated using the data in these aggregate tables.
SectionScores
ClientAccounts
PK
ClientId
ShopCode
ClientName
Sections
PK
SectionKey
SectionId
V_ID
SectionTitle
SectionTitleFull
SectionRank
SectionPointsPossible
Sites
PK
SiteKey
FK1
clientId
ShopCode
SiteId
SiteName
SiteRegionId
SiteRegions
PK
PK
SectionScoreId
FK2
FK3
FK5
FK1
FK4
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
ShopperId
SectionPointsEarned
SectionPointsPossible
SectionScoreValue
aggrSiteMonthReportVersionSectionScore
FK5
FK1
FK3
FK4
FK2
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
ShopMonth
PK
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
aggrRegionMonthReportVersionSectionScore
FK1
FK2
FK3
FK4
FK5
ReportVersions
ClientId
SiteRegionId
ShopMonthId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
VersionKey
ClientId
V_ID
ScenarioId
VersionPattern
VersionTitle
StartDate
EndDate
Reports
PK
FK1
ShopDateId
SiteRegionId
ClientId
RegionName
RegionAbbr
ShopCode
PK
ClientId
SiteKey
ShopMonthId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
ShopDate
PK
aggrAccountMonthReportVersionSectionScore
FK1
FK2
FK3
FK4
ClientId
ShopMonthId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
ReportId
ShopCode
ClientId
V_ID
Rpt_No
Figure 4: Aggregate Tables Schema Diagram
19
4.3. Web-based Data Analysis Tool Design
In designing the data analysis tool, I used a three-tier architecture to divide components
of the tool into 3 layers. The presentation layer consists of the front-end design of the
look and feel of the web interface. The business logic layer contains the back-end code
that is used to communicate with the data layer which consists of the data warehouse,
queries, and stored procedures.
Presentation Layer
(Webpages, HTML, ASP.NET, Chart controls)
Business Logic Layer
(ASP.NET, VB.NET code behind)
Data Layer
(SQL Server, Stored Procedures)
Stored Procedure
Stored Procedure
Data Warehouse
Figure 5: Three-tier Architecture
Stored Procedure
20
4.3.1. Presentation Layer
In the design of the web interface, it was important to ensure that the layout and
navigation of the tool were clean and simple. A key user requirement for the tool was
usability with a minimum number of clicks to reach the various summary reports. As a
result, a simple layout design with a header, footer, navigation menu with 2 buttons, and
a content area in the middle was selected. In ASP.NET, Master Pages are used to create a
template page that can be applied to all the pages across the application. This helps
maintain uniformity and allows changes that need to be made across the application to be
made in one place. Figure 6 show a screen-shot of the webpage layout used to create the
master page for the web-based analysis tool.
Figure 6: Page Layout
4.3.2. Business Logic Layer
Separating the business logic from the presentation layer makes it easy to change the
presentation layer without affecting the backend code that handles the logic and processes
21
of the application. In ASP.NET storing the logic in the code-behind pages gives
programmers the freedom to code without worrying about the presentation layer.
Variables, classes, event-handlers, functions, and sub-routines for this application were
written in VB.NET and stored in a code-behind page for each webpage in the application.
This layer is the communication link between the presentation layer and the data layer;
taking form inputs and parameters, performing required logic and data access functions
with the data layer, and then returning the results for display in the presentation layer.
4.3.3. Data Layer
This layer consists of the data warehouse and database objects that are used to store and
manipulated the evaluation data that is transformed from the operational databases. Fact,
dimension, and housekeeping tables store the data in the format needed. To ensure
reusability and modularization, all logic and interactions performed with and within the
database are done through stored procedures and user-defined functions. The application
only has permissions to execute certain stored procedures and functions that are required
to perform its functions. In addition, for security reasons, the direct querying of the
database is limited to users that have database administrator access.
22
4.4. ETL Process Design
The ETL process was designed as a scheduled process to run periodically to transform
new and modified data from the operational database to the data warehouse. It comprises
of ActiveX components and various SQL Server Integration Services (SSIS) components
needed to copy the required data from the sources, manipulate the data as needed, and
store the results in the various tables of the data warehouse. Figure 7 shows the process
flow chart for the ETL process.
23
Start ETL Process
Query Reports &
Sites tables in
Operational DB for
new and modified
records
Loop Through
Accounts
Yes
Copy Reports & Sites
data to Temp
staging tables in
Data Warehouse
Update Existing Site
information in
Dimension tables
Update Existing
Reports information
in Dimension & Fact
tables
Yes
Yes
Existing sites in DW
need updating
No
Add new Sites
information in
Dimension tables
Add new Reports
information in
Dimension & Fact
tables
Yes
Yes
New sites in DW
need Adding?
A
Records requiring
Warehousing
No
New reports in DW
need Adding?
Aggregates to be
generated?
Existing reports in
DW need updating
No
No
Determine and log
aggregates that
need to be re-run
No
End ETL Process
Yes
No
Generate
Aggregates
Figure 7: ETL Process
Determine and log
aggregates that
need to be re-run
A
24
Chapter 5
IMPLEMENTATION
This section consists of the steps that were followed to convert the user requirements and
design into a working system. The design and implementation of the data marts in the
data warehouse were done in an incremental fashion. Once the data warehouse tables
were created, the individual summary reports in the web-based tool were also created in
an incremental fashion. Finally, the ETL process was developed to populate the data
warehouse. Software required to develop the data warehouse and ETL process were SQL
Server Management Studio 2008 and Business Intelligence Development Studio. To
develop the Web application, Visual Studio 2010 was used. The testing and production
servers on which the data warehouse is housed required a SQL Server 2008 instance
installed and configured as well as IIS and .NET 4.0 framework install for the webpages
to run.
5.1. Data Warehouse Implementation
Because the data marts were very similar, the RatingAnswerScore fact table was first
created and tested to ensure that data could be efficiently loaded and queried. Once the
RatingAnswerScore data mart was completed and tested, it was replicated and used as a
template to create the remaining data marts. The process of creating the data marts
involved writing CREATE TABLE statements for the dimension and fact tables and
executing them in SQL Server Management Studio. After the tables were created, the
foreign key relationships between the tables were then applied. I then created INSERT
25
statements to load the tables with test data. Once the tables were loaded, I created a few
queries to test how aggregates would be calculated from the fact table’s data.
The following are examples of some of the SQL statements used in the implementation of
the RatingAnswerScores data mart in figure 8:
RatingAnswerScores
Questions
PK
QuestionId
V_ID
SectionId
QuestionLabel
Questiontext
PointsPossible
OptionCount
bigint identity
RatingScoreId
bigint
bigint
bigint
varchar(50)
varchar(100)
float
int
RatingOptionId
QuestionId
SectionId
V_ID
ShopDateId
SiteKey
RatingPointValue
isSelected
bigint
bigint
bigint
bigint
bigint
bigint
float
tinyint
FK3
FK1
FK2
Sites
PK
SiteKey
bigint identity
clientId
ShopCode
SiteId
SiteName
bigint
varchar(3)
varchar(50)
varchar(150)
ShopDate
PK
ShopDateId
bigint identity
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
bigint
smalldatetime
varchar(20)
smallint
varchar(50)
varchar(20)
smallint
bit
bit
bit
smallint
varchar(50)
varchar(20)
int
bigint
Figure 8: RatingAnswerScores simplified schema
a.) CREATE TABLE statements for RatingAnswerScores fact table:
USE [NssDataWarehouse]
GO
CREATE TABLE [dbo].[RatingAnswerScores](
[RatingScoreId] [bigint] NOT NULL,
[RatingOptionId] [bigint] NOT NULL,
[QuestionId] [bigint] NOT NULL,
[SectionId] [bigint] NOT NULL,
[SectionKey] [bigint] NOT NULL,
[ReportId] [bigint] NOT NULL,
[V_ID] [bigint] NOT NULL,
[ShopDateId] [bigint] NOT NULL,
[ClientId] [bigint] NOT NULL,
[SiteKey] [bigint] NOT NULL,
[RatingPointValue] [float] NOT NULL,
[isSelected] [tinyint] NOT NULL
) ON [PRIMARY]
GO
b.) CREATE TABLE statement for Questions dimension table:
USE [NssDataWarehouse]
26
GO
CREATE TABLE [dbo].[Questions](
[QuestionId] [bigint] IDENTITY(1000008,1) NOT NULL,
[V_ID] [bigint] NULL,
[SectionId] [bigint] NULL,
[QuestionLabel] [varchar](50) NULL,
[Questiontext] [varchar](100) NULL,
[PointsPossible] [float] NULL,
[OptionCount] [int] NULL,
CONSTRAINT [PK_Questions_1] PRIMARY KEY CLUSTERED
(
[QuestionId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS
ON) ON [PRIMARY]
) ON [PRIMARY]
GO
=
c.) CREATE TABLE statement for aggrSiteMonthQuestionRatingScore aggregate
table:
USE [NssDataWarehouse]
GO
CREATE TABLE [dbo].[aggrSiteMonthQuestionRatingScore](
[ClientId] [bigint] NULL,
[SiteKey] [bigint] NULL,
[ShopMonthId] [bigint] NULL,
[V_ID] [bigint] NULL,
[QuestionId] [bigint] NULL,
[occurCnt] [int] NULL,
[possibleCnt] [int] NULL,
[aggrScore] [float] NULL
) ON [PRIMARY]
GO
d.) Query testing aggregation by client, site, report version, and month:
SELECT
RatingAnswerScores.ClientId, RatingAnswerScores.SiteKey,
ShopDate.ShopMonthId, RatingAnswerScores.V_ID,
RatingAnswerScores.QuestionId, Count(isSelected), Qcnt,
(Sum((RatingAnswerScores.RatingPointValue)/convert(float,
Questions.PointsPossible))/convert(float, COUNT(*)))
FROM RatingAnswerScores
INNER JOIN Questions
ON RatingAnswerScores.QuestionId=Questions.QuestionId
INNER JOIN ShopDate
ON RatingAnswerScores.ShopDateId=ShopDate.ShopDateId
INNER JOIN
27
(
SELECT ClientId, SiteKey, ShopMonthId, V_ID, QuestionId,
Qcnt=COUNT(*)
FROM RatingAnswerScores s1
INNER JOIN ShopDate s2
ON s1.ShopDateId=s2.ShopDateId
Group By ClientId, SiteKey, ShopMonthId, V_ID, QuestionId
) tbl2
ON
RatingAnswerScores.ClientId=tbl2.ClientId And
RatingAnswerScores.SiteKey=tbl2.SiteKey And
ShopDate.ShopMonthId=tbl2.ShopMonthId And
RatingAnswerScores.QuestionId=tbl2.QuestionId
GROUP BY RatingAnswerScores.ClientId,
RatingAnswerScores.SiteKey, ShopDate.ShopMonthId,
RatingAnswerScores.V_ID, RatingAnswerScores.QuestionId, Qcnt
Figure 9 shows the report generated with the query above testing aggregation on the
RatingAnswerScores fact table grouped by client, site, report version, and month
Figure 9: RatingAnswerScores average question score report
28
5.2. Web-based Data Analysis Tool Implementation
The web-based tool was created in a manner to make it easy to modify and add new
features going forward. To make it easy to modify the theme and look of the entire
application, a master page was used to hold all the HTML and VB.NET code that is
common across all the webpages. This master page was then applied to all the webpages
in the application. Similar to developing the data warehouse, the reports in the web-based
tool have a similar layout and components, so I developed them incrementally and reused
common components. Functions and subroutines used in more than one page are stored in
a base page that is inherited by all the pages in the applications. All calls to the database
are done by passing parameters to stored procedures and displaying the returned results.
All the use cases from the user requirements were implemented incrementally.
5.2.1. Login/Logout
This feature allows specific users to login to the web-based tool by supplying their
account code, username, and password. If incorrect information is provided, an error
message is displayed to the user asking them to try again. Once the user supplies valid
login information, they are authenticated and redirected to the Summation Lobby that
displays a menu of summary reports to select from. Figures 10 and 11 show the user
login page and the summations lobby page an authenticated user is redirected to.
29
Figure 10: Login page
Figure 11: Summations Lobby
30
5.2.2. Generate Summary Reports
Authenticated users can select what summary reports they want to generate from the
menu of reports on the summation lobby. The summary reports available to a user are
dependent on the account’s suite of reports and the reports the user has been granted
access to.
a.) Category Score Report
This report allows a user to generate a report of the average category or section scores
over a period grouped by site, region, or account. The user is required to select the shop
scenario they want to generate the report for, the grouping and the corresponding site or
region, the date or period, and the categories they want to average. The user may also
select an export option of Html, PDF, or Excel. After selecting the appropriate values to
generate the report, the user must then click the “Generate” button to run the report. A
report containing a chart/visual representation and/or a table of values is then displayed to
the user. Figures 12 and 13 show the filters selected to generate the Category Scores
report and the resulting report.
31
Figure 12: Category Scores filters
Figure 13: Category Scores Report
32
b.) Category Scores Trend Report
This report allows a user to generate a trending report of the average category or section
scores over a period grouped by site, region, or account. The user is required to select the
shop scenario they want to generate the report for, the grouping and the corresponding
site or region, up to 3 trending periods, and the categories they want to average. The user
may also select an export option of Html, PDF, or Excel. After selecting the appropriate
values to generate the report, the user must then click the “Generate” button to run the
report. A report containing a chart/visual representation and/or a table of values is then
displayed to the user. Figures 14 and 15 show the filters selected to generate the Category
Scores Trend report and the resulting report.
Figure 14: Category Scores Trend Report Filters
33
Figure 15: Category Scores Trend Report
c.) Day of Week Performance Report
This report allows a user to generate a report to compare the average evaluation scores
over a period grouped by site, region, or account to the number of employees and number
of customers present in the stores on a specific day of the week. The user is required to
select the shop scenario they want to generate the report for, the grouping and the
corresponding site or region, and a period. The user may also select an export option of
Html, PDF, or Excel. After selecting the appropriate values to generate the report, the
user must then click the “Generate” button to run the report. A report containing a
chart/visual representation and/or a table of values is then displayed to the user. Figures
34
16 and 17 show the filters selected to generate the Day of Week Performance report and
the resulting report.
Figure 16: Day of Week Performance Report Filters
Figure 17: Day of Week Performance Report
35
d.) Time of Day Performance Report
This report allows a user to generate a report to compare the average evaluation scores
over a period grouped by site, region, or account to the number of employees and number
of customers present in the stores with respect to the time of day. The user is required to
select the shop scenario they want to generate the report for, the grouping and the
corresponding site or region, and a period. The user may also select an export option of
Html, PDF, or Excel. After selecting the appropriate values to generate the report, the
user must then click the “Generate” button to run the report. A report containing a
chart/visual representation and/or a table of values is then displayed to the user. Figures
18 and 19 show the filters selected to generate the Time of Day Performance report and
the resulting report.
Figure 18: Time of Day Performance Report Filters
36
Figure 19: Time of Day Performance Report
e.) Top and Bottom performers Report
This report allows a user to generate a report to view the top and bottom performers
based on the average evaluation scores over a period grouped by site, or region. The user
is required to select the shop scenario they want to generate the report for, the grouping
and the corresponding site or region, a period, and the Top/Bottom count. The user may
also select an export option of Html, PDF, or Excel. After selecting the appropriate
values to generate the report, the user must then click the “Generate” button to run the
report. A report containing a chart/visual representation and/or a table of values is then
displayed to the user. Figures 20 shows the filters selected to generate the Top and
37
Bottom performers report and the resulting report.
Figure 20: Top and Bottom Performers Report
38
f.) Sites Category Ranking Report
This report allows a user to generate a report to rank the performances of sites based on
the average evaluation scores or category scores over a period. The user is required to
select the shop scenario they want to generate the report for, a period, and the category
they want to rank by. The user may also select an export option of Html, PDF, or Excel.
After selecting the appropriate values to generate the report, the user must then click the
“Generate” button to run the report. A report containing a chart/visual representation
and/or a table of values is then displayed to the user. Figures 21 shows the filters selected
to generate the Sites Category Ranking report and the resulting report.
Figure 21: Sites Category Ranking Report
39
g.) Sites Listing Report
This report is similar to the Sites Category Ranking Report. It allows a user to generate a
report to list the sites and average evaluation and category scores over a period. The user
is required to select the shop scenario they want to generate the report for, and a period.
The user may also select an export option of Html, PDF, or Excel. After selecting the
appropriate values to generate the report, the user must then click the “Generate” button
to run the report. A report containing a table of values is then displayed to the user.
Figures 22 shows the filters selected to generate the Sites Listing report and the resulting
report.
Figure 22: Sites Listing Report
40
5.2.3. Manage Account Settings
This feature gives NSS Administrative users the ability to manage the suite of summary
reports available to an account. Because clients may require different reports, this tool
makes it possible to add or remove summary reports from an account’s suite of reports.
To add a summary, the user may select a report from the drop down of summaries and
click the “Add to Suite” button. To remove a summary report from an account’s suite, the
user must click the red “X” icon beside the summary report they would like to remove
from the account’s summary suite. Figure 23 shows the webpage used to manage account
summary suites.
Figure 23: Manage Account Summary Suite
41
5.2.3. Manage User Account Settings
This feature gives NSS and client Administrative users the ability to manage the
summary reports specific users can generate. Under an account, different user types may
have access to different Summary Reports. For example, a Regional Manager called John
Brown should only be able to generate the Category Scores Trend report. To grant access
to a summary, the user may select an account user and the report to grant access to from
the available drop downs and click the “Grant Access” button. To remove access to a
summary report, the user must click the red “X” icon beside the summary report they
would like to remove the user’s access from. Figure 24 shows the webpage used to
manage account users summary access.
Figure 24: Manage User Summary Access
42
5.3. ETL Process Implementation
This process involves the transferring of data from the operational database to the data
warehouse. The transferred data is formatted and loaded into the corresponding fact and
dimension tables. Aggregates are then generated from the loaded data. SQL Server
Integration Services (SSIS) is a service that ships with Microsoft SQL Server 2005 and
later versions. Using SSIS, I created a package that could be scheduled to run periodically
and perform the ETL process. The package consists of ActiveX modules written in
VBScript, Server connection objects, and workflow objects. The ActiveX objects contain
logic that is executed to process and format the data, the Server connections are used to
establish connections between the Operational database and the data warehouse to
transfer data. The workflow objects are used to control the flow of execution in the
package based on success or failure parameters.
43
Chapter 6
TESTING
Testing was done to identify errors and ensure the implementation was done correctly.
Because of the scope of this project, testing was performed on each module or component
in conjunction with implementation. Adjustments and corrections were made to any
errors found before proceeding to the next module.
6.1. Data Warehouse Testing
Data marts in the data warehouse were tested to ensure that data could be effectively
loaded and queried. Test data from a demo account, A+ Company, was loaded into the
data warehouse tables. The test data consisted of data from 7452 evaluation reports for
the report scenario: Best Customer Experience. The evaluations were performed for 265
sites in 2 regions. Question and section score data was inserted into the corresponding
fact tables. Dimension tables such ShopDate, ShopMonth, ClientPeriod, ReportScenarios,
and Reports were also populated. Various test queries were then ran to ensure that joins,
aggregates, and other actions and calculations could performed on the data. Adjustments
such as table redesign, and indexing were performed to improve the performance of the
queries. After aggregate tables were loaded with the aggregated values from the fact
tables, testing was done to compare the performance of selecting records from the
aggregate tables as opposed to running calculations directly on the data in the fact tables.
The results of testing showed that the queries on the aggregate tables ran 90% faster that
queries in the fact tables with the same query results. In one example, the query on the
44
aggregate table aggrRegionMonthReportVersionSectionScore ran in 1 ms compared to a
query on the SectionScores table with identical results that ran in 11 ms. These query
response times were less than the max allowed response time of 2 seconds that the client
emphasized in the requirements.
6.2. Web-based Data Analysis Tool Testing
Various forms of testing were performed on the different parts of the web application as
they were being developed. Ranging from authentication, to logic, and data validation.
Adjustments and corrections were made to any error that were identified before
proceeding to the next module or Webpage.
6.2.1. Authentication testing
I tested to ensure that a user could not access the summary reports without logging in to
the tool. I did this by trying to access the CategoryScores.aspx page. Because I had not
been authenticated, I was redirected to the logout.apx page which displayed “You do not
have an active session. Please log in!”. I then tested being able to access the summary
reports by logging in with a valid account code, user id, and password. Upon submitting
valid login information, I was redirected to the Summaries lobby page which displayed
the links to the summary reports the authenticated user had access to. I was then able to
access and run summary reports including the Category Scores report. To test logging
out, I closed the window with the active login to end my session and then opened a new
45
window and tried to access the CategoryScores.aspx page again. I was once again
redirected to the logout.aspx page and shown an error message instructing me to login.
6.2.2. Summary Reports Display and Logic Testing
I tested each summary report to ensure the labels and corresponding filters were
displayed and populated correctly. I first proofread each summary report to ensure all the
filters were properly labeled and arranged in an orderly manner. I then ensured that filters
that were dynamically populated based on the selection of other filters were functioning
properly. On every summary report most of the filters are dependent on the account and
scenario selected. I changed the values in those filters and crosschecked to ensure that the
dependent filters such as sites, regions, periods, and categories were populated with the
expected data. I then tested the summary reports with different combinations of filter
values and confirmed the correct parameters were passed to the stored procedures called
to generate the reports. Lastly, I tested to ensure that the expected results were displayed
for the different combinations of filter values submitted. For queries with returned
records, I ensured the correct dataset was returned and displayed and the corresponding
chart was displayed correctly. For queries with no returned records, I confirmed that a
“no records found” message was displayed.
6.2.3. Summary Reports Validation
To minimize the validation of syntax and any complex validation, the reports were
designed in an intuitive manner with filters populated with options for the user to select.
46
The only filters that require the user to type-in values are date fields, and these give the
user the option of using a date picker to browse to the dates they wish to select. For all
other validation to ensure that required data is select and entered, I used the
RequiredFieldValidator and the CompareValidator controls. I also added a
ValidationCallout extender to each validator control to display a pop-up message next to
the filter being validated. Restricting users to selecting from options and validating the
filter in the forms help ensure that valid data is submitted to generate the reports. I tested
the form validation by leaving required fields blank and click the “Generate” button to
confirm that validation messages were displayed and the report was not generated. Below
is an example of the use of a CompareValidator control to validate an account dropdown.
<asp:DropDownList ID="ddAccount" runat="server" AutoPostBack="True"
DataSourceID="dsAcctlist" DataTextField="ClientName" DataValueField="ClientId">
<asp:ListItem Value="0">Select Account...</asp:ListItem></asp:DropDownList>
<asp:CompareValidator ID="CompareValidator1" runat="server"
ControlToValidate="ddAccount" ErrorMessage="Account Required!"
ForeColor="#990000" Operator="NotEqual" Type="Integer"
ValueToCompare="0">*</asp:CompareValidator>
<asp:ValidatorCalloutExtender ID="CompareValidator1_ValidatorCalloutExtender"
runat="server" Enabled="True" TargetControlID="CompareValidator1">
</asp:ValidatorCalloutExtender>
Figure 25: Summary Reports Validation
47
6.3. ETL Process Testing
Testing the ETL process involved testing each module/component to make sure it was
executing properly and resulting in the required output for the following module. As the
individual ActiveX modules were created, they were tested individually and then added
to the ETL process workflow and tested in combinations with the modules created before
them. The activity and completion of each module is also logged in the ETLActivityLog
table to log and track the activity and affected records at each step of the ETL process in
case the completed steps need to traced back to identify the root of an error or
inconsistency. The Data connections were also tested to make sure the operational
database and data warehouse could be connected to without any problems.
48
Chapter 7
CONCLUSION
In this project, I have learned a lot about data warehouses and creating summary reports
to analyze the data. Before working on this project, I was under the impression that
normalized relational databases were the only way to store and query data efficiently.
After completing CSC 177, the Data Warehousing and Data Mining class, I learned the
value of a data warehouse and how it makes it easier to store data and generate
summaries and reports. The data warehouse I designed and developed for this project will
make it easier for summaries and reports to be generated from the market research data
that is collected by mystery shoppers with the use of a web-based data analysis tool. I
also learned how powerful aggregate tables are in improving performance of queries. In
addition, from loading the data marts with test data earlier in the project, I learned how
tedious the ETL process can be. SQL Server Integration Services (SSIS) provides
modules and components that make it easier to transform the data from the operational
database to the data warehouse.
Future work on this project will involve creating new reports to add to the suite of
summary reports available to clients. It will also involve adding the ability to generate
and email summary reports through scheduled notifications or upon request through the
web-based data analysis tool.
49
APPENDIX A
Data Warehouse Schema Diagrams
a.) RatingAnswerScores, OccurrenceCountValues, DurationInSecondsValues Data
Mart Schema diagram:
ClientAccounts
PK
ClientId
ShopCode varchar(5)
ClientName varchar(150)
Sections
PK
SectionKey
bigint identity
SectionId
V_ID
SectionTitle
SectionTitleFull
SectionRank
SectionPointsPossible
bigint
bigint
varchar(50)
varchar(150)
int
float
FK8
FK2
FK6
FK5
FK3
FK1
FK4
FK7
FK1
SiteKey
bigint identity
clientId
ShopCode
SiteId
SiteName
SiteRegionId
bigint
varchar(3)
varchar(50)
varchar(150)
bigint
SiteRegions
PK
SiteRegionId
bigint identity
ClientId
RegionName
RegionAbbr
ShopCode
bigint
varchar(150)
varchar(50)
varchar(5)
PK
VersionKey
bigint identity
ClientId
V_ID
ScenarioId
VersionPattern
VersionTitle
StartDate
EndDate
bigint
RatingOptionId
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
RatingPointValue
isSelected
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
tinyint
FK8
FK2
FK6
FK5
FK1
FK4
FK3
FK7
CountValueId
QuestionId
SectionId
SectionKey
ReportId
V_ID
ClientId
SiteKey
ShopDateId
EvaluatorId
countValue
PK
FK1
ShopDateId
bigint identity
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
bigint
smalldatetime
varchar(20)
smallint
varchar(50)
varchar(20)
smallint
bit
bit
bit
bigint
smallint
varchar(50)
varchar(20)
int
bigint
bigint
int
bigint
varchar(20)
varchar(150)
smalldatetime
smalldatetime
bigint identity
ShopMonth
PK
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
bigint identity
MonthNum
MonthName
MonthAbbr
YearNum
smallint
varchar(50)
varchar(10)
int
DurationValueId
bigint
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
durationValue
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
ShopYearId
bigint identity
ShopYearNum int
DurationInSecondsValues
FK2
FK6
FK5
FK3
FK1
FK4
FK7
ShopMonthId
ShopYear
PK
FK8
ReportVersions
PK
RatingScoreId
OccurrenceCountValues
Sites
PK
ShopDate
RatingAnswerScores
bigint identity
Evaluators
PK
EvaluatorId
bigint identity
Gender
Ethnicity
DateOfBirth
ShopperNumber
varchar(20)
varchar(50)
datetime
bigint
Questions
PK
QuestionId
bigint identity
V_ID
SectionId
QuestionLabel
Questiontext
PointsPossible
OptionCount
bigint
bigint
varchar(50)
varchar(100)
float
int
Reports
PK
ReportId
bigint identity
ShopCode
ClientId
V_ID
Rpt_No
varchar(3)
bigint
bigint
bigint
50
b.) PriceValues, TimeValues, and TextValues Schema Diagram:
ClientAccounts
PK
ClientId
ShopDate
PriceValues
bigint identity
ShopCode varchar(5)
ClientName varchar(150)
FK8
FK2
FK6
FK5
FK3
FK1
FK4
FK7
Sections
PK
SectionKey
bigint identity
SectionId
V_ID
SectionTitle
SectionTitleFull
SectionRank
SectionPointsPossible
bigint
bigint
varchar(50)
varchar(150)
int
float
FK1
SiteKey
bigint identity
clientId
ShopCode
SiteId
SiteName
SiteRegionId
bigint
varchar(3)
varchar(50)
varchar(150)
bigint
SiteRegions
PK
SiteRegionId
bigint identity
ClientId
RegionName
RegionAbbr
ShopCode
bigint
varchar(150)
varchar(50)
varchar(5)
PK
ReportVersions
VersionKey
bigint identity
ClientId
V_ID
ScenarioId
VersionPattern
VersionTitle
StartDate
EndDate
bigint
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
PriceValue
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
FK8
FK2
FK6
FK5
FK1
FK4
FK3
FK7
TimeValueId
QuestionId
SectionId
SectionKey
ReportId
V_ID
ClientId
SiteKey
ShopDateId
EvaluatorId
timeValueMilitary
FK1
bigint identity
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
bigint
smalldatetime
varchar(20)
smallint
varchar(50)
varchar(20)
smallint
bit
bit
bit
bigint
smallint
varchar(50)
varchar(20)
int
bigint
bigint
int
bigint
varchar(20)
varchar(150)
smalldatetime
smalldatetime
bigint identity
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
varchar(20)
ShopMonth
PK
FK2
FK6
FK5
FK3
FK1
FK4
FK7
bigint
QuestionId
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
EvaluatorId
textValue
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
varchar(50)
bigint identity
MonthNum
MonthName
MonthAbbr
YearNum
smallint
varchar(50)
varchar(10)
int
ShopYearId
bigint identity
ShopYearNum int
TextAnswerValues
TextValueId
ShopMonthId
ShopYear
PK
FK8
PK
PriceValueId
ShopDateId
TimeValues
Sites
PK
PK
Evaluators
PK
EvaluatorId
bigint identity
Gender
Ethnicity
DateOfBirth
ShopperNumber
varchar(20)
varchar(50)
datetime
bigint
Questions
PK
QuestionId
bigint identity
V_ID
SectionId
QuestionLabel
Questiontext
PointsPossible
OptionCount
bigint
bigint
varchar(50)
varchar(100)
float
int
Reports
PK
ReportId
bigint identity
ShopCode
ClientId
V_ID
Rpt_No
varchar(3)
bigint
bigint
bigint
51
c.) SectionScores, EvaluationScores Schema Diagram:
SectionScores
ClientAccounts
PK
ClientId
bigint identity
ShopCode varchar(5)
ClientName varchar(150)
Sections
PK
SectionKey
bigint identity
SectionId
V_ID
SectionTitle
SectionTitleFull
SectionRank
SectionPointsPossible
bigint
bigint
varchar(50)
varchar(150)
int
float
PK
FK2
FK3
FK5
FK1
FK4
ShopDate
SectionScoreId
bigint identity
SectionId
SectionKey
ReportId
V_ID
ShopDateId
ClientId
SiteKey
ShopperId
SectionPointsEarned
SectionPointsPossible
SectionScoreValue
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
float
float
EvaluationScores
Sites
PK
FK1
SiteKey
bigint identity
clientId
ShopCode
SiteId
SiteName
SiteRegionId
bigint
varchar(3)
varchar(50)
varchar(150)
bigint
SiteRegions
PK
SiteRegionId
bigint identity
ClientId
RegionName
RegionAbbr
ShopCode
bigint
varchar(150)
varchar(50)
varchar(5)
ReportVersions
PK
VersionKey
bigint identity
ClientId
V_ID
ScenarioId
VersionPattern
VersionTitle
StartDate
EndDate
bigint
int
bigint
varchar(20)
varchar(150)
smalldatetime
smalldatetime
Reports
PK
ReportId
bigint identity
ShopCode
ClientId
V_ID
Rpt_No
varchar(3)
bigint
bigint
bigint
PK
EvaluationScoreId
bigint identity
FK6
ReportId
V_ID
ShopDateId
ClientId
SiteKey
ShopperId
SectionKey
VersionKey
EvaluationPointsEarned
EvaluationPointsPossible
EvaluationScoreValue
bigint
bigint
bigint
bigint
bigint
bigint
bigint
bigint
float
float
float
PK
FK1
ShopDateId
bigint identity
ClientId
FullDateAmerican
FullDateEuropean
DayOfWeekNum
DayOfWeekName
DayOfWeekAbbr
DayOfMonth
WeekDayFlag
WeekendFlag
HolidayFlag
ShopMonthId
MonthNum
MonthName
MonthAbbr
YearNum
ClientPeriodId
bigint
smalldatetime
varchar(20)
smallint
varchar(50)
varchar(20)
smallint
bit
bit
bit
bigint
smallint
varchar(50)
varchar(20)
int
bigint
ShopMonth
FK4
FK1
FK3
FK2
FK5
PK
ShopMonthId
bigint identity
MonthNum
MonthName
MonthAbbr
YearNum
smallint
varchar(50)
varchar(10)
int
52
d.) Aggregate tables:
aggrSiteMonthWeekDayReportVersionEvaluationScore
ClientId
SiteKey
ShopMonthId
DayOfWeekNum
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
smallint
decimal(19,0)
int
float
aggrAccountDayOfWeekQuestionDurationSecsValues
aggrRegionYearReportVersionSectionScore
ClientId
SiteRegionId
ShopYearId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
bigint
bigint
bigint
int
float
ClientId
DayOfWeekNum
QuestionId
durationSum
recordCnt
aggrDurationValues
decimal(19,0)
smallint
decimal(19,0)
int
int
float
aggrAccountYearReportVersionSectionScore
aggrSiteMonthWeekDayQuestionOccurrenceCountValues
aggrRegionYearReportVersionEvaluationScore
ClientId
V_ID
SiteKey
ShopMonthId
DayOfWeekNum
QuestionId
countSum
recordCnt
aggrCountValues
bigint
bigint
bigint
bigint
smallint
bigint
int
int
float
ClientId
SiteRegionId
ShopYearId
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
bigint
int
float
ClientId
ShopYearId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
decimal(19,0)
decimal(19,0)
decimal(19,0)
decimal(19,0)
decimal(19,0)
int
float
aggrAccountMonthHourOfDayReportVersionEvaluationScore
aggrSiteMonthHourOfDayQuestionOccurrenceCountValues
ClientId
V_ID
SiteKey
ShopMonthId
HourNum
QuestionId
countSum
recordCnt
aggrCountValues
bigint
bigint
bigint
bigint
smallint
bigint
int
int
float
aggrSiteYearReportVersionSectionScore
ClientId
SiteKey
ShopYearId
SectionId
SectionKey
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
bigint
bigint
bigint
int
float
aggrRegionMonthHourOfDayQuestionOccurrenceCountValues
ClientId
V_ID
SiteRegionId
shopMonthId
HourNum
QuestionId
countSum
recordCnt
aggrCountValues
bigint
bigint
bigint
bigint
smallint
decimal(19,0)
int
int
float
aggrRegionMonthReportVersionEvaluationScore
ClientId
SiteRegionId
ShopMonthId
V_ID
occurCnt
aggrScore
decimal(19,0)
decimal(19,0)
decimal(19,0)
decimal(19,0)
int
float
aggrRegionMonthHourOfDayVersionEvaluationScore
aggrSiteYearReportVersionEvaluationScore
ClientId
SiteKey
ShopYearId
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
bigint
int
float
aggrSiteMonthHourOfDayReportVersionEvaluationScore
ClientId
SiteKey
ShopMonthId
HourNum
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
smallint
bigint
int
float
aggrSiteMonthReportVersionEvaluationScore
ClientId
SiteKey
ShopMonthId
V_ID
occurCnt
aggrScore
bigint
bigint
bigint
bigint
int
float
ClientId
SiteRegionId
ShopMonthId
HourNum
V_ID
occurCnt
aggrScore
decimal(19,0)
decimal(19,0)
decimal(19,0)
smallint
decimal(19,0)
int
float
ClientId
ShopMonthId
HourNum
V_ID
occurCnt
aggrScore
decimal(19,0)
decimal(19,0)
smallint
decimal(19,0)
int
float
aggrAccountMonthWeekDayReportVersionEvaluationScore
ClientId
ShopMonthId
DayOfWeekNum
V_ID
occurCnt
aggrScore
decimal(19,0)
decimal(19,0)
smallint
decimal(19,0)
int
float
53
e.) Additional tables:
ShopTimes
PK
ClientPeriods
ShopTimeId
bigint identity
ClientId
ReportId
ShopDateId
TimeIn
TimeInMilitary
TimeInHourNum
TimeOut
TimeOutMilitary
TimeOutHourNum
bigint
bigint
bigint
varchar(20)
varchar(20)
smallint
varchar(20)
varchar(20)
smallint
PK
ClientPeriodId
bigint identity
ClientId
Period
Year
StartDate
EndDate
ClientQuarterId
bigint
varchar(50)
int
datetime
datetime
bigint
ReportShopTypes
ShopTypeId
ClientId
ShopTypeTitle
ShopTypeAbbr
ShopTypeNo
RatingTypes
ClientQuarters
PK
ClientQuarterId
bigint identity
Quarter
varchar(50)
bigint identity
bigint
varchar(150)
varchar(5)
smallint
PK
RatingTypeId
bigint identity
Title
varchar(150)
ClientAccountUserActivityLog
PK
ClientActivityLogId bigint identity
RatingOptions
ClientUserKey
ActivityDescription
LogTimeStamp
bigint
varchar(50)
smalldatetime
PK
SummationAccountReportXref
PK
PK
RatingOptionId
bigint identity
QuestionId
RatingTypeId
RatingOptionText
RatingOptionNumeric
PointsPossible
bigint
bigint
varchar(200)
int
float
SummaryId
bigint
ClientAccountKey bigint
SummationReportTypes
PK
SummationAccountUserReportXref
SummaryId
bigint
ClientUserKey bigint
SummaryId
bigint identity
SummaryTitle varchar(50)
54
APPENDIX B
Webpages Code
a.) Master Page (summsMain.master)
<%@ Master Language="VB" CodeFile="summsMain.master.vb"
Inherits="mpages_summsMain" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
<title></title>
<link href="~/Styles/default.css" rel="stylesheet" type="text/css" />
<asp:ContentPlaceHolder id="head" runat="server">
</asp:ContentPlaceHolder>
<style type="text/css">
.style1
{
width: 650px;
}
</style>
</head>
<body>
<form id="form1" runat="server">
<asp:ScriptManager ID="ScriptManager1" runat="server">
</asp:ScriptManager>
<div>
<table width="100%" border="0" cellpadding="0" cellspacing="0" align="center">
<tr>
<td height="75" width="200" class="left_top_border" bgcolor="White">
<asp:Image ID="imgAcctLogo" runat="server"
ImageUrl="~/images/color_small.jpg" />
</td>
<td class="right_top_border" bgcolor="White">
<asp:Label ID="lbSummMainTitle" runat="server" Text="Summations
Engine"></asp:Label>
</td>
</tr>
<tr>
<td height="10" colspan="2" class="top_left_bottom_right_border"
bgcolor="#000D47">
<asp:Image ID="imgMidspacer" runat="server" ImageUrl="~/images/1.gif"
Height="10" Width="5" /></td>
</tr>
<tr>
<td valign="top" class="left_border" align="left" bgcolor="White"
rowspan="3">
<br />
55
<asp:Menu ID="Menu1" runat="server" CssClass="menu"
StaticSubMenuIndent="16px">
<Items>
<asp:MenuItem Text="Main Menu" Value="Main Menu"
NavigateUrl="~/summaries/Default.aspx"></asp:MenuItem>
<asp:MenuItem Text="Help" Value="Help"
NavigateUrl="~/summaries/support/Help.aspx"></asp:MenuItem>
</Items>
</asp:Menu>
<asp:Menu ID="Menu2" runat="server" CssClass="menu"
StaticSubMenuIndent="16px">
<Items>
<asp:MenuItem Text="Main Menu" Value="Main Menu"
NavigateUrl="~/summaries/Default.aspx"></asp:MenuItem>
<asp:MenuItem Text="Help" Value="Help"
NavigateUrl="~/summaries/support/Help.aspx"></asp:MenuItem>
<asp:MenuItem NavigateUrl="~/summaries/mgmt/user_summs.aspx"
Text="Mange User Access" Value="Mange User
Access"></asp:MenuItem>
</Items>
</asp:Menu>
<asp:Menu ID="Menu3" runat="server" CssClass="menu"
StaticSubMenuIndent="16px">
<Items>
<asp:MenuItem Text="Main Menu" Value="Main Menu"
NavigateUrl="~/summaries/Default.aspx"></asp:MenuItem>
<asp:MenuItem Text="Help" Value="Help"
NavigateUrl="~/summaries/support/Help.aspx"></asp:MenuItem>
<asp:MenuItem NavigateUrl="~/summaries/mgmt/acct_summs.aspx"
Text="Manage Account Access" Value="Manage Account
Access"></asp:MenuItem>
<asp:MenuItem NavigateUrl="~/summaries/mgmt/user_summs.aspx"
Text="Mange User Access" Value="Mange User
Access"></asp:MenuItem>
</Items>
</asp:Menu>
</td>
<td valign="middle" class="right_border" bgcolor="White" height="25">
<asp:Label ID="lbMasterPageTitle" runat="server" Font-Bold="True"
Font-Names="Verdana" Font-Size="Medium" ForeColor="Black"></asp:Label>
</td>
</tr>
<tr>
<td valign="middle" class="right_border" bgcolor="White" height="25"
align="left">
<asp:Panel ID="lbMasterPageDesc" runat="server" BorderColor="#CCCCCC"
BorderStyle="Solid" BorderWidth="1px" Height="20px" Width="700px">
<table cellpadding="0" class="style1">
<tr>
<td width="25">
<asp:Image ID="imgMasterinfo" runat="server"
ImageUrl="~/images/icn_info.gif" />
</td>
<td>
56
<asp:Label ID="lbMasterPageInfo"
runat="server"></asp:Label>
</td>
</tr>
</table>
</asp:Panel>
</td>
</tr>
<tr>
<td valign="top" height="400" class="right_border" bgcolor="White">
<asp:ContentPlaceHolder id="ContentPlaceHolder1" runat="server">
</asp:ContentPlaceHolder>
</td>
</tr>
<tr>
<td height="20" colspan="2" class="top_left_bottom_right_border"
align="center"
bgcolor="White">
<asp:Label ID="Label1" runat="server"
Text="&amp;copy; National Shopping Service"></asp:Label>
</td>
</tr>
</table>
</div>
</form>
</body>
</html>
57
b.) Category Scores (CategoryScores.aspx)
<%@ Page Title="" Language="VB" MasterPageFile="~/mpages/summsMain.master"
AutoEventWireup="false" CodeFile="CategoryScores.aspx.vb"
Inherits="summaries_reports_CategoryScores" %>
<%@ Register assembly="AjaxControlToolkit" namespace="AjaxControlToolkit"
tagprefix="asp" %>
<%@ Register assembly="System.Web.DataVisualization, Version=4.0.0.0,
Culture=neutral, PublicKeyToken="
namespace="System.Web.UI.DataVisualization.Charting" tagprefix="asp" %>
<asp:Content ID="Content1" ContentPlaceHolderID="head" Runat="Server">
<style type="text/css">
.style1
{
width: 700px;
}
.style2
{
height: 26px;
}
</style>
</asp:Content>
<asp:Content ID="Content2" ContentPlaceHolderID="ContentPlaceHolder1"
Runat="Server">
<asp:UpdatePanel ID="UpdatePanel1" runat="server">
<ContentTemplate>
<table cellpadding="2" cellspacing="0" class="style1">
<tr>
<td width="100">
&nbsp;</td>
<td width="400">
&nbsp;</td>
<td width="200">
&nbsp;</td>
</tr>
<tr>
<td class="bottom_border_gray" width="100">
<asp:Label ID="lbAccount" runat="server"
Text="Account:"></asp:Label>
</td>
<td class="bottom_border_gray" width="400">
<asp:DropDownList ID="ddAccount" runat="server"
AutoPostBack="True"
DataSourceID="dsAcctlist" DataTextField="ClientName"
DataValueField="ClientId">
<asp:ListItem Value="0">Select Account...</asp:ListItem>
</asp:DropDownList>
<asp:SqlDataSource ID="dsAcctlist" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
58
SelectCommand="GetShopCodeList"
SelectCommandType="StoredProcedure">
</asp:SqlDataSource>
<asp:CompareValidator ID="CompareValidator1" runat="server"
ControlToValidate="ddAccount" ErrorMessage="Account
Required!"
ForeColor="#990000" Operator="NotEqual" Type="Integer"
ValueToCompare="0">*</asp:CompareValidator>
<asp:ValidatorCalloutExtender
ID="CompareValidator1_ValidatorCalloutExtender"
runat="server" Enabled="True"
TargetControlID="CompareValidator1">
</asp:ValidatorCalloutExtender>
<asp:Label ID="lbShopCode" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbAccountId" runat="server"
Visible="False">0</asp:Label>
<asp:Label ID="lbUserId" runat="server" Text="M0"
Visible="False"></asp:Label>
</td>
<td class="bottom_border_gray">
&nbsp;</td>
</tr>
<tr>
<td class="bottom_border_gray">
<asp:Label ID="lbScenario" runat="server"
Text="Scenario:"></asp:Label>
</td>
<td class="bottom_border_gray">
<asp:DropDownList ID="ddScenario" runat="server"
DataSourceID="dsScenario"
DataTextField="ScenarioTitle" DataValueField="ScenarioId"
AutoPostBack="True">
<asp:ListItem Value="0">Select Scenario...</asp:ListItem>
</asp:DropDownList>
<asp:SqlDataSource ID="dsScenario" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetShopScenarioListByClientId"
SelectCommandType="StoredProcedure">
<SelectParameters>
<asp:ControlParameter ControlID="lbAccountId"
DefaultValue="0"
Name="intClientId" PropertyName="Text"
Type="Int64" />
</SelectParameters>
</asp:SqlDataSource>
<asp:CompareValidator ID="CompareValidator2" runat="server"
ControlToValidate="ddScenario" ErrorMessage="Scenario
Required!"
ForeColor="#990000" Operator="NotEqual"
ValueToCompare="0">*</asp:CompareValidator>
<asp:ValidatorCalloutExtender
ID="CompareValidator2_ValidatorCalloutExtender"
59
runat="server" Enabled="True"
TargetControlID="CompareValidator2">
</asp:ValidatorCalloutExtender>
<asp:Label ID="lbScenarioId" runat="server"
Visible="False"></asp:Label>
</td>
<td class="bottom_border_gray">
&nbsp;</td>
</tr>
<tr>
<td class="style2">
<asp:Label ID="lbGrouping" runat="server"
Text="Grouping:"></asp:Label>
</td>
<td class="style2">
<asp:DropDownList ID="ddGrouping" runat="server"
AutoPostBack="True">
<asp:ListItem Value="0">Select Grouping...</asp:ListItem>
<asp:ListItem Value="SITE">Sites</asp:ListItem>
<asp:ListItem Value="REGION">Regions</asp:ListItem>
<asp:ListItem Value="ALL">All (Entire
Company)</asp:ListItem>
</asp:DropDownList>
<asp:CompareValidator ID="CompareValidator3" runat="server"
ControlToValidate="ddGrouping" ErrorMessage="Grouping
Required!"
ForeColor="#990000" Operator="NotEqual"
ValueToCompare="0">*</asp:CompareValidator>
<asp:ValidatorCalloutExtender
ID="CompareValidator3_ValidatorCalloutExtender"
runat="server" Enabled="True"
TargetControlID="CompareValidator3">
</asp:ValidatorCalloutExtender>
</td>
<td class="style2">
&nbsp;</td>
</tr>
<tr>
<td class="bottom_border_gray">
&nbsp;</td>
<td class="bottom_border_gray">
<asp:ListBox ID="lbxSitesList" runat="server"
DataSourceID="dsGetSites"
DataTextField="SiteId" DataValueField="SiteKey"
Visible="False" Width="100px"></asp:ListBox>
<asp:SqlDataSource ID="dsGetSites" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetShopSitesListByClientIdAndUserId"
SelectCommandType="StoredProcedure">
<SelectParameters>
<asp:ControlParameter ControlID="lbAccountId"
Name="intClientId"
PropertyName="Text" Type="Int64" />
60
<asp:ControlParameter ControlID="lbUserId"
Name="strUserId" PropertyName="Text"
Type="String" />
</SelectParameters>
</asp:SqlDataSource>
<asp:ListBox ID="lbxRegionList" runat="server"
DataSourceID="dsRegions"
DataTextField="RegionName" DataValueField="SiteRegionId"
Visible="False"
Width="100px"></asp:ListBox>
<asp:SqlDataSource ID="dsRegions" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetShopRegionListByClientIdAndUserId"
SelectCommandType="StoredProcedure">
<SelectParameters>
<asp:ControlParameter ControlID="lbAccountId"
Name="intClientId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="lbUserId"
Name="strUserId" PropertyName="Text"
Type="String" />
</SelectParameters>
</asp:SqlDataSource>
</td>
<td class="bottom_border_gray">
<asp:RequiredFieldValidator ID="vdSitesList" runat="server"
ControlToValidate="lbxSitesList" ErrorMessage="Sites must
be selected!"
ForeColor="#990000">*</asp:RequiredFieldValidator>
<asp:ValidatorCalloutExtender
ID="vdSitesList_ValidatorCalloutExtender"
runat="server" Enabled="True"
TargetControlID="vdSitesList">
</asp:ValidatorCalloutExtender>
<asp:RequiredFieldValidator ID="vdRegionsList" runat="server"
ControlToValidate="lbxRegionList"
ErrorMessage="1 or more Regions must be selected!"
ForeColor="#990000">*</asp:RequiredFieldValidator>
</td>
</tr>
<tr>
<td>
<asp:Label ID="lbDate" runat="server"
Text="Date:"></asp:Label>
</td>
<td align="left">
<asp:Label ID="lbPeriodType" runat="server"
Text="Period:"></asp:Label>
</td>
<td align="left">
&nbsp;</td>
</tr>
<tr>
<td>
61
&nbsp;</td>
<td align="left" colspan="2">
<asp:DropDownList ID="ddPeriodType" runat="server"
AutoPostBack="True">
<asp:ListItem Selected="True"
Value="MONTH">Month</asp:ListItem>
<asp:ListItem Value="YEAR">Year</asp:ListItem>
<asp:ListItem Value="PERIOD">Client Period</asp:ListItem>
</asp:DropDownList>
&nbsp;
<asp:Image ID="imgPeriodPointer" runat="server"
ImageAlign="AbsMiddle"
ImageUrl="~/images/icn_arrow_blue.gif" />
&nbsp;
<asp:DropDownList ID="ddPeriod" runat="server"
DataSourceID="dsGetPeriod"
DataTextField="PeriodTitle" DataValueField="PeriodTypeId">
</asp:DropDownList>
<asp:SqlDataSource ID="dsGetPeriod" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetShopPeriodsByClientIdAndPeriodType"
SelectCommandType="StoredProcedure">
<SelectParameters>
<asp:ControlParameter ControlID="lbAccountId"
Name="intClientId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="ddPeriodType"
DefaultValue="MONTH"
Name="strPeriodType" PropertyName="SelectedValue"
Type="String" />
</SelectParameters>
</asp:SqlDataSource>
</td>
</tr>
<tr>
<td>
&nbsp;</td>
<td align="left" colspan="2">
<asp:Label ID="Label2" runat="server" Font-Bold="True"
ForeColor="Black"
Text="OR"></asp:Label>
</td>
</tr>
<tr>
<td>
&nbsp;</td>
<td align="left" colspan="2">
<asp:CheckBox ID="cbxUseDates" runat="server" Text="Use Date
Range:" />
</td>
</tr>
<tr>
<td>
&nbsp;</td>
62
<td align="left" colspan="2">
<asp:TextBox ID="txbStartDate0" runat="server" MaxLength="10"
Width="70px"></asp:TextBox>
<asp:CalendarExtender ID="txbStartDate0_CalendarExtender"
runat="server"
TargetControlID="txbStartDate0">
</asp:CalendarExtender>
&nbsp;
<asp:Label ID="lbDateTo0" runat="server"
Text="To"></asp:Label>
&nbsp;
<asp:TextBox ID="txbEndDate0" runat="server" MaxLength="10"
Width="70px"></asp:TextBox>
<asp:CalendarExtender ID="txbEndDate0_CalendarExtender"
runat="server"
Enabled="True" TargetControlID="txbEndDate0">
</asp:CalendarExtender>
</td>
</tr>
<tr>
<td class="bottom_border_gray" colspan="3" height="10">
<asp:Image ID="Image1" runat="server" Height="10px"
ImageUrl="~/images/1.gif"
Width="5px" />
</td>
</tr>
<tr>
<td class="bottom_border_gray" valign="top">
<asp:Label ID="lbCategory" runat="server"
Text="Category:"></asp:Label>
</td>
<td class="bottom_border_gray" colspan="2">
<asp:ListBox ID="lbxCategoryList" runat="server"
DataSourceID="dsGetSections"
DataTextField="SectionTitle" DataValueField="SectionKey"
SelectionMode="Multiple" Width="200px">
<asp:ListItem>Select Categories...</asp:ListItem>
</asp:ListBox>
<asp:RequiredFieldValidator ID="vdCategory" runat="server"
ControlToValidate="lbxCategoryList" ErrorMessage="Category
required!"
ForeColor="#990000">*</asp:RequiredFieldValidator>
<asp:ValidatorCalloutExtender
ID="vdCategory_ValidatorCalloutExtender"
runat="server" Enabled="True"
TargetControlID="vdCategory">
</asp:ValidatorCalloutExtender>
<asp:SqlDataSource ID="dsGetSections" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetShopSectionsByScenarioIdAndUserId"
SelectCommandType="StoredProcedure">
<SelectParameters>
63
<asp:ControlParameter ControlID="lbScenarioId"
Name="intScenarioId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="lbUserId"
Name="strUserId" PropertyName="Text"
Type="String" />
</SelectParameters>
</asp:SqlDataSource>
</td>
</tr>
<tr>
<td>
<asp:Label ID="lbExport" runat="server" Text="Export
option:"></asp:Label>
</td>
<td colspan="2" align="left">
<asp:RadioButtonList ID="RadioButtonList1" runat="server"
RepeatDirection="Horizontal">
<asp:ListItem Selected="True"
Value="HTML">Html</asp:ListItem>
<asp:ListItem Value="PDF"></asp:ListItem>
<asp:ListItem Value="EXCEL">Excel</asp:ListItem>
</asp:RadioButtonList>
</td>
</tr>
<tr>
<td>
&nbsp;</td>
<td colspan="2">
<asp:Button ID="btnGenerate" runat="server" Text="Generate" />
<asp:ValidationSummary ID="ValidationSummary1" runat="server"
ForeColor="#990000" HeaderText="Please correct the
following:" />
</td>
</tr>
<tr>
<td>
&nbsp;</td>
<td colspan="2">
<asp:Label ID="lbSiteIdList" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbRegionIdList" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbsectionIdList" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbGroupingVal" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbPeriodTypeVal" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lnPeriodIdVal" runat="server"
Visible="False"></asp:Label>
<asp:Label ID="lbDateRange" runat="server" Text="0"
Visible="False"></asp:Label>
<asp:Label ID="lbStartDate" runat="server"
Visible="False"></asp:Label>
64
<asp:Label ID="lbEndDate" runat="server"
Visible="False"></asp:Label>
</td>
</tr>
</table>
<br />
<asp:Label ID="lbTest" runat="server" Visible="False"></asp:Label>
<br />
<asp:SqlDataSource ID="dsGetSummary" runat="server"
ConnectionString="<%$
ConnectionStrings:NssDataWarehouseConnectionString %>"
SelectCommand="GetAggrRegionOrSiteMonthReportScenarionSectionScore"
SelectCommandType="StoredProcedure">
<SelectParameters>
<asp:ControlParameter ControlID="lbAccountId" Name="intClientId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="lbScenarioId"
Name="intScenarioId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="lbGroupingVal"
Name="strGroupingType"
PropertyName="Text" Type="String" />
<asp:ControlParameter ControlID="lbRegionIdList"
Name="strRegionIds"
PropertyName="Text" Type="String" />
<asp:ControlParameter ControlID="lbSiteIdList" Name="strSiteKeys"
PropertyName="Text" Type="String" />
<asp:ControlParameter ControlID="lbPeriodTypeVal"
Name="strPeriodType"
PropertyName="Text" Type="String" />
<asp:ControlParameter ControlID="lnPeriodIdVal"
Name="intPeriodTypeId"
PropertyName="Text" Type="Int64" />
<asp:ControlParameter ControlID="lbsectionIdList"
Name="strSectionIds"
PropertyName="Text" Type="String" />
<asp:ControlParameter ControlID="lbDateRange"
Name="intUseDateRange"
PropertyName="Text" Type="Int16" />
<asp:ControlParameter ControlID="lbStartDate" Name="datStartDate"
PropertyName="Text" Type="DateTime" />
<asp:ControlParameter ControlID="lbEndDate" Name="datEndDate"
PropertyName="Text" Type="DateTime" />
</SelectParameters>
</asp:SqlDataSource>
<asp:Label ID="lbSummaryTitle" runat="server" Font-Bold="True"
Text="Category Scores" Visible="False"></asp:Label>
<br />
<br />
<asp:Chart ID="chrtSummary" runat="server" BorderlineColor="Black"
BorderlineDashStyle="Solid" DataSourceID="dsGetSummary"
Palette="None" Visible="False" Width="800px"
PaletteCustomColors="255, 255, 128; Green; Blue; Purple; Red; Silver;
White; Black; 0, 64, 64"
65
Height="500px">
<Series>
<asp:Series Name="Series1" XValueMember="SectionTitle"
YValueMembers="AvergareScore" BorderColor="Black">
</asp:Series>
</Series>
<ChartAreas>
<asp:ChartArea Name="ChartArea1">
<AxisY Title="Score (%)">
</AxisY>
<AxisX Title="Categories">
</AxisX>
<Area3DStyle Enable3D="True" LightStyle="Realistic"
PointDepth="20"
PointGapDepth="20" Rotation="20" WallWidth="2" />
</asp:ChartArea>
</ChartAreas>
<BorderSkin BorderDashStyle="Solid" />
</asp:Chart>
<br />
<br />
<asp:GridView ID="gdvSummary" runat="server" BackColor="White"
BorderColor="#999999" BorderStyle="Solid" BorderWidth="1px"
CellPadding="3"
DataSourceID="dsGetSummary" EmptyDataText="records found.."
ForeColor="Black"
GridLines="Vertical" Visible="False">
<AlternatingRowStyle BackColor="#CCCCCC" />
<FooterStyle BackColor="#CCCCCC" />
<HeaderStyle BackColor="Black" Font-Bold="True" ForeColor="White" />
<PagerStyle BackColor="#999999" ForeColor="Black"
HorizontalAlign="Center" />
<SelectedRowStyle BackColor="#000099" Font-Bold="True"
ForeColor="White" />
<SortedAscendingCellStyle BackColor="#F1F1F1" />
<SortedAscendingHeaderStyle BackColor="#808080" />
<SortedDescendingCellStyle BackColor="#CAC9C9" />
<SortedDescendingHeaderStyle BackColor="#383838" />
</asp:GridView>
<br />
</ContentTemplate>
</asp:UpdatePanel>
</asp:Content>
66
c.) Category Scores – code behind page (CategoryScores.aspx.vb)
Partial Class summaries_reports_CategoryScores
Inherits Secured
Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs)
Handles Me.Load
Call SetPageTitle("Category Scores", "", "Generate Category Scores
summaries")
End Sub
Protected Sub ddAccount_SelectedIndexChanged(ByVal sender As Object, ByVal e
As System.EventArgs) Handles ddAccount.SelectedIndexChanged
Me.lbAccountId.Text = Me.ddAccount.SelectedValue
Me.ddScenario.DataBind()
Me.gdvSummary.Visible = False
Me.chrtSummary.Visible = False
End Sub
Protected Sub ddScenario_SelectedIndexChanged(ByVal sender As Object, ByVal e
As System.EventArgs) Handles ddScenario.SelectedIndexChanged
Me.lbScenarioId.Text = Me.ddScenario.SelectedValue
Me.gdvSummary.Visible = False
Me.chrtSummary.Visible = False
End Sub
Protected Sub ddGrouping_SelectedIndexChanged(ByVal sender As Object, ByVal e
As System.EventArgs) Handles ddGrouping.SelectedIndexChanged
Dim strGrouping As String = Me.ddGrouping.SelectedValue
If (strGrouping = "SITE") Then
Me.lbxSitesList.DataBind()
Me.lbxSitesList.Visible = True
Me.lbxRegionList.Visible = False
Me.vdRegionsList.Visible = False
Me.vdSitesList.Visible = True
ElseIf (strGrouping = "REGION") Then
Me.lbxRegionList.DataBind()
Me.lbxSitesList.Visible = False
Me.lbxRegionList.Visible = True
Me.vdRegionsList.Visible = True
Me.vdSitesList.Visible = False
Else
Me.lbxSitesList.Visible = False
Me.lbxRegionList.Visible = False
Me.vdRegionsList.Visible = False
Me.vdSitesList.Visible = False
End If
Me.gdvSummary.Visible = False
Me.chrtSummary.Visible = False
67
End Sub
Protected Sub btnGenerate_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles btnGenerate.Click
Dim strGrouping As String = Me.ddGrouping.SelectedValue
Dim strSitesList As String = ""
Dim strRegionsList As String = ""
Dim strPeriodType As String = Me.ddPeriodType.SelectedValue
Dim strPeriodId As String = Me.ddPeriod.SelectedValue
Dim strStartDate As String = Me.txbStartDate0.Text
Dim strEndDate As String = Me.txbEndDate0.Text
Dim intUseDateRange As Boolean = Me.cbxUseDates.Checked
Dim strCategoryList As String = ""
Dim bolUseDateRange As Boolean = Me.cbxUseDates.Checked
If (strGrouping = "SITE") Then
'Build Sites list
For Each li As ListItem In Me.lbxSitesList.Items
If li.Selected = True Then
If strSitesList = "" Then
strSitesList = li.Value
Else
strSitesList = strSitesList & "," & li.Value
End If
End If
Next
Else
'Build Region list
For Each li As ListItem In Me.lbxRegionList.Items
If li.Selected = True Then
If strRegionsList = "" Then
strRegionsList = li.Value
Else
strRegionsList = strRegionsList & "," & li.Value
End If
End If
Next
End If
'Build Category list
For Each li As ListItem In Me.lbxCategoryList.Items
If li.Selected = True Then
If strCategoryList = "" Then
strCategoryList = li.Value
Else
strCategoryList = strCategoryList & "," & li.Value
End If
End If
Next
68
Me.lbGroupingVal.Text = Me.ddGrouping.SelectedValue
Me.lbPeriodTypeVal.Text = Me.ddPeriodType.SelectedValue
Me.lnPeriodIdVal.Text = Me.ddPeriod.SelectedValue
If bolUseDateRange = True Then
Me.lbDateRange.Text = 1
Me.lbStartDate.Text = strStartDate
Me.lbEndDate.Text = strEndDate
Else
Me.lbDateRange.Text = 0
Me.lbStartDate.Text = "1/1/1900"
Me.lbEndDate.Text = "1/1/1900"
End If
If strSitesList = "" Then
Me.lbSiteIdList.Text = "0"
Else
Me.lbSiteIdList.Text = strSitesList
End If
'Me.lbSiteIdList.Visible = True
If strRegionsList = "" Then
Me.lbRegionIdList.Text = "0"
Else
Me.lbRegionIdList.Text = strRegionsList
End If
'Me.lbRegionIdList.Visible = True
Me.lbsectionIdList.Text = strCategoryList
'Me.lbsectionIdList.Visible = True
Me.gdvSummary.DataBind()
Me.gdvSummary.Visible = True
Me.chrtSummary.DataBind()
'Me.chrtSummary.Visible = True
End Sub
Protected Sub ddPeriodType_SelectedIndexChanged(ByVal sender As Object, ByVal
e As System.EventArgs) Handles ddPeriodType.SelectedIndexChanged
Me.ddPeriod.DataBind()
Me.gdvSummary.Visible = False
Me.chrtSummary.Visible = False
End Sub
Protected Sub dsGetSummary_Selected(ByVal sender As Object, ByVal e As
System.Web.UI.WebControls.SqlDataSourceStatusEventArgs) Handles
dsGetSummary.Selected
Dim intAssignCount As Integer
intAssignCount = e.AffectedRows
'Me.lbTest.Text = "-" & intAssignCount.ToString
69
Me.lbTest.Visible = True
If CInt(intAssignCount) = 0 Then
Me.lbSummaryTitle.Visible = False
Me.chrtSummary.Visible = False
Else
Me.lbSummaryTitle.Visible = True
Me.chrtSummary.Visible = True
End If
End Sub
End Class
70
BIBLIOGRAPHY
[1] Jiawei Han, Micheline Kamber, and Jian Pei, Data Mining: Concepts and
Techniques. 3nd ed., Morgan Kaufmann 2012.
[2] “A Data Warehousing Courseware.” Manashree Kulkarni. 2009.
http://gaia.ecs.csus.edu/~enroll/enrollDW/Intro.php
[3] “An ETL Courseware.” California State University, Sacramento.
http://gaia.ecs.csus.edu/~web_etl/etl/
[4] “ASP.NET” Wikipedia. 2012. http://en.wikipedia.org/wiki/ASP.NET
[5] “Non-functional Requirement” Wikipedia. 2012. http://en.wikipedia.org/wiki/Nonfunctional_requirement