Data Warehouse User
Engagement Readout
Daniel Gold, Project Manager
© 2012 Boise State University
1
Agenda:
© 2013 Boise State University
•
•
•
•
•
•
•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
2
Executive Summary
Goal of User Engagement
• Understand the key features and functions that users need from our ProClarity
replacement
• Understand barriers that exist today that create complexity when obtaining or using
data
• Understand what training and support will be needed as we roll out our ProClarity
replacement
Outcome
• Based on features requested by users, Pyramid is a suitable replacement for ProClarity
• Key reporting areas were identified that if improved would return immediate value to
end users
• Hands-on training based on knowledge level (i.e. Novice, Power User, Expert), robust
self-help resources, BIRS point of contact and peer groups will be needed to support
new tool adoption
• Process changes are needed to streamline report intake and prioritization
• Process changes are needed to build trust in the data warehouse and improve usability
© 2013 Boise State University
3
Methodology
• Two user engagement sessions were
held
– 12 participants on 7/15
– 18 participants on 7/18
• Participants were broken into small
groups and led through a series of
activities including:
1.
2.
3.
4.
Brainstorm and categorization of data
needs
Completing a “reporting story” form
Extracting tool features and process
enhancements from their stories
Prioritization of features and process
changes
© 2013 Boise State University
30
• Participants
50
• Reporting stories
collected
43
• Unique tool feature
needs identified
12
• Unique process
changes requested
4
Participant Population
Areas Represented:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
College of Arts and Sciences
Institutional Research
College of Engineering
eCampus
Extended Studies
College of Health
Registrar
College of Health Sciences
Provost
College of Education
College of Social Sciences and Public Affairs
Admissions
Honors College
University Financial Services
BSU Foundation
STEM
OIT
HR and Finance under-represented! Further
investigation to needs of these teams required.
© 2013 Boise State University
Roles Represented:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
5 directors
3 associate directors
3 business managers
2 administrative assistants
2 data analysts
1 adjunct faculty coordinator
1 advisor
1 associate registrar
1 business operations manager
1 chair
1 coordinator
1 financial technician
1 management assistant
1 management systems coordinator
1 program information coordinator
1 project coordinator
1 project manager
1 senior business manager
1 vice provost
5
Agenda:
© 2013 Boise State University


•
•
•
•
•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
6
Requested Tool Features
Top 10 Prioritized Features:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
© 2013 Boise State University
Drill to Detail (that I need)
Snapshots (compare points in time)
Dashboards
Build in Calculations
Easy export to Excel and PDF
Trend over time
Ability to combine multiple reports
Sort/filter data in web tool
Data “Table of Contents” and improved logical structure
Ability to add charts/graphs in web tool
Support for custom reports (MyViews)
Distribute reports to email audience from tool
7
Pyramid Can Address Our Tool Needs
Requested Feature
Available in Pyramid Suite
Drill to detail (that I need)
Support for Snapshots
Demo links
Basic Analytics ; Cascading
Slicers ; Custom Parameters
BIRS Process Controlled
Dashboards
Building Dashboards
Trending over time
Basic Analytics
Combine multiple reports
Data Mash up with Power Pivot
Built in calculations
Built in and Custom Calculations
Data “Table of Contents”
BIRS Process Controlled
Sort/Filter data
Basic Analytics
Ability to add charts/graphs
Multi/Combo Charts
Distribute reports via email
Easy export to excel/PDF
Support for custom views
© 2013 Boise State University
8
Agenda:
© 2013 Boise State University



•
•
•
•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
9
Report Frequent Flyers
• 4 report categories made up
nearly 2/3rds of all reporting
stories written by participants.
• Short Term: Understanding the
usage of reports in these
categories and improving them
could have a significant impact to
user satisfaction and eliminate
needless effort
The “Core Four”
© 2013 Boise State University
• Long Term: Consider prioritizing
the creation of dynamic
dashboards in Pyramid for these
common reporting tasks
10
Recommendations:
• Recognize the “core four” reporting needs and prioritize
enhancements for these areas
• Form tiger teams for each area and investigate reporting
enhancements
• Identify standard reporting tasks for the “core four” that will
be good candidates for dashboards when Pyramid is released
What’s a Tiger Team?
“ A team of undomesticated and uninhibited technical specialists, selected for their experience, energy,
and imagination, and assigned to track down relentlessly every possible source of failure”
- Program Management in Design and Development
© 2013 Boise State University
11
Common Data Manipulation Tasks
Pain Points:
User comment:
•
All this would be better if… “we were able to gather point in time comparison
and make the comparisons in a more efficient manner. Right now this takes 2
people approximate 6-8 hours every week to complete.”
•
•
•
•
Lack of complete data (HR,
Finance, shadow systems) in data
warehouse leads to users having
to combine DW reports with
queries from other sources
Comparing data for particular
points in time (aka snapshots)
requires running multiple reports
Drill to detail often provides
significantly more data than is
needed and excess rows/columns
must be deleted
Lack of built in calculations and
sort/filter options in web tool
makes exporting data to excel
mandatory
Lack of native graph building
capabilities makes exporting data
to excel mandatory
© 2013 Boise State University
12
Recommendations:
• Utilize tiger teams to understand why reports frequently
need to be combined.
• Make snapshot data available to users where it exists
• Review report frequent flyers to determine how simple
changes in formatting can reduce manual manipulation
© 2013 Boise State University
13
Use of Production Queries
• 52% of participants
rely on queries against
the production
PeopleSoft system
– Expected for data
that is not in the
warehouse (HR,
Finance, and
Financial Aid)
– Expected for data
that needs to be realtime
Potential Reasons for production queries:
1. Historical point in time comparisons (snapshots)
2. Validate data obtained through MyInsights or
ProClarity
3. Familiarity with querying production over using DW
tools
• 35% of participants
rely on queries for
student data that is in
the warehouse
– Unexpected since this
data is available
© 2013 Boise State University
14
Recommendations:
• Utilize tiger teams to understand why production queries are
needed to support the “core four”
– Is data missing in the warehouse?
– What kind of snapshots are needed for comparison?
– Are we missing key data elements like type/rank of faculty?
• Ensure that Pyramid is marketed well and encourage users to
begin using the warehouse to meet reporting needs
© 2013 Boise State University
15
Agenda:
© 2013 Boise State University




•
•
•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
16
Data Integrity
During group discussions two
types of data integrity /
validation issues emerged:
1.
2.
Trust in the accuracy of the data
in the warehouse
(checking for warehouse errors)
Trust that manual manipulation
performed on data had not
corrupted the report
(checking for user errors)
1 in 4 participants cited data
validation as a time consuming
and required step.
© 2013 Boise State University
User Comments:
“I think the most complex part of the reporting is
validating the accuracy of the data. So much of
what we do is this it takes the most time and is
the most frustrating.”
“[M]anual manipulation in excel is time
consuming and there is risk of error. Data
validation.”
All of this would be better if… “[It] didn't take
extensive manipulation each time. Confidence
that the data was correct.”
“[N]umerous proclarity reports are referenced in
order to create the summer report. Difficult to
identify the criteria used to create the ProClarity
report and difficult to validate the data. Time
consuming to create report.”
17
Recommendations
• Look for ways to limit the need to combine multiple reports
in excel through enhancements to current reports
• Build trust in data warehouse reports by consistently
validating data with data stewards and stating explicitly on
reports when validation took place in each report
© 2013 Boise State University
18
Data Definitions
• Users reported difficulty
interpreting the data due to
irregular naming conventions and
inconsistent, unavailable, or out of
date definitions.
• Knowing which fields should be
used in a report was often difficult
when data from multiple sources
had to be combined.
User Comment:
User Comment:
All this would be better if… “You could get a
clear definition for what "active" means (e.g.
not discontinued on PS, or what?!)”
“have you seen the terminology? there are
five categories for each line. Things like
STEM secondary non-STEM major”
© 2013 Boise State University
19
Recommendations
• Determine what information needs to be stored in each data
definition to aid users in making decisions when building
reports
• Ensure that updating data definitions is built into the
development teams “definition of done” any time a report is
altered or created
• Improve data definition discoverability by creating links or
embedding definitions in reports
© 2013 Boise State University
20
Data Access and Integration
Access:
•
•
Access to data was not a wide spread issue, and most users who spoke about
access were concerned with limiting it (security).
A small number of users stated that they needed access to HR data to perform
their job functions
Integration:
•
•
Integrating HR, Finance and Student data into the data warehouse was a frequent
request
Many non-BIRS supported reporting systems are used on campus. Frequently
these are used for storing snapshot data for comparisons.
© 2013 Boise State University
21
Recommendations
• Leverage the Identity and Access Management Project to help
define role based data access authorization
• Continue roadmap work to bring in HR and Finance data into
the warehouse
• Investigate commonly used reporting systems and determine
if it would be reasonable to bring the data into the warehouse
© 2013 Boise State University
22
Process Enhancements
Top 10 Prioritized Process Enhancements
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
© 2013 Boise State University
Data validation
Data integration
Data definitions
Data access
Clear report structure
Standardize reports for wider usage
Make the data current (where is a student now)
Migrate custom views to new tool
Improve report request workflow
Map tools to user needs
23
Recommendations
• Continue with roadmap work to bring HR and Financial data into the
warehouse
• Establish a holistic “definition of done” for reports that includes data
validation and data definitions for every report
• Utilize Identity and Access Management project to help define roles for data
access
• Leverage tiger teams to standardize reports for wider use
• Develop a clear and logical hierarchy for reports
• Revise and evangelize report request intake process with a simple workflow
that that takes into consideration requirements gathering and prioritization
• Begin working to establish a “data doctor” program that includes support
from subject matter experts as well as BIRS
• Begin developing a rollout strategy for Pyramid that will help map training
and tool sets to groups of users based on their reporting needs
© 2013 Boise State University
24
Agenda:
© 2013 Boise State University





•
•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
25
Success Requires…
• Hands on training for users
– Levels of training (novice, power user, expert)
• Robust documentation and tutorials
• An accessible reporting expert
– Data Dr. from BIRS
– Peer groups
•
•
•
•
Migrate existing ProClarity reports into new tool
Clearly define what tool should be used for what task
Early adopter program for new tool
More report building resources on the BIRS team
© 2013 Boise State University
26
Agenda:
© 2013 Boise State University






•
Executive Summary
Study Methodology and Participants
Tool Needs and How They Align to Pyramid
Common Reporting Tasks
Process Opportunities
What is Needed for Success
Summary and Next Steps
27
Tool Implementation
User Rollout Track
Develop Self Help
Tools
•Early adopter
program for IR and
key users
•BIRS developer
training
•FAQ and how-to
documents
•Tutorials
•Tool documentation
Create Experts
Early adopter
rollout
Migrate ProClarity
Reports
© 2013 Boise State University
•Data Dr.
•Peer groups
•Ongoing training
opportunities
Train Users
Development Track
Install Pyramid
•Classroom based
training
•Training based on
data tasks and
complexity
Provide Ongoing
Support
Develop Report
Hierarchy
End User
Rollout
Develop Core Reports
and Dashboards
28
Recommended Next Steps
• Stay the Course for Roadmap Activities
– Continue to move forward with Pyramid purchase and implementation
– Continue to integrate HR and Finance data into the warehouse
• Pause Report Development and Modernize Process
– Define and evangelize project intake process
– Refine the team “definition of done”
– Respond to user engagement requests by generating specific
enhancement projects for “core four” report issues identified in
sessions
• Identify Goals and Participants for Tiger Teams
– Develop specific and measurable goals for tiger teams
– Identify subject matter experts, power users, and developers to
participate
– Establish a timeline for tiger teams to provide actionable
recommendations (likely post-roadmap)
© 2013 Boise State University
29
Daniel Gold, Project Manager
Email daniellgold@boisestate.edu
THANK YOU
© 2013 Boise State University
30
BACKUP
© 2013 Boise State University
31
Breakdown of the Core Four
Enrollment
• Out of State Students –
Currently Enrolled
• Class Enrollment Report
• Fall Enrollment by Plan
• Enrollment Activity Report
• Student Enrollment
• Enrollment Profile
• Credit and Enrollment for
Extended Studies
• Summer Enrollment and
Credit Data
Student Success
• Subsequent Course Success
• Graduation Rate and
Retention
• # of Graduates
• Count of Graduates
• Cohort Graduation
• Cohort Retention and
Graduation
• Student Progress through
Curriculum
• Retention and Graduation
Success Rates
• Academic Progress Report
• Honors Students Graduated
from Honors College
Admissions
• Funnel report
• Admissions comparative
data over time
• Admissions reporting
• Daily snapshot of
applications and
registration
• Student registrations
Course/Class Management
• Class management
• Time utilization
• Class availability and
utilization
• Course statistics for BAS
students
• Course schedule
The above items represent the report or report goal that participants identified in their
reporting story forms. Duplicates intentionally included.
© 2013 Boise State University
32
Tiger Teams
Tiger teams work best when they are small
hand-picked groups focused on a specific
task (e.g. simplifying enrollment reporting).
The team should include not only subject
matter experts and power users, but also
report and back-end database developers to
speed up the process of weighing out
potential solutions. These “tigers” are easily
identified by being the go-to people in their
area when a problem arises.
The activity should be time-boxed and the
output should be a list of enhancements the
team can prioritize and implement with the
goal of delivering on the specific task
assigned.
© 2013 Boise State University
Student
Records
SME
2-3 key
report
users
Simplify
Enrollment
Reporting
Report
Developer
ETL
Developer
33