THE QUANTITATIVE/
QUALITATIVE IMPACT
STUDY
WHAT IS AN IMPACT STUDY?
The Program Impact Study is one of
the most important processes that
identifies over time the impact of the
Institution’s programmatic outputs
and inputs and the impact that these
components had on the target
population and the University, as a
whole.
SOME BENEFITS OF THE
QUANTITATIVE/QUALITATIVE IMPACT
STUDY
Specific outcomes can be
compared to resources
expended; and successful
programs
can
be
highlighted;
SOME BENEFITS (CONTINUED)
Programs that have not
accomplished their intended
outcomes can be identified;
The extent to which the
planned outcomes were met
can be documented;
SOME BENEFITS (CONTINUED)
The
adequacy
of
the
program’s
outcomes
as
related to the problem that
the program was developed
and implemented to address
can be documented;
SOME BENEFITS (CONTINUED)
Strengths and weaknesses of each
activity can be identified that
affected the Effectiveness and
Adequacy of the programmatic
inputs ($$$, human resources,
space, etc.) in meeting the
activity ’s planned outcomes.
SOME BENEFITS (CONTINUED)
The institution has the
ability to determine the
impact of the Title III funds
awarded to various discrete
activities and programs.
SOME BENEFITS (CONTINUED)
• The process determines
not only the success or
failure of the program, but
also provides explanations
for the actual outcomes.
PRIMARY BENEFIT
A COMPREHENSIVE
picture of the overall
impact of the programs
being evaluated is the
major outcome.
WHY CONDUCT EVALUATIONS AND
IMPACT STUDIES
As a result of the MANDATE of
funding agencies and institutions to
demonstrate programmatic
accountability, one of the services
requested most by institutions of
h i g h e r e d u c a t i o n i s p r o g ra m
A S S E S S M E N T/ E VA L U AT I O N .
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
• NCCU, chartered in 1909,
opened its doors to
students in 1910 as the
National Religious Training
School and Chautauqua.
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
• North Carolina Central University is
a part of the University of North
Carolina System, made up of 16
individual campuses (17 when we
count the School of the Arts) and
headed by a single president and
governed by the University of North
Carolina Board of Governors.
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
• North Carolina Central University is
a comprehensive university offering
bachelor’s degrees in more than 100
disciplines, master’s degrees in
more than 40 programs and three
professional degrees in law.
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
• Enrollment at North Carolina Central
University increased from 5,753 in Fall 2001
to 8, 619 for Fall 2006, an increase over the
six year period of 33%. We currently enroll
8,500 students (2010-2011).
• Major emphasis for the current planning
period (2007-2012) is “Enhancing Retention,
Graduation, and Placement Rates through
Increased Student Learning.”
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
Our goals:
• to provide programs, activities, and services that
will assist students in becoming successful
during their tenure at the University so that they
persist and graduate and are successful in the
career markets of the future;
• to put in place those systems designed to make
the matriculation more seamless and easier to
negotiate;
ABOUT NORTH CAROLINA
CENTRAL UNIVERSITY
• to provide the training and upgrading of the
skills of our faculty to work with those
students whom we enroll that have been
underserved and are from diverse
backgrounds; and
• to promote a more efficient, effective, and
accountable environment as relates to
student learning and student success.
INCEPTION OF THE IMPACT STUDY:
INTERNAL MOTIVATION
• Impetus for conducting an
Study – Chancellor Charlie Nelms;
Impact
• A directive he gave me as Title III Director,
which I passed on to the Activity
Coordinators in our planning meeting;
• His absolute need to have concrete
documentation of the impact that our Title
III dollars had had over the years.
CONTRACT OUT OR CONDUCT
INTERNALLY
North
Carolina Central
University
(NCCU) contracted with Associates for
Institutional Development, Inc. (AID, Inc.)
to conduct an Impact Study of select Title
III activities encompassing the grant years
2004-2009 to document the extent to
which each activity had accomplished the
outcomes for which the activity was
funded.
THE KEY IS COLLABORATION
THE INSTITUTION
MUST:
THE RESEARCH TEAM
MUST:
•PROVIDE ACCESS
TO ALL
RECORDS
PERTINENT TO
EACH ACTIVITY
WORK CLOSELY
WITH THE TITLE
III STAFF TO
COLLECT THE
DATA
ASSESSMENT GOALS OF THE
IMPACT STUDY
A data collection process MUST be
implemented that includes:
1. Onsite visits;
2. Interviews with key staff;
3. Participant self-assessments;
ASSESSMENT GOALS OF THE IMPACT
STUDY
4. Interviews
with
faculty,
students,
and
external
stakeholders;
5. The administration of surveys
and
questionnaires,
as
needed.
ASSESSMENT GOALS OF THE
IMPACT STUDY
To guarantee objectivity, the
process relies upon multiple
sources of data encompassing
multiple points of view. Finally, to
be adaptable, the assessment’s
design must meet the assessment
needs of the institution.
INFORMATION NEEDED FOR
THE STUDY
• PROGRAM PLANS (RATIONALE, TARGET
AUDIENCE, BASELINE DATA;
• MEASURABLE OBJECTIVES;
• MEASURABLE
AND
TIME-SPECIFIC
PERFORMANCE INDICATORS;
• IMPLEMENTATION
STRATEGIES
IN
CHRONOLOGICAL ORDER
INFORMATION NEEDED FOR THE
STUDY
• DOCUMENTATION ON ACTUAL
PROGRAMMATIC
OUTCOMES
(DESCRIPTIVE AND STATISTICAL)
• EXTERNAL EVALUATION REPORTS
• PROGRESS REPORTS (MONTHLY
OR QUARTERLY)
INFORMATION NEEDED FOR
THE STUDY
• PROGRAMMATIC
OUTCOMES
(QUANTIFIABLE AND
DESCRIPTIVE)
INFORMATION NEEDED FOR THE
STUDY
• ANNUAL REPORTS
• RESULTS OF SURVEYS AND
QUESTIONNAIRES
• FUNDS
AWARDED
EACH
ACTIVITY DURING THE YEARS
INCLUDED IN THE STUDY
INFORMATION NEEDED FOR
THE STUDY
• ACTIVITY BUDGET BURN RATES
• ANEDOTAL INFORMATION THAT
WILL ADD VALUE TO THE
OUTCOMES
CRITICAL CHALLENGES TO
CONDUCTING AN IMPACT STUDY
• OBJECTIVES THAT ARE NOT
MEASURABLE
• TOO MANY TASKS MASQUERADING
AS OBJECTIVES
• PERFORMANCE INDICATORS THAT
ARE NOT MEASURABLE AND NOT
TIME-SPECIFIC
CRITICAL CHALLENGES TO
CONDUCTING AN IMPACT STUDY
• THE ABSENCE OF BASELINE DATA FOR THE
FIRST YEAR OF IMPLEMENTATION AND EACH
YEAR OF FUNDING (WHERE APPROPRIATE)
• LACK OF ANNUAL EVALUATION REPORTS OR
SOME OTHER TYPE OF ASSESSMENT TO
VERIFY THE ATTAINMENT OF THE OBJECTIVES
EACH YEAR OF FUNDING
CRITICAL CHALLENGES TO
CONDUCTING AN IMPACT STUDY
• ACTIVITY
DIRECTORS
NOT
HAVING A CLEAR VISION OF THE
PRIMARY ANTICIPATED/PLANNED
OUTCOMES OF THE ACTIVITY
AND HOW THE ACTIVITY WILL
STRENGTHEN THE INSTITUTION
IMPACT STUDY PROCESS USED AT
NCCU
• Identified the over-arching
goal or planned outcome for
each program;
• Identified the actual outcome
based on data;
IMPACT STUDY PROCESS USED AT
NCCU
• Analyzed the gap between the
desired outcome and the
actual outcome by utilizing
the “Effectiveness Estimate
(EE)” technique:
“EFFECTIVENESS ESTIMATE (EE)”
TECHNIQUE:
• DOCUMENTS:
THE EFFECTIVENESS OF THE
PROGRAM IN AMELIORATING
OR STRENGTHENING THE
ENTITY FOR WHICH IT WAS
FUNDED
ELEMENTS OF THE EFFECTIVENESS
ESTIMATE
• (R) Actual Results
• (C) Results Without Program
• (P) Planned Impact
THE EFFECTIVENESS ESTIMATE
EFFECTIVENESS ESTIMATE =
(R) Actual Results - (C) Results
Without Program
Divided By
(P) Planned Impact - (C) Results
Without Program
ADEQUACY RATIO (AR)
• QUANTIFIES:
PROGRAM ACCOMPLISHMENT
BY CONCEPTUALIZING THE
“PROPORTION
OF
THE
PROBLEM” ELIMINATED BY
THE PROGRAM
ELEMENTS OF THE ADEQUACY
RATIO
• (R) Actual Results
• (C) Results Without
Program
ADEQUACY RATIO
ADEQUACY RATIO =
(R) Actual Results
Divided By
1 - (C) Results Without Program
THE IMPACT STUDY REPORT
INCLUDES:
1.
Institutional enhancements as a
result of implementing the program;
2.
Enhancements to student and faculty
performance, where applicable;
THE IMPACT STUDY REPORT
INCLUDES:
3.
Affect on administrative personnel
effectiveness, where applicable;
4. Overall academic and/or administrative
program effectiveness enhancements;
5. Problems encountered that adversely
affected the implementation of the
program during the specified timeperiod;
THE IMPACT STUDY REPORT
INCLUDES:
• Recommendations to strengthen
the continued implementation of
each program regardless of
funding status;
THE IMPACT STUDY REPORT
INCLUDES
• Results of the Effectiveness
Estimate
analysis
to
determine the level of
program effectiveness;
THE IMPACT STUDY REPORT
INCLUDES
•Results of the Adequacy Ratio
analysis documenting the extent
to which the program addressed
the problem it was designed and
implemented to address and the
extent to which extramural
funding
had
on
program
effectiveness.
CONCLUSION
Information garnered through the
impact
study
process,
in
combination with other sources of
data, will provide an opportunity
for the institutional leadership to
incorporate the findings into
programmatic and budgetary
decision-making.
CONTACTS
• Dr. Gloria P. James: gpryorjames@comcast.net
• Dr. Brenda R. Shaw: bshaw@nccu.edu
• Mr. Samuel T. Rhoades: strhoades@aol.com