Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 AQIP CATEGORY 7: MEASURING EFFECTIVENESS
Measuring Effectiveness examines how your institution collects, analyzes, and uses information to
manage itself and to drive performance improvement. It examines your institution's processes and
systems related to collection, storage, management, and use of information and data – at the
institutional and departmental/unit levels; institutional measures of effectiveness; information and data
alignment with institutional needs and directions; comparative information and data; analysis of
information and data; effectiveness of information system and processes; measures; analysis of results;
and efforts to continuously improve these areas.
Here are the Key Critical Characteristics of Eastern Michigan University that were identified by the
Systems Appraisal Team as most relevant for its interpretation of its Systems Portfolio section covering
Category 7, Measuring Effectiveness:
Item
Critical Characteristic
O1e.
Eastern Michigan University is currently engaged in a thoughtful, evidence-based approach
to campus-wide planning.
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University. Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 O1f.
Eastern Michigan University is guided by a mission that emphasizes excellence in teaching,
applied research, and artistic expression.
O1g.
Eastern Michigan University has adopted five core principles and four strategic directions
that guide decision-making, which emphasize learning, engagement with the community, and
diversity and multiculturalism.
Here are what the Systems Appraisal Team identified as Eastern Michigan University’s most important
strengths and opportunities for improvement relating to processes encompassed by Category 7,
Measuring Effectiveness.
Item
S/O
7P1
O
Comment
The University has identified a series of measures in four key areas. However, it
has not specified any processes by which information is selected, managed and
used. It is not clear the extent to which data is available for the range of identified
measures, how it is being used, by whom and for what purposes. It is also
unclear how the University uses its information systems to directly measure and
improve student learning.
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University. Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 7P1
S
Figure 7.2 (Identified Measures) outlines the key categories and current
measures being used at Eastern Michigan University. The categories and
measures were selected as part of the University’s AQIP Balanced Scorecard
action process. The development and use of a balanced scorecard will allow
Eastern Michigan University to develop a culture of continuous improvement,
strengthen academic programs, enhance student learning, and improve
institutional operations.
7P2
O
Eastern Michigan University’s data needs appear largely determined by the
requirements of external agencies, such as accrediting bodies, advisory
committees, and government agencies, although departments and other units
also have certain needs. However, the University has not identified the
processes by which information needs are actually determined. Given the
development of new planning systems, Eastern Michigan University will benefit
by a more focused effort to associate internal data needs with the KPI’s and
guide the organization through its continuous improvement initiatives. While all
institutions must be responsive to external agencies, Eastern Michigan University
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University. Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 should insure that it also focuses on data needs related to its newly-developing
internal systems.
7P4-7P5
O
Processes by which the analysis of information and data is conducted at the
institutional level have not yet been identified. The processes described focus
predominantly on reports being generated at the division or office level and, as
appropriate, disseminated through divisional web sites. However, institutionallevel analysis, involving systematic review and integration of information, is not
identified. The development of the University’s Balanced Scorecard, when fully
implemented, is expected to provide a mechanism whereby information is more
systematically utilized at the institutional level. At this point, it is assumed that
processes for aligning departmental and unit analysis with institutional objectives
will emerge from the use of the balanced scorecard.
7P6
S
Eastern Michigan University has implemented adequate systems to provide
access to data at the departmental level, where appropriate. A Data Standards
Committee meets regularly to review established standards for entering and
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University. Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 processing data. Security mechanisms appear to be in place to ensure that data
are secure and free from intrusion or corruption.
7P7
O
Measures of the effectiveness of the information and data systems have been
identified, which may include timeliness, usefulness, accuracy, reliability,
meaningfulness and accessibility. However, it is not clear the extent to which
measurement of these criteria are utilized across all areas of the University.
Faculty and student surveys are used to elicit feedback, along with help-desk
satisfaction data. It appears that concerns remain about the quality and response
rates of these systems. Eastern Michigan University is currently waiting for the
full implementation of the Balanced Scorecard in order to make more effective
judgments about the effectiveness of its information systems.
7R1
S
Eastern Michigan University has found its data and information systems
adequate for its reporting needs regarding governmental and accrediting
agencies. As the University moves forward with its new strategic planning and
continuous improvement processes, along with the implementation of the
Balanced Scorecard, the adequacy of the data systems will be judged from new
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University. Eastern Michigan University Systems Appraisal Feedback Report October 10, 2008 vantage points and under new criteria. Given these institutional changes, it will
be important for the University to regularly monitor the effectiveness of the data
systems.
7R2
O
There are currently no comparison data available which allow the University to
compare its data and information systems with those of other institutions.
7I1
S
The institution uses a variety of means to improve current processes and
systems for measuring effectiveness. For instance, the University uses process
improvement teams, AQIP Action Projects, strategic planning and the Balanced
Scorecard.
7I1-7I2
O
Improvements in these systems are being developed in conjunction with the new
planning and improvement systems. Some of these changes appear to have
been initiated in response to prior evidence of weaknesses obtained from the
existing systems. It will be important for the University to regularly utilize its new
measures to assess the new systems as they are implemented.
© 2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. This report may be reproduced and distributed freely by Eastern Michigan University.