CLEI
College Learning Effectiveness Inventory
Administration and
Scoring Manual for CLEI
© 2008 K-CAT · All rights reserved
2005 Research Park Circle · Manhattan, KS 66502
www.k-cat.org
ADMINISTRATION AND SCORING MANUAL FOR
THE COLLEGE LEARNING EFFECTIVENESS INVENTORY (CLEI)
by
Fred B. Newton, Eunhee Kim, Dan Wilcox, and Nathan Beemer
Kansas State University
Contributions in the development of the CLEI include recognition to
Ann Johnson, Wen-Chih Tseng, Kang-Hyun Shin, Mary Elizabeth Yeager, Ron Downey,
Stephen Benton, and participating staff from the KSU Counseling Services
2008 Edition
Copyright 2008 by Kansas State University Research Foundation
CONTENTS
Chapter 1
Overview ………………………………………………….………… 4
Introduction
Purpose
Item Development
Data Collection Procedures
Chapter 2
Administration and Scoring ……………………………………… 7
Administration
Appropriate Use
User Qualification
Scoring Procedures
Chapter 3
Interpretation of Results …………………………………………. 11
Interpretation of the Six Scales
Profile Interpretation
Chapter 4
Psychometric Information ………………………………………... 16
Sample Description
Exploratory Factor Analysis
Confirmatory Factor Analysis
Reliability
Validity
Chapter 5
Summary, Future Research, & Limitations …………………….. 19
References
Tables
2
Tables
Table 1
Items of the Six-CLEI Scales
Table 2
Means and Standard Deviations for the CLEI Scores for a Normative
Sample of College Undergraduates by Gender
Table 3
Means and Standard Deviations for the CLEI Scores for a Normative
Sample of College Undergraduates by Year in College
Table 4
Item Statistics for the Academic Self-Efficacy Scale
Table 5
Item Statistics for the Organization and Attention to Study Scale
Table 6
Item Statistics for the Stress and Time Press Scale
Table 7
Item Statistics for the Involvement with College Activity Scale
Table 8
Item Statistics for the Emotional Satisfaction Scale
Table 9
Item Statistics for the Class Communication Scale
Table 10
Scale Statistics: Mean, Standard Deviations, Intercorrelations, and Reliability
Coefficients with 95% Confidence Intervals for the Six Scales of the
CLEI
Table 11
Overall Fit Indices of the Confirmatory Factor Analysis on the Six Scales of
the CLEI
Table 12
Relationship between the CLEI and Validation Instruments
3
Chapter 1: Overview
Introduction
Counselors, academic advisors, and other professional support personnel at academic
institutions have a long history of using assessment to determine students’ academic ability
and achievement potential. Common practice has been to use ability tests and past academic
performance as standard measures. However, while ability and achievement measures are
important factors in predicting a student’s academic success, these assessments do not
explain a significant amount of individual variability in performance. Research summaries
by Astin (1993), Pascarella and Terenzini (1991), Russell and Petrie (1992), and Tinto (1992)
support the premise that student learning is impacted by many forces that include academic,
personal, social, and environmental variables. Russell and Petrie (1992) use three category
labels to describe factors: academic, social/environmental, and personality. These labels
served as a guide during the initial development of items.
Several writers have described principles of learning, based upon individual behavior
and impact of the learning environment that affect performance outcomes. Angelo (1993)
discusses thirteen research-based principles that can guide efforts to maximize learning.
Others have shown how specific personal factors can be directly influenced through
intervention strategies (Halstead, 1993; Newton and Smith, 1996; Newton, 1990). These
variables associated with the person within the context of campus experience are described as
psychosocial factors. They include personal, social, and environmental variables such as
aptitudes and abilities, attitudes, motivation, study approach, vocational interests, utilization
of campus resources, and sources of personal support available and utilized.
Critical to the identification of these variables has been the determination of the
relationship between the psychosocial variables and outcome performance. Identifiable
outcomes have included GPA, persistence or attrition in academic enrollment, satisfaction
with college life, adjustment defined by emotional wellbeing, self and other judgment toward
achieving goal success, and positive change over time measuring specified criteria.
Empirical studies of influence factors have provided evidence to suggest that relationships
exist between such variables and college performance (Chemers, Hu, & Garcia, 2001;
Davidson & Beck, 2006; Friedlander, Reid, Shupak & Cribbie, 2007; Lahmers & Zulauf,
2000; Macan, Shanhani, Dipboye & Phillips, 1990). Understanding the relationship between
certain psychosocial variables and success outcomes provides the opportunity for designing
educational and supportive strategies. Because of this relationship between psychosocial
influences and academic success, we believe that academic services personnel, with some
relevant training in both assessment and intervention, can make a significant difference in
student success in both the classroom and the overall campus environment.
Related to psychosocial influence on academic performance tasks, researchers have
evaluated the impact of specific variables such as time utilization (Lahmers & Zulauf, 2000;
Macan, Shahani, Dipboye & Phillips, 1990; Nonis & Hudson, 2006), strategic organization
and study approach (VanZile-Tamsen, 2001), academic self-esteem, efficacy, and confidence
(Chemers, Hu & Garcia, 2001; Friedlander, Reid, Shupak, & Cribbie, 2007; Lent, Brown, &
Larkin, 1984; Zajacova, Lynch, & Espenshade, 2005), stress and emotional components
(Davidson & Beck, 2006; Pritchard & Wilson, 2003), student involvement with campus life
(Anaya, 1996; Cooper, Healy, & Simpson, 1994), and motivation and task relevance (Bong,
4
2004; VanZile-Tamsen, 2001). Background research on these types of variables provides a
basis for identifying the need for an assessment instrument that can provide a quantitative
measure of the attitudes, behaviors, and dispositions that lead to success outcomes.
Purpose
The College Learning Effectiveness Inventory (CLEI) was conceived as a method to
measure some of these psychosocial factors that impact a student’s learning. The CLEI is an
inventory of six scales with 50 questions representing a continuum of individual behaviors,
attitudes, and dispositions related to academic activity. The purpose of the inventory is to
organize the self-reported student responses into thematic domains or categories that have
shown to contribute to academic success. These domains comprise the six scales of
measurement on the CLEI. After students respond to the items they immediately receive a
profile that reports their overall pattern including strengths and weaknesses. The initial
profile provides some immediate explanation of results to the student, however, it has greater
utility when used as an advising and planning tool by student service personnel working with
students to make improvements to optimize their learning approach. The major objectives of
creating the CLEI were to:
(a) Develop a series of clearly definable and operational items that a student could
use to measure their own thoughts, feelings, or behaviors related to academic
pursuits;
(b) Include a continuum, that reflects the degree from positive to negative, thereby
showing how the item content might support or interfere with academic
pursuits;
(c) Utilize an on-line survey format for ease of access, user-friendly administration
and rapid retrieval of results;
(d) Provide immediate feedback and information to the student completing the
inventory by showing a pattern of strengths and weakness on an individualized
profile;
(e) Provide information for advising and counseling a student, making it a tool for
discussion of goals, selection of interventions, referral to relevant student
services, and a measurement of progress and involvement in the change process;
(f) And, to utilize the CLEI as a tool for research describing relationships between
variables, measures of change, and outcome comparisons.
Item Development
The CLEI was developed over the past ten years in two stages. Initially, three
professional counseling staff, knowledgeable in the literature on learning, began generating
behavioral statements that would serve as potential measures of the six categories that were
seen as important to success. These categories included motivation, self-concept, study
habits, emotions, support, and involvement. Eventually, an item pool of over 300 items was
generated. After these items were identified a panel of nine expert judges, with experience
and credentials related to student learning, went through a systematic process to refine and
reduce these items. The panel judged each question on three criteria: (1) clarity and relevance
to college students, (2) accurateness and goodness of fit to the operational definitions of the
5
categories, and (3) placement of the item on a five point Likert scale (a score of “1” equals a
high negative behavior and “5” represents a high positive behavior). The intent of the judging
process was to eliminate non-discriminating and invalid questions. This process resulted in
the identification of 144 items covering the six categories and ranging from positive to
negative in wording.
The first generation CLEI with 144 items was piloted with over 500 students and
utilized with student “academic assistance groups” over a period of five years. Results from
the initial piloting of the CLEI found it to be a useful tool for counseling and advising
students on their academic problems. However, a review of this first version identified the
following problems: repetitive items similar in nature and the large number of items took too
much time to complete the inventory. An exploratory factor analysis was used to identify
and eliminate redundant questions and questions that lacked predictive value. This resulted
in the second version of the CLEI with 61 items compiling the six scales (Newton, Kim,
Wilcox, & Yeager, 2007).
The second version of the CLEI was tested with over 1,000 students for additional
studies including confirmatory factor analyses, reliability studies, and validity studies. It was
also tested to examine a scoring system utilizing T-Scores based on normative scores. These
studies suggested that the scales could have stronger psychometric properties by respecifying
the scales. Examinations of normative score distributions and individual response patterns
on each of the scales indicated that there was a slight positive skewing on several scales but a
significant skewing in positive direction for the Academic Self-Efficacy Scale. This
suggested that utilizing T-Scores could provide better interpretation of the individual profile
of the scales. Revisions were then made to the second version of the CLEI and yielded the
current 50 item, six scale version (2008 Edition).
The current College Learning Effectiveness Inventory (CLEI) consists of six scales.
This 50-item assessment tool measures individual attitudes and behaviors that may impact
academic performance. The CLEI serves both diagnostic and prescriptive functions.
Academic advisors, counselors and others whose work involves supporting student success
and retention can use the CLEI to assess individual student’s strengths and weaknesses.
Interventions can then be custom designed to address each student’s weaknesses, build upon
their strengths, and enhance their chances of success.
In addition, demographic variables are included with this on-line version of the CLEI.
These variables are gender, age, year in school, overall GPA, academic major, ethnic
identity, and residence type. Each college or university using the CLEI may elect to identify
demographic variables that are of particular interest in their on their campus.
Data Collection Procedures
Data for developing this revised CLEI and for developing normative profiles were
collected from undergraduate students who were enrolled in a large public university
(enrollment greater than 20,000 students) in the Midwest. Participants completed the online
CLEI via the Internet at remote locations during academic years 2006 and 2007. Sample
descriptions are presented in Chapter 4. These studies were approved by the Committee for
Research Involving Human Subjects under the University Research Compliance Office at
Kansas State University.
6
Chapter 2: Administration and Scoring
Administration
The CLEI is administered on-line via the Internet and scored automatically. Students
are given a password in order to access and take the CLEI. The inventory can usually be
completed in 10 to 15 minutes. Individual results are provided immediately after the CLEI is
completed. Students can print out their results and are provided a brief interpretative
summary. It is recommended that students also work with an advisor, counselor, trained peer
mentor, or faculty and staff familiar with the CLEI in order to utilize the results most
effectively. A follow-up interview or group explanation session is a way to review student
CLEI profiles, discuss the results, and begin to set goals and make recommendations for
action. The process follows the basic problem solving model utilizing the steps of “what”,
“so what”, and “now what”. Alternatively stated, the goal for students using the CLEI for
academic enhancement is to utilize the process of identification, implication, and
improvement.
Appropriate Use
(1)
(2)
(3)
(4)
(5)
The CLEI can be used in the following ways:
As an assessment tool that helps each student become aware of attitudes and
behaviors that affect their learning and studying.
As an organizing assessment that identifies specific areas in which each student
could benefit most from interventions.
To develop specific interventions designed to remediate weaknesses and
capitalize on strengths.
In pre-post measure to determine the effectiveness of specific interventions, and
to determine if additional interventions are needed.
As a counseling or psychoeducational strategy for college orientation, advising,
educational development, and learning skills programs.
User Qualifications
The CLEI is designed as an easy to use tool that organizes the pattern and potential
meaning of a student’s self report on personal behaviors, attitudes, and feelings toward
academic activity. At the most basic level it can become an information source for the
student to view their own personal motivations and approach to academic activity.
Definitions, examples, and suggestions for follow-up options are provided on the profile
screen. The questions asked are not psychologically sensitive and the terminology does not
reflect concern for pathology or danger. The CLEI may also be used as a source of input for
advising or counseling a student on their approach to learning. It is recommended that the
best results in the use of CLEI will come from a professional or paraprofessional staff person
who has training with the assessment of academic concerns and is familiar with the
development and theory behind the CLEI.
7
An important supplement to CLEI results is the Student Learning Effectiveness
Workbook. The workbook outlines a step by step process to translate CLEI profile results
into personal action plan to make individualized improvements in one’s approach toward
learning.
8
Scoring Procedures
All scoring is generated automatically on-line by the use of formula based upon the
scale construction explained in Chapter 4. Immediately after the CLEI is completed,
individual results are provided to the student in the form of a profile chart. An individual
profile displays individual’s mean raw scores and T-Scores for each of the six scales. Prior
to calculating individual scores, items in negative continuum are transformed to reverse
scores (i.e., a score of 1 is transformed to a score of 5, score 2 to 4, score 4 to 2, score 5 to 1).
Table 1 provides a list of items for each of the six scales, and items with asterisk (*) indicates
items in negative continuum. Items are grouped and ordered by the loading size on each
scale to facilitate interpretation. The mentor working with a student can administratively
obtain an individual score sheet of the student that contains raw item scores, mean scores,
and T-Scores in addition to the profile chart.
Raw Mean Scores. The purpose of providing an individual raw mean score for each
of the six scales is to provide an interpretation that demonstrates an individual’s profile of
high and low scores indicating strengths and weaknesses from an intrapersonal perspective.
The mean score for each scale can vary from 1.0 (lowest possible score) to 5.0 (highest
possible). A low score reflects a negative response to the attributes of the scale, and a high
score represents a positive response to the attributes of the scale. A raw mean score will fall
somewhere between these two extremes. A score of 3.0 is the mid-point and any score
within the middle range of between 2.6 and 3.4 is more neutral and less likely to reflect a
strength or weakness. However, it should be noted that on any scale a mean score of 3.5 or
above would indicate a more positive response pattern and considered an area of personal
strength. A scale score of 2.5 or below would reflect more negative response or potential
area of weakness.
T-Scores. An individual student’s mean raw score for each scale is compared with
average scale scores generated by a normative sample, and T-Scores are calculated for each
of the six scales. Means and standard deviations (SD) of scores from the normative sample
are used to transform an individual student’s mean scores into T-Sores for each of the scales
using the following formula:
T-Score = 10*[(Individual Mean – Normative Mean)/SD of Normative Score]+50.
T-Scores are interpreted using a mean of 50 and a standard deviation of 10:
•
•
•
68% of the normative group falls between T=40 and T=60
96% of the normative group falls between T=30 and T=70
99% of the normative group falls between T=20 and T=80.
Typically, the T-scores for each scale will distribute in a manner called the normal inverted U
curve. This means more students cluster around the middle or average score of 50 with
fewer students as the scores move away from the middle. The one exception to this rule
when interpreting the CLEI scales is the first scale called Academic Self-Efficacy. The
9
majority of students rate their responses positively on this scale. As a result, a score around
50 to slightly below may still reflect a high positive self rating.
Normative Scores. The means and standard deviations for the CLEI scales obtained
from a normative sample are reported in Table 2 and Table. The participants in the
normative sample were university students (N=879) who were enrolled in a large public
university in the Midwest during academic years of 2006-2007. The normative scores for
each of the six scales are presented by the demographical factors of gender (Table 2) and
year-in-college (Table 3). This normative sample was weighted to reflect proportions of
male and female students. It was also weighted to reflect class level proportions in the
university population. The weighted sample consisted of 439 male students (49.9%) and 440
female students (50.1%). With regard to class level, 24.0% were freshmen (N=211), 21.1%
were sophomore students (N=186), 22.0% were juniors (N=193), and 33.0% were seniors
(N=290).
10
Chapter 3: Interpretation of Results
Interpretation of the Six Scales
!
Academic Self-Efficacy (ASE Scale): Items on this scale measures an expression of
confidence in academic ability, awareness of effort toward study, and expectations of
success in college attainment. Those who score high have expectations to succeed
and accomplish important outcome goals. Those who score low are more likely to
feel uncertain about possible achievement and what the future may hold.
*Note: The Academic Self-Efficacy scale is skewed toward overall positive
responses, therefore, caution should be made in interpretation recognizing that a T-score
of 40
would be below the 50% of the norm group but would still reflecting a
positive overall
response on the scale items.
!
Organization and Attention to Study (OAS Scale): This scale measures the
organization of tasks and structuring of time to set goals, plan, and carry out
necessary academic activity. Those who score high are likely to use effective
organizational planning and time management skills to achieve academic success.
Those who score low are more likely to avoid planning strategies and lack focus of
attention in providing self-direction.
!
Stress and Time Press (STP Scale): This scale measures how student deals with the
pressures of time, environmental concerns, and the academic demand that impacts
academic study. Those who score high manage the pressures of academics without
reactions such as being overwhelmed, procrastination, or avoidance. Those who
score low may experience symptoms of stress and do not believe they can handle the
academic demands they experience.
!
Involvement with College Activity (ICA Scale): Involvement is defined by the ICA
Scale as belonging to organizations and participating in activities. Those who score
high belong to many organizations and often participate in formal and/or informal
campus activities. Those who score low are more socially isolated and are less likely
to participate or engage in campus activities.
!
Emotional Satisfaction (ES Scale): This scale measures the degree of interest and
emotional response in academic life including people and the campus educational
environment. Those who score high express encouragement, interest, and positive
anticipation for academic life, whereas those who score low are more likely to
express discouragement, negative reactions, and a sense of being overwhelmed with
academic life.
!
Class Communication (CC Scale): Communication in the CC Scale includes both
verbal and non-verbal effort to engage in class activity. Those who score high scorers
are assertive and active with written and oral communication in-class and with their
11
instructors. Those who score low may experience uncertainty and reluctance in
expressing and asserting their ideas in-class and with their instructors.
12
Profile Interpretation
Profiles from the CLEI may be best used as stimulus for discussion with the student.
One approach is to examine the individualized summary noting the position of scale scores
from high to low. To provide examples of profile interpretation, Figure 1, 2, and 3 illustrate
profile charts for three students.
Example Profile A. The profile chart A is for a female student, who is a sophomore,
majoring in Business/Marketing with a GPA of 3.0-3.4 on a 4.0 scale. Her two highest
scores are in Academic Self-Efficacy and Involvement with College Activity with a midrange score in Class Communication. Her lower scores are in Emotional Satisfaction,
Organization and Attention to Study, and Stress and Time Press. These scores indicate that
she is a confident and out-going student who is likely to be very active on campus and
engaged in her college experience. However, these results also suggest that her strengths
may create problems such as being pressed for time, feeling pressure to become better
organized and having some worry or stress about academics.
Profile A
Fig. 1. Example Profile Chart A
13
Example Profile B. The profile chart B is for a female student, who is a junior,
majoring in Elementary Education with a GPA of 2.5-2.9 on a 4.0 scale. The student’s
profile is marked by low scores on the Scales 2, 3, and 4 (lack of organization and attention
to study, stress and time press, and low involvement with college activity). The student is in
the mid-range on Academic Self-Efficacy and Emotional Satisfaction Scales. The reason for
stress and time press, lack of organization and attention to study, and low activity level may
reflect the presence of some situational stressor occurring at this time in her life. In some
cases, it might be family issues, the need to work extra hours to meet financial obligations, or
a personal problem that is interfering with college life. Is this a situation stressor occurring
in her life? Or, does she have a history of anxiety or stress when preparing her assignments
or when performing tasks? Exploring these contrasting scores with the student could help to
identify possible solutions or result in referral to the appropriate service or support program
on campus.
Profile B
Fig. 2. Example Profile Chart B
14
Example Profile C. This sophomore male student is struggling in college with a
grade point average below 1.9. The profile shown in Figure 3 indicates flat and below
average scores across five of the scales (1, 2, 4, 5, & 6). Scale 3 indicates a fairly average
amount of stress or time pressure on the individual. This suggests several possibilities to
explore. One area might be competing involvement in work, family, or other activity
unrelated to college that distracts the student from sufficient engagement to succeed in
academic work. Another possibility could reflect uncertainty as to whether college is desired
or necessary to achieve his personal goals. Career planning, personal decision-making, or
more exploration could be important for this student to find a successful niche.
Profile C
Fig. 3. Example Profile Chart C
15
Chapter 4: Psychometric Information
Sample Description
Samples of university students were used in three separate studies: (a) a derivation
sample for an exploratory factor analysis (EFA), (b) a replication sample for a confirmatory
study (CFA), and (c) a validation sample for a cross-validation study. Participants were
university undergraduate students who were enrolled in a large public university (enrollment
greater than 20,000 students) in the Midwest.
Derivation Sample. A sample of 587 university undergraduate students was used in
the derivation sample for EFA. The average age of the participants was 21.21 (SD= 4.33),
with an age range of 17 to 56 years. About two-thirds of the participants were women
(N=405, 67.8%). Class levels included freshmen (51.4%), sophomores (12.4%), juniors
(16.4%), and seniors (19.8%). Reported overall Grade Point Averages (GPA) were: 3.5 and
above (33.9%), 3.0 to 3.4 (32.9%), 2.5 to 2.9 (21.7%), 2.0 to 2.4 (6.5%), and below 2.0
(5.0%). Ethnically, 86.6% reported that they were Caucasian, 3.4% were Hispanic/Latino
American, 3.0% were African American/Black, 1.5% were Asian American, 0.3% were
Native American, and 0.2% were multicultural. Majority students lived off campus (55.3%)
and 44.7% lived on-campus.
Replication Sample. A sample of 282 university undergraduate students was used in
the replication sample for CFA. The average age of the participants was 20.91 (SD= 3.71),
with an age range of 18 to 45 years. In this sample, 73.4% were women (N=207). Class
levels included freshmen (36.9%), sophomores (4.6%), juniors (20.6%), and seniors (37.9%).
Reported overall Grade Point Averages (GPA) were: 3.5 and above (30.9%), 3.0 to 3.4
(31.2%), 2.5 to 2.9 (18.8%), 2.0 to 2.4 (3.5%), and below 2.0 (0.4%). Regarding ethnicity,
90.4% reported that they were Caucasian, 1.8% were Hispanic/Latino American, 2.1% were
African American/Black, 1.4% were Mexican American, 0.7% were Asian American, 0.7%
were Native American, and 2.8% were multicultural. Majority students lived off campus
(60.3%), 29.1% lived on-campus, and 10.6% lived at fraternity/sorority houses.
Validation Sample. A sample of 160 university undergraduate students was used in
the construct validity study. The average age of the participants was 22.44 (SD= 4.74), with
an age range of 19 to 49 years. In this sample, 73.1% were women (N=117). Majority of the
participants were seniors (61.9%), 33.1% were juniors, and 3.8% were sophomores. There
were no freshmen among the respondents. Most students participating in the validation study
have a cumulative grade point average of greater than or equal to 3.00 (77.9%). Reported
overall Grade Point Averages (GPA) were: 3.5 and above (39.9%), 3.0 to 3.4 (38.0%), and
2.5 to 2.9 (20.3%). None of the participants had their GPA below 2.5. Ethnically, most of
the participants were Caucasian (93.1%), and minority group consisted of Hispanic/Latino
American or Mexican/Mexican Americans (2.4%), African Americans (1.2%), Asian/Pacific
Islander (1.2%), Native American/Alaskan Native (1.2%), and multiracial (0.6%). Majority
students lived off campus (83.8%), 4.4% lived on-campus, and 8.8% lived at
fraternity/sorority houses.
16
Exploratory Factor Analysis
An exploratory factor analysis was conducted on a derivation sample (N=597) to
identify underlying factors of the 62 items of the CLEI. We performed a principal
component extraction with promax (oblique) rotation of the correlation matrix from the
derivation data. Based on the factor analysis and conceptual analysis by the expert panel, we
adopted the revised CLEI version comprised of six scales and 50 items. Summaries of the
item statistics can be found in Table 4 through Table 9 for each of the six scales. Items are
ordered and grouped by size of coefficients to facilitate interpretation. Table 10 presents
scale statistics including means, standard deviations, intercorrelations, and reliability
coefficients with 95% confidence intervals for the six scales of the CLEI. The scale labels
with number of items and reliability coefficients of internal consistency for these six scales
are provided below.
(1) Academic Self-Efficacy (ASE Scale: 14 items, Cronbach’s Alpha=.87)
(2) Organization and Attention to Study (OAS Scale: 8 items, Cronbach’s Alpha=.81)
(3) Stress and Time Press (STP Scale: 6 items: Cronbach’s Alpha=.77)
(4) Involvement with College Activity (ICA Scale: 9 items, Cronbach’s Alpha=.81)
(5) Emotional Satisfaction (ES Scale: 7 items, Cronbach’s Alpha =.72)
(6) Class Communication (CC Scale: 6 items, Cronbach’s Alpha=.68)
Confirmatory Factor Analysis
To examine whether the six factors of the CLEI from the derivation sample of the
exploratory factor analysis adequately applied to the replication sample, we conducted a
confirmatory factor analysis on a replication sample (CFA sample, N=292) using unweighted
least squares estimate and correlated errors within scales estimation. Seven fit indices were
used to determine the data fit of the hypothesized model: (a) chi-square (Satorra-Bentler
Scaled Chi-Square), (b) normed fit index (NFI), (3) goodness-of-fit index (GFI), (d) adjusted
goodness-of-fit index (AGFI), (e) root mean square residual (RMSR), (f) root mean square
error of approximation residual (RMSEA), and (g) comparative fit index (CFI). As shown in
Table 11, overall fit indices indicated that the CFA for the six CLEI scales resulted in a
generally acceptable fit.
Reliability
The internal consistency of scores for the six scales was examined on the EFA sample
(N=597) and CFA sample (N=292). The respective Cronbah alphas for the EFA and CFA
samples were: ASE (.87 and .86), OAS (.81 and .81), STP (.77 and .71), ICA (.81 and .71),
ES (.72 and .74), and CC (.68 and .68). Trends in reliability alphas for the exploratory
sample and for the confirmatory sample were consistent. Scores for five of the six scales had
reliabilities that were adequate ranging from .71 to .87, while scores for the Class
Communication Scale had marginal reliabilities for both EFA and CFA samples. Reliability
coefficients with 95% confidence intervals for the six scales were presented in Table 10.
17
Validity
To examine the degree to which the six scales measure what they were intended to
measure, construct validity was tested on a validation sample (N=160). The validation
instruments used were: (a) selected subscales of the Learning and Study Strategies Inventory
(LASSI) (Weinstein & Palmer, 2002), (b) Rosenberg Self-Esteem Scale (Rosenberg, 1965),
(c) Crombag College Adaptation Questionnaire (Van Rooijen, 1986), and (d) Student
Propensity to Ask Questions (Cunconnan, 1996; Cayanus, 2005).
Correlations between the validation instruments and the CLEI scales are shown in
Table 12. The Academic Self-Efficacy Scale of the CLEI was correlated with both the
Motivation Scale of the LASSI (r=.46) and Rosenberg Self-Esteem Scale (r=.45). The
Organization and Attention to Study Scale showed correlations with two subscales of the
LASSI: Concentration Scale (r=.71) and Self-Testing Scale (r=.46). The Stress and Time
Press Scale was correlated with the Time Management Scale of the LASSI (r=.44). The
correlation coefficient between the Involvement with College Activity Scale and the Student
Adaptation to College Questionnaire was a bit low (r=.31), but it was statistically significant.
The Emotional Satisfaction Scale was correlated with the Attitude Scale of the LASSI
(r=.50), and the Class Communication Scale was correlated with the Student Propensity to
Ask Questions (r=.53).
18
Chapter 5: Summary, Future Research, & Limitations
Summary
The CLEI is an instrument designed to measure individual attitudes and behaviors
that may impact academic performance. This instrument was developed because a group of
professionals (counselors and advisors) identified the need for a tool to explore what was
described as personal/social influences that impacted learning performance.
Over 300 items were originally developed through both the inductive experience in
working with students and through the collective published information on college student
learning. During the past several years the instrument has gone through a process to clarify
and define the psychometric properties of the instrument. The present edition of the CLEI
(2008 Edition) contains 50 items that load on to six scales measuring positive and negative
levels of student attitudes and behaviors.
The CLEI can function as a tool of exploration for students to increase their
awareness of personal strengths and weaknesses as a learner and be able to make changes
that may improve academic performance. It may be most effective with professionals trained
to assist students with academic problems (counselors, advisors, learning center staff).
However, it is also available as a self-help on-line inventory that provides individual students
feedback and suggestion on their learning effectiveness.
Future Research
The CLEI should be considered as an instrument in development. There are several
areas of research that are needed to contribute to the understanding and use of this
instrument. The followings are suggestions for research, studies that are either in process or
anticipated within the near future. Results from these studies will be reported at a later date
and included within this manual.
1.
A larger and more diverse sample of students to be tested to establish a
normative profile of information. This sample would be drawn from
institutions representing different geographical areas of the country, different
institutional types (private, public, 2-year), size, and degree emphasis.
2.
Comparisons between identifiable sub-groups of students to discriminate
response patterns including demographic profile and academic definition
clusters (gender, major, year in school, ethnicity, first generation, probation
status, honor student status, etc.).
3.
Additional studies of psychometric properties including test-retest
examination of the instrument’s stability, and comparison of a group of
students and a group of advisors to examine convergent validity.
4.
Utilizing a complete battery of the three K-CAT inventories to provide a
broad picture of college students’ attitudes, behaviors, and health patterns
related to their overall functioning in college. These three inventories (K-
19
PIRS, HBA, & CLEI) measure three dimensions of health behaviors, mental
health, learning attitudes and behaviors providing a comprehensive view of
psycho-social variables that may impact student performance. The suggestion
is to utilize testing of freshmen students early in their college career and track
changes with academic outcomes through their college experience.
5.
Utilizing the CLEI as a tool for prescribed programs of intervention that can
measure students at pre, post, and follow-up periods of time to determine the
efficacy of the instrument for tracking change.
Limitations
The CLEI is a tool to be used to identify potential strengths and weaknesses of
student attitudes and behaviors related to academic performance. As such, it should be used
for exploration, clarification, and a way to initiate conversation on ways to maintain or
improve their learning activity. The CLEI is not proven as an evaluation, diagnostic, or
prescriptive instrument. It should not be used to determine learning disabilities, eligibility
for accommodations, or as a basis for determination of success or failure decisions.
The CLEI has not been tested on many specific sub-populations of students and may
not generalize to the behavior and attitudes considered important for the success of students
in special or unique situations. For example, graduate students or distance learning students
may not be appropriate for many items that measure involvement in on-campus activities or
in-class performance activities.
20
REFERENCES
Angelo, T. A. (1993). A teacher’s dozen: Fourteen general findings from research that can
reform classroom teaching and assessment and improve learning. American Association of
Higher Education Bulletin, 3-8.
Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco:
Jossey-Bass, Inc.
Bong, M. (2004). Academic motivation in self-efficacy, task value, achievement goal
orientations, and attributional beliefs. The Journal of Educational Research, 97 (6), 287-297.
Cayanus, J. L. (2005). Student’s propensity to ask questions: Do cognitive flexibility, teacher
self-disclosure, student motives to communicate, and affective learning influence question
asking in the classroom? Unpublished doctoral dissertation, West Virginia University,
Morgantown.
Chemers, M.M., Hu, L. & Garcia, B. F. (2001). Academic self-efficacy and first-year college
student performance and adjustment. Journal of Educational Psychology, 93 (1), 55-64.
Cunconan, T. M. (1996). The conceptualization, measurement, and validation of a student’s
propensity to ask questions in the college classroom. Unpublished doctoral dissertation,
University of Oklahoma, Norman.
Davidson, W. B. & Beck, H. P. (2006). Using the survey of academic orientations to predict
undergraduates’ stress levels. NACADA Journal, 26 (2), 13-20.
Friedlander, L.J., Reid, G.j., Shupak, N. & Cribbie, R. (2007). Social support, self-esteem,
and stress as predictors of adjustment to university among first-year undergraduates. Journal
of College Student Development, 48 (3), 259-274.
Gibson, D.M. & Myers, J.E. (2006). Perceived stress, wellness, and mattering: A profile of
First-year Citadel cadets. Journal of College Student Development, 47 (6), 647-660.
Halstead, R. W. (1993). The 16 principles: A guide to getting better grades in college.
Worcester, MA: COB Press.
Lahmers, A. G. & Zulauf, C.R. (2000). Factors associated with academic time use and
academic performance of college students: A recursive approach. Journal of College Student
Development, 41 (5), 544-556.
Lent, R.W., Brown, S.D., & Larkin, KC. (1984). Relation of self-efficacy expectations to
academic achievement and persistence. Journal of Counseling Psychology, 31, 352-362.
21
Macan, T. H., Shahani, C., Dipboye, R. L. & Phillips, A. P. (1990). College students’ time
management: Correlations with academic performance and stress. Journal of Educational
Psychology, 82 (4), 760-767.
Newton, F. B. (1990). Academic support seminars: A program to assist students
experiencing academic difficulty. Journal of College Student Development, 31(2), 183-186.
Newton, F.B., Kim, E., Wilcox, D. & Yeager, M.E. (2007). Administration and scoring
manual for the College Learning Effectiveness Inventory 2007 edition. Kansas State
University Research Foundation.
Newton, F. B. & Smith, J. H. (1996). Principles and strategies for enhancing student learning.
In. Ender, S. C., Newton, F. B. & Caple, R. B. Contributing to learning: The role of student
affairs. San Francisco: Jossey-Bass, Inc.
Nonis, S. A. & Hudson, G. I. (2006). Academic performance of college students: Influence
of time spent studying and working. Journal of Education for Business, 81 (3), 151-159.
Pascarella, E. T. & Terenzini, P. T. (1991). How college affects students: Findings and
insights from twenty years of research. San Francisco: Jossey-Bass, Inc.
Pritchard, M.E. & Wilson, G.S. (2003). Using emotional and social factors to predict student
success. Journal of College Student Development, 44 (1), 18-28.
Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton
University Press.
Russell, R. K. & Petrie, T. P. (1992). Academic adjustment of college students: Assessment
and Counseling. In. S.D. Brown and R.W. Lent (Eds.) Handbook of Counseling Psychology,
New York: Wiley.
Tinto, V. (1992). Leaving college: Rethinking the causes and cures of student attrition 2nd
edition. Chicago: University of Chicago Press.
Van Rooijen, L. (1986). Advanced students' adaptation to college. Higher Education.
15(3/4), 197-209. New York: Springer.
VanZile-Tmsen, C. (2001). The predictive power of expectancy of success and task value for
college students’ self-regulated strategy use. Journal of College Student Development, 42 (3),
233-241.
Weinstein, C.E. & Palmer, D.R. (2002). User’s manual: Learning and study strategies
inventory 2nd edition, H&H Publishing Company, Inc.
Zajacova, A., Lynch, S. M. & Espenshade, T. J. (2005). Self-efficacy, stress and academic
success in college. Research in Higher Education, 46 (6), 677-706.
22
Table 1
Items of the Six-CLEI Scales
1. Academic Self-Efficacy (ASE Scale) (14 items)
*
*
*
*
*
23. I believe that I have the ability to complete college.
26. I have goals that I want to achieve by being in college.
43. I have high academic expectations of myself.
24. I believe it is possible for me to make good grades.
28. I turn in assignments only partially completed.
42. I doubt that I can make the effort to finish college.
50. I am determined to do what it will take in order to succeed with my goals.
5. I do not turn in assignments.
21. My family cares how I do academically.
38. Family members criticize me because I am not a great student.
4. I am aware of the assignments that are due in the next week.
47. Gaining knowledge is important to me.
49. I question why I need a degree for the career I want to pursue.
20. People in my community value a college education.
2. Orga ni zation and Attention to Study (OAS Scale) (8 items)
*
*
*
*
2.
30.
1.
31.
51.
48.
25.
6.
I organize my time so that I have plenty of time to study.
I make study goals and keep up with them.
I wait to study until the night before the exam.
I break big assignments into manageable pieces.
I cannot get into studying even if there is nothing else to do.
I find myself daydreaming when I study.
I find my attention wandering in class
I organize class information in a way that helps me retain and apply it later.
3. Stress and Time Press (STP Scale) (6 Items)
* 36. I feel there are so many things to get done each week that I am stressed.
* 13. I have symptoms of stress from all of the pressure I have been under since coming to
college.
* 3. I do not seem to have time to get everything done that I need to do.
* 32. It seems as though I am playing catch-up.
* 37. My living situation distracts me from my studies.
7. I plan in advance to prevent becoming overwhelmed with assignments at the last minute.
23
4. Involvement with Co llege Activity (ICA Scale) (9 Items)
9. I participate in social activities on campus.
11. I belong to an organized club on campus.
34. I attend events such as concerts, plays, speakers, or athletic contests as a part of the college
experience.
29. I know someone with whom I can study.
40. I have friends here at school.
10. I belong to a study group.
15. I consider college to be a great time in my life.
41. My friends have good study habits.
17. I enjoy being a student here.
5. Emotional Satisfaction (ES Scale) (7 items)
14.
39.
* 18.
27.
* 12.
19.
* 16.
I like my courses.
My instructors show interest in me.
I hate school, but I know I have to do it.
I see connections between my classes and my career goals.
I am discouraged with how I am treated by my instructors.
I can talk with people who provide encouragement to me about what I am learning.
I become overwhelmed when I think of my assigned class requirements.
6. Class Communication (C C Scale) (6 Items)
* 8.
33.
* 46.
* 35.
* 44.
* 22.
I avoid speaking in class.
I ask questions in class.
I cannot seem to express my ideas on paper very well.
I avoid classes in which participation is required.
I dread the thought of getting test results in certain classes.
I find it difficult to get the assistance I need for my academic success.
“Experi mental Item”
45. I can make connections between what I learn in class and my plans for a career.
Note. *, Indicates items in negative continuum of which the raw scores need to be transformed to
reverse scores: 1 to 5, 2 to 4, 4 to 2, and 5 to 1.
24
Table 2
Means and Standard Deviations for the CLEI Scores for a Normative Sample of College Undergraduates by Gender
Gender
Male
CLEI Scal es
Femal e
Total
1. Academic Self-Efficacy
Mean
SD
4.25
.51
4.54
.43
4.40
.50
2. Organization and Attention to Study
Mean
SD
3.03
.60
3.18
.57
3.10
.59
3. Stress and Time Press
Mean
SD
3.16
.68
2.98
.67
3.07
.68
4. Involvement with College Activity
Mean
SD
3.41
.67
3.51
.60
3.46
.64
5. Emotional Satisfaction
Mean
SD
3.55
.55
3.73
.55
3.64
.56
6. Class Communication
Mean
SD
3.41
.64
3.44
.59
3.42
.62
** Global Score
Mean
SD
3.47
.42
3.57
.39
3.52
.41
Note. N=879 (Male n=439, Female n=440). Means and standard deviations are weighted by gender ratios (.512 for male student
sample and .488 for female student sample) and by class level proportions in the larger university population (.242 for First Year,
.205 for Sophomore, .225 for Junior, and .327 for Senior) in the larger university population.
25
Table 3
Means and Standard Deviations for the CLEI Scores for a Normative Sample of College Undergraduates by Year in College
CLEI Scales
First
Year
Year in School
Sopho mor
Junior
e
Senior
Total
1. Academic Self-Efficacy
Mean
SD
4.26
.53
4.34
.55
4.54
.39
4.44
.47
4.40
.50
2. Organization and Attention
to Study
Mean
SD
3.08
.60
3.09
.58
3.13
.63
3.10
.56
3.10
.59
3. Stress and Time Press
Mean
SD
3.17
.71
3.15
.64
2.96
.69
3.02
.68
3.07
.68
4. Involvement with College
Activity
Mean
SD
3.32
.70
3.61
.66
3.55
.55
3.40
.60
3.46
.64
5. Emotional Satisfaction
Mean
SD
3.54
.57
3.56
.62
3.77
.49
3.69
.52
3.64
.56
Mean
SD
3.27
.62
3.38
.68
3.49
.55
3.52
.59
3.42
.62
Mean
SD
3.44
.44
3.52
.44
3.58
.35
3.53
.39
3.52
.41
6. Class Communication
** Global Score
Note. N=879 (First Year n=211, Sophomore n=186, Junior n=193, Senior n=290).
Means and standard deviations are weighted by gender ratios (.512 for male student sample and .488 for female student sample) and by class level proportions
(.242 for First Year, .205 for Sophomore, .225 for Junior, and .327 for Senior) in the larger university population.
26
Table 4
Item Statistics for the Academic Self-Efficacy Scale
Item
Number
23
Item
Mean
4.64
Item
Std. Deviation
.697
Item-Total
Correlation
.690
Cronbach’s Alpha
if Item Deleted
.855
26
4.55
.707
.623
.858
43
4.22
.850
.660
.855
24
4.42
.808
.631
.857
28
4.37
.797
.499
.864
42
4.65
.745
.595
.859
50
4.30
.801
.626
.857
5
4.45
.833
.483
.864
21
4.66
.688
.424
.867
38
4.32
1.006
.505
.864
4
4.07
.965
.392
.871
47
4.34
.790
.522
.862
49
4.20
1.074
.479
.867
20
4.39
.811
.426
.867
Table 5
Item Statistics for the Organization and Attention to Study Scale
Item
Number
2
Item
Mean
3.21
Item
Std. Deviation
.888
Item-Total
Correlation
.593
Cronbach’s Alpha
if Item Deleted
.777
30
3.07
.965
.572
.779
1
2.77
.946
.481
.793
31
3.19
.936
.512
.788
51
3.18
1.015
.610
.773
48
2.82
.897
.546
.783
25
2.73
.879
.482
.793
6
3.59
.864
.378
.806
27
Table 6
Item Statistics for the Stress and Time Press Scale
Item
Number
36
Item
Mean
2.76
Item
Std. Deviation
1.059
Item-Total
Correlation
.660
Cronbach’s Alpha
if Item Deleted
.694
13
3.04
1.102
.599
.710
3
3.12
1.060
.546
.725
32
3.01
1.023
.545
.726
37
3.19
1.194
.408
.764
7
3.14
.943
.329
.775
Table 7
Item Statistics for the Involvement with College Activity Scale
Item
Number
9
Item
Mean
3.28
Item
Std. Deviation
1.122
Item-Total
Correlation
.645
Cronbach’s Alpha
if Item Deleted
.768
11
3.03
1.627
.559
.786
34
2.74
1.152
.529
.783
29
3.49
1.123
.505
.787
40
4.45
.888
.542
.784
10
1.98
1.109
.381
.802
15
4.03
.925
.514
.787
41
3.54
.905
.395
.799
17
4.12
.926
.516
.787
28
Table 8
Item Statistics for the Emotional Satisfaction Scale
Item
Number
14
Item
Mean
3.52
Item
Std. Deviation
.804
Item-Total
Correlation
.531
Cronbach’s Alpha
if Item Deleted
.674
39
3.30
.899
.494
.679
18
3.57
1.142
.504
.675
27
3.72
.985
.444
.690
12
4.06
.900
.327
.717
19
3.73
1.017
.385
.706
16
3.41
.824
.392
.702
Table 9
Item Statistics for the Class Communication Scale
Item
Number
8
Item
Mean
3.07
Item
Std. Deviation
1.089
Item-Total
Correlation
.532
Cronbach’s Alpha
if Item Deleted
.592
33
2.90
1.027
.410
.637
46
3.58
1.047
.337
.662
35
4.07
.935
.416
.636
44
2.72
1.031
.350
.657
22
3.74
1.028
.410
.637
29
Table 10
Scale Statistics: Means, Standard Deviations, Intercorrelations, and
Reliability Coefficients with 95% Confidence Intervals for the Six Scales of the CLEI
N of
Items
M
SD
1.
ASE
Factor
1. ASE
2.
OSA
3. STP
4. ICA
5. ES
6. CC
14
8
6
9
7
6
Reliability
Coefficient
Intercorrelation Coefficient
2.OS
A
3.
STP
4.ICA
5.ES
4.40
0.51
---
3.06
0.60
.45**
---
3.04
0.73
.26**
.41**
---
3.40
0.69
.43**
.35**
.18**
---
3.62
0.58
.56**
.51**
.38**
.38**
---
3.34
0.64
.39**
.36**
.32**
.35**
.47*
*
6.C
C
(95% CI)
a
.87 (.855,
.885)
.81 a (.784,
.831)
.77 a (.738,
.796)
.81 a (.782,
.829)
.72 (.689,
.757)
.68 (.637,
.717)
---
Note. N=597.
ASE=Academic Self-Efficacy, OSA=Organization and Attention to Study, STP=Stress and Time
Press,
ICA=Involvement with College Activity, ES=Emotional Satisfaction, CC=Class Communication.
a
Cronbach’s alpha was significantly greater than hypothesized value of .70 (p<.05).
**p<.01.
Table 11
Overall Fit Indices of the Confirmatory Factor Analysis on the Six Scales of the CLEI
Independence
Model
Fit Statistics
22,447.46**
1.898.62**
df
NFI
GFI
AGFI
RMSR
RMSEA
CFI
1,225
1,118
.92
.92
.90
.08
.05
.96
Note. N=292.
Fit indices include chi-square (Satorra-Bentler Scaled Chi-Square), NFI=normed fit index,
GFI=goodness-of-fit index, AGFI= adjusted goodness-of-fit index, RMSR=root mean square
residual , RMSEA=root mean square error of approximation residual, and CFI=comparative fit
index.
**p<.01.
Table 12
30
Relationship between the CLEI and Validation Instruments
CLEI Scales
Validation Instruments
Pearson r
1. Academic Self-Efficacy
LASSI (Motivation Scale)
Rosenberg Self-Esteem Scale
.46**
.45**
2. Organization and Attention to Study
LASSI (Concentration Scale)
LASSI (Self-Testing Scale)
.71**
.46**
3. Stress and Time Press
LASSI (Time Management Scale)
.44**
4. Involvement with College Activity
Crombag College Adaptation
Questionnaire
.31**
5. Emotional Satisfaction
LASSI (Attitude Scale)
.50**
6. Class Communication
Student Propensity to Ask Questions
.53**
Note. N=160.
LASSI=Learning and Study Strategies Inventory
**p<.01.
31