Addendum Exhibit 2.6.a Data Driven Improvements in Advance Programs
The University of Texas at Brownsville
1. Master of Educational Technology
The Educational Technology faculty continually evaluates their courses to: (1) strengthen the program, (2) improve
candidate performance, and, (3) respond quickly to the changing needs of the educational technology practitioner.
The pursuit for academic excellence began long before the College of Education made the commitment to seek
NCATE accreditation. For example, in 2005, the Educational Technology program adopted the electronic
portfolio, or e-portfolio, as its final form of assessment in lieu of the Comprehensive Exam. In order to provide a
framework for the e-portfolio, the Educational Technology (EDTC) Professional Responsibilities were developed.
These professional responsibilities were based on the AECT standards but included additional standards to
specifically target the areas of leadership and professional development, collaboration, and research. Since 2005,
the Educational Technology faculty have been working to align the educational technology courses and their
assessments with the EDTC Professional Responsibilities and AECT standards. To further this endeavor, in 2008,
the faculty commissioned an external program review through the University of Texas TeleCampus.
Since 2009, the Educational Technology faculty have analyzed and addressed all of the program review
recommendations. Recommendations 2, 3, and 5 were immediately adopted and incorporated into program
revisions in preparation for AECT/NCATE accreditation.
Recommendation #2 addresses the areas of Content Knowledge, Professional and Pedagogical Knowledge, Skill,
and Dispositions, and Student Learning. To introduce students to a variety of instructional design approaches and
incorporate different design models throughout the program, a Cooperative Project was added to EDTC 6321 –
Instructional Design to get students, working in groups of 3-5, to conduct research on one or instructional design
models to analyze their appropriateness for use in an instructional design scenario. Furthermore, in EDTC 6323 –
Multimedia Hypermedia, students used elements of the Dick and Carey and ADDIE design models to design and
build their interactive multimedia projects. In EDTC 6332 – Practicum in Educational Technology, elements of
Kirpatrick’s Four Levels of Evaluation, were incorporated into the evaluation component of the practicum project.
In September 2010, the Educational Technology faculty submitted a curriculum action request for the development
of a new course in advanced instructional design. The new course will be added to the Educational Technology
core and is being designed to specifically address many of the concerns raised in Recommendation 2.
One West University Boulevard • Brownsville, Texas 78520 • utb.edu
Recommendation #3 addresses the areas of Content Knowledge and Professional and Pedagogical Knowledge,
Skill, and Dispositions. To provide more opportunities for students to engage in various forms of evaluation, all of
the core courses were revised to include some elements of evaluation of the learning objects that were developed
by students. In EDTC 6321 – Instructional Design, students learned how to conduct three forms of formative
evaluation of their instructional units: subject matter expert review, one-to-one evaluation, and small group
evaluation. Furthermore, after each stage of the instructional design process, students subjected drafts of their
projects to a peer review and made revisions accordingly. A second round of revisions was conducted after
students received instructor feedback. In EDTC 6323 –Multimedia/Hypermedia, students learned how to conduct
summative evaluations of their interactive multimedia objects, using actual members of their target audience. In
EDTC 6325 – Educational Telecommunications, students evaluated the graphical user interface they designed for
their E-Learning Modules to address usability issues and learner preferences. In the Cooperative Project of the
same course, students researched, analyzed, and evaluated the potential of 3D virtual worlds for
instructional/training applications. The output from this study is a report on the feasibility and appropriateness of
using 3D virtual words in K-12 and higher education. As previously described, in EDTC 6332 – Practicum in
Educational Technology, the practicum project was expanded in order to get students to use summative and
formative evaluations of an instructional solution designed to respond to a real-world problem. To address the
issues of transfer of learning, return on investment, elements of Kirpatrick’s Four Levels of Evaluation were
incorporated into the evaluation component of the practicum project. Lastly, in every core course where an
instructional object is created, some form of formative evaluation will be conducted, using actual members of the
target populations.
Recommendation #5 addresses the areas of Content Knowledge, Professional and Pedagogical Knowledge, Skill,
and Dispositions, and Student Learning. To provide students with more opportunities to engage in meaningful
research, literature review activities have been added to projects in several core courses. For example, the
cooperative projects in EDTC 6320 and 6325 require students to conduct research on an instructional technology
strategy or tool to evaluate their potential for educational/instructional uses. In each case, students must justify the
selection and use of their chosen instructional technology strategies or resources. In EDTC 6321 and EDTC 6332,
as part of the instructional design process, students must also justify the selection of instructional strategies and
resources (multimedia presentations, podcasts, videos, animations, etc.) that will be incorporated into their
instructional products. These justifications must be grounded based on best practices
2. Master of Education-Educational Leadership-School Building Level
Changes to the EDLR Program have been made in several phases:
Fall 2011
1. Beginning Fall 2011, the EDLR Program as a whole worked to align its program with ELCC standards.
2. The EDLR program aligned its principal internship with ELCC standard 7. This was accomplished by moving
the number of required internship hours from 160 as required in the state of Texas to 240 hours to meet standard 7.
Additionally, the internship began requiring pre-internship field hours to be earned throughout the program across
a variety of settings. Experiences for the capstone experience in the internship have been aligned to the ELCC
standards.
3. The program rewrote the course descriptions for EDLR 6394: Curriculum Leadership and EDLR 6386: Pupil
Personnel Services and EDLR 6389: School Finance to bring them into alignment with the ELCC standards. These
changes were taken before the University’s Graduate Committee in Fall of 2011 and approved.
4. The program has decided that all students will take EDLR 6386 to ensure that ELCC standard 4 is addressed by
our program curriculum. Previously the course was an elective for principal certification only students.
5. E portfolio content for students in the Master’s Degree Program will be aligned with both the TeXes
competencies and the ELCC standards.
6. An assessment calendar was created to illustrate the assessment cycle for data collection and analysis.
3. Master of Education-Educational Leadership-School District Leadership Level
Changes to the EDLR Program have been made in several phases:
Fall 2011
1. Beginning Fall 2011, the EDLR Program as a whole worked to align its program with ELCC standards.
2. The EDLR program aligned its superintendency internship with ELCC standard 7. This was accomplished by
moving the number of required internship hours from 160 as required in the state of Texas to 240 hours to meet
standard 7. Additionally, the internship began requiring pre internship field hours to be earned throughout the
program across a variety of settings. Experiences for the capstone experience in the internship have been aligned to
the ELCC standards.
3. An assessment calendar was created to illustrate the assessment cycle for data collection and analysis.
4. Master Advanced Special Education
Standard 1: To ensure preponderance of evidence of our candidates’ meeting this standard. We strengthened Key
Assessment 2: the eportfolio review and replaced Item 2 with requiring candidates’ expanded thinking as a
pedagogical leader based on special education case scenarios to emphasize Standard 1 candidate performance
outcomes. In addition, data from newly implemented assessments are provided below:
Assessment 1: Comprehensive Core Content Exam has been implemented in early fall, 2013.
Assessment 2: Rubric further developed and partial data collected.
5. Master of Special Education-Board Certified Behavior Analyst
The Special Education Program faculty meets at least once each semester to review candidate data across all
programs. The goals of the meetings are: 1) to determine any specific problems with Key Assessment instruments
and/or rubrics in order to assure valid and reliable assessment, 2) to determine any program weaknesses that
indicate standards are not met or not addressed sufficiently within the programs, and, 3) to identify any specific
students who are not providing evidence of mastery of content or skills in specific standards and plan specific
remediation for each individual student.
Data Analysis for Years 1 and 2
Data collected for all seven Key Assessments are displayed below according to the criteria established by the
faculty. The criteria for the Comprehensive Core Content Exam are displayed by the aggregate percentage of
correct items for each standard. This new assessment indicates that two of the standards fell below the passing
standard of 75%. Item analysis of the two scales revealed that some items may not be representative of the content
within the standard. Additionally, the curriculum is now being examined to determine if these two standards are
adequately represented within the first year of study. Overall, other standard scales meet the criteria set by the
faculty. One standard, Standard 2, was not acceptable in the Key Assessment 4 Instrument evaluating the
Practicum I (2013) students, however, upon completion of Practicum II (2013), all students were rated as “Target”
or “Distinguished” on this standard.
6. Master of Special Education-Educational Diagnostician
Remediation plans for candidates who fall below target criteria will be implemented at strategic places in their
program, including after receiving results for KA2 (after taking the 18 hours of core special education
coursework). In addition, all candidates will undergo review at the 12 hour mark of their program for the purpose
of providing instructive feedback. Utilizing the college’s new data base, TK-20, faculty will be able to more easily
store candidate data and recognize struggling candidates for the purpose of providing unified faculty feedback and
support. Reviewing candidates’ performance at these two key points in the candidates’ programs of study will
provide appropriate opportunities for struggling candidates to receive remediation and/or make decisions about
their future in the program. In addition to formal benchmark points of evaluation, faculty meet regularly
throughout the semester and discuss individual candidates who are struggling with coursework, communicating
concerns and offering help to these candidates.
7. Master’s in Bilingual Education
Content Knowledge - All the key assessments evaluate candidates’ content knowledge: 1)
Comprehensive exam, 2) Strategies portfolio, 3) Case study, 4) Paper on the systematic nature of
language, 5) Key paper on assessment, 6) Lesson demonstration, 7) Philosophy and advocacy refection
paper. These key assessments were created when the program was revised in an attempt to improve the
overall quality and academic rigor of the program. They key assessments help faculty evaluate their
instruction, the texts and articles they choose to support content learning, and the assignments they give
candidates to help them understand and internalize theory and research in the field. Moreover, the key
assessments help faculty evaluate candidate content knowledge in an ongoing format throughout the
program. The faculty disaggregates data by off campus cohorts.
Faculty reflections show that
candidates in the different contexts respond differently to assignments and assessments. As a result,
faculty adapt to the needs of candidates in different contexts.
The assessment results showed that the majority of the candidates met the standards. Nevertheless,
program faculty has identified specific areas of concern. For example, it was noted that candidates have
a difficult time answering the comprehensive exam questions synthesizing the knowledge acquired and
applying it to analytical situations. Therefore, faculty will incorporate midterm exams into several
courses in which the questions given will resemble the nature of questions that students will encounter
in their comprehensive exams. This will allow candidates to practice synthesizing content knowledge
and elaborating on the instruction implications of the concepts acquired.
Professional and Pedagogical Knowledge - The following assessments specifically evaluate candidates’
professional and pedagogical knowledge: Assessment #3: Case Study; Assessment #6: Lesson demonstration;
and Assessment #7: Philosophy and Advocacy Reflection Paper. All three of these assessments require that
candidates translate theory and research into practice.
1) The Case Study (Assessment #3) requires that candidates analyze the past and present context of an ELL
with whom they are working to identify the factors that influence second language acquisition and academic
achievement. In addition, candidates make recommendations for the future instruction of the case study student
which requires candidates’ application of professional and pedagogical knowledge.
2) The Lesson Demonstration (Assessment #6) also requires candidates to apply their professional and
pedagogical knowledge when they identify a specific grade level and grade level standards to design a lesson in
which they demonstrate their ability to design instruction according to research-based best practices for
emergent bilinguals to promote a supportive classroom environment for bilingual students.
3) Finally, in the Philosophy and Advocacy Reflection Paper (Assessment #7) candidates apply their
professional and pedagogical knowledge on their role in advocating for students and their families assuring that
schools meet their academic and social needs.
The scoring rubrics used for the above-mentioned assessments are specifically designed to show how
candidates are applying the knowledge of research-based best practices. Candidates’ performance on these
three assignments indicates that the standards that require attention are 2.b Cultural Groups and Identity, and
5.c Professional Development and Collaboration. One major change in the program that has been discussed by
the faculty is the design of a new key assessment that will specifically address standards 2.a Nature and Role of
Culture and 2.b Cultural Groups and Identity. This new assessment will be implemented in EDFR 6388 –
Intercultural Foundations of Education, which is a core course in the master’s program and required of all
candidates. Regarding Standard 5.c Professional Development and Collaboration, faculty will introduce and
reinforce the importance of collaboration among staff to better meet the needs of emergent bilingual students
throughout coursework, even when the standard is not being formally assessed, in order to give candidates
increased opportunities to develop competency in the standard and perform satisfactorily once it is assessed.
Student Learning - The Strategies Portfolio (Assessment #2) is used to assess student learning. This
assessment has been modified by the professors to focus on candidates’ effect on ELLs’ learning through the
implementation of strategies while working with students in the classroom and collecting artifacts as evidence
of their field work. Candidates are expected to apply specific strategies that allow students to access content
and develop academic language in content area courses. Candidates collect student work samples and
additional evidence of the application of the strategies. Additional evidence can include, but is not limited to,
materials used in delivery of strategy, pictures of students at work, etc.
Candidates are also expected to evaluate the effectiveness of the strategies by analyzing student performance
and artifacts. Because these strategies and reflective analyses are compiled in a portfolio over the course of a
semester, the portfolio also allows candidates and professors to monitor candidates’ growth as the semester
progresses.
The scoring rubric used to evaluate candidates on the above-mentioned assessment is specifically designed to
show how candidates are applying the knowledge of research-based best practices. The assessment results
showed that the majority of the candidates met the standards. Nevertheless, the standard in which candidates
score the lowest is 3a Candidates know, understand, and apply concepts, research, and best practices to plan
classroom instruction in a supportive learning environment for bilingual students. Based on analysis of the
data and reflecting on the assessment results, faculty have decided to provide extended modeling for candidates
on how to plan effective lessons for emergent bilingual learners drawing on best practices and research.
Instructors plan to embed time to demonstrate effective planning of instruction, and to provide extended time
for candidates to work collaboratively, with the support of the professor, in joint planning of instruction. In
other words, faculty plan to incorporate the gradual release of responsibility model to scaffold instruction for
candidates and maximize the opportunities for success in the attainment of standard 3.a.
Overall, each program standard was addressed by several key assessments. Candidates performed satisfactorily
in most key assessments. Therefore, when all assessments are considered as a whole, every program standard is
met by the great majority of candidates. The program faculty plans to engage in regular conversations to
disaggregate data in an effort to ensure continuous improvement.
8. Masters in Counseling and Guidance
All current counseling and guidance faculty continuously review data to identify strengths and areas in need of
improvement. Given that students’ scores were below target on the research part of the comprehensive
examination, program faculty implemented changes such as including a different textbook in the Research
Methods course. All program faculty members also emphasize the importance of research in coursework
throughout the curriculum. As another improvement to increase students’ scores on research, faculty in other
coursework will include an article analysis assignment in order to require students to not only analyze articles but
also continue to think about research methods. As an improvement to the practice section, program faculty added a
textbook to Practicum and Internship coursework to help students learn about the counseling profession and how to
improve counseling skills. Program faculty also added a new requirement to the Group Counseling course in which
students facilitate a group with adults or children. This new task encourages students to develop counseling skills
prior to enrollment in practicum and internship. Finally, as an improvement to ethics, program faculty provided an
ethics seminar to help students remain current with ethical developments in the counseling profession.
9. Masters of Curriculum and Instruction
The process of NCATE program review for the M.Ed. Curriculum and Instruction has been an informative process.
It began with individuals from each of the eight respective programs meeting to review each program’s current
process and learning outcomes. As each program was further explored through the lens of NCATE it became clear
that current standards were inadequate. One of the first major revisions was to develop new curriculum and
instruction standards for the M.Ed. program. The standards were adopted for both the M.Ed. and the Ph.D./Ed.D.
programs to maintain rigor and continuity. The following standards were adopted:
· Standard 1: Knowledge of Curriculum. Program completers will demonstrate advanced ability to design,
implement, and evaluate curriculum that promotes student learning.
· Standard 2: Knowledge of Instruction. Program completers will demonstrate advanced ability to plan,
implement, and evaluate instruction to facilitate student learning.
· Standard 3: Knowledge of Content. Program completers will demonstrate advanced depth and breadth of
knowledge and skills in the academic discipline and pedagogy.
· Standard 4: Knowledge of Students. Program completers will demonstrate advanced knowledge of the student
as influenced by cognitive, physical, emotional, social, cultural, environmental, and economic factors.
· Standard 5: Knowledge of Inquiry. Program completers will demonstrate ability to use research to promote
student learning and to contribute to the teaching profession.
· Standard 6: Knowledge of Assessment. Program completers will demonstrate advanced knowledge of
assessment and the ability to use multiple sources of assessment for maximizing student learning.
· Standard 7: Professional Practices. Program completers will demonstrate high standards for professional
practice.
· Standard 8: Technology Integration. Program completers will demonstrate ability to integrate current
technology into instruction and communications/collaboration activities where appropriate.
Adoption of these standards has resulted in changes to the curriculum, as well as to the six targeted key
assessments. Data from the six key assessments have been reviewed and recommendations for improvement to the
masters program have been identified.
1. Addition of course number EDFR 6388 Socio-cultural Foundations of Education. In this course students analyze
socio-cultural forces which shape the direction of American education with emphasis on education in
philosophical, sociological, psychological and anthropological context, and the intercultural factors in society
which affect public schools and influence learning and acquiring skills important to educational growth and selffulfillment are stressed.
2. Professional Portfolio. All students will be required to purchase instructional materials including (but not limited
to) Tk20 which is an online academic electronic workbook to be used in building their professional portfolio
designed to provide evidence of mastery of class and state/professional standards.
Additional information
10. Doctoral degree in Education
The NCATE program review of the Doctor of Education (Ed.D.) in Curriculum and Instruction has been an
illuminating process, which has resulted in many recommendations for improvement, several of which have
already been implemented. The process started with a review of the doctoral program’s current student learning
outcomes. When seen through the lens of an NCATE program review, it became immediately evident that the
existing standards were inadequate. Therefore, one of the first major revisions was the development of new
curriculum and instruction standards for the doctoral program. The following standards were adopted:
Standard 1: Knowledge of Curriculum. Program completers will demonstrate advanced ability to synthesize indepth knowledge of major theories, philosophies, and current issues in curriculum and their implications for
practice; as well as articulate the contribution their own inquiry makes to scholarship and practices relevant to
curriculum
Standard 2: Knowledge of Instruction. Program completers will demonstrate advanced ability to apply research
and best practices to lead; plan; implement; and evaluate instruction; as well as articulate the contribution their
own inquiry makes to the scholarship and practices relevant to instruction
Standard 3: Knowledge of Content. Program completers will demonstrate advanced depth and breadth of
specialization-specific knowledge and skills, and the ability to conduct research appropriate to their specialization
Standard 4: Knowledge of Students. Program completers will demonstrate advanced knowledge of the
sociocultural, psychological and developmental dimensions of learning, and the implications of these dimensions
for teaching, learning, leading, and conducting research.
Standard 5: Knowledge of Inquiry. Program completers will demonstrate advanced knowledge of approaches to
and the ethical dimensions of inquiry, as well as specific skills related to conducting research relevant to
curriculum and instruction that advances the field of education.
Standard 6: Knowledge of Assessment. Program completers will demonstrate advanced knowledge of the
methods, issues, and ethical dimensions of assessment, as well as an understanding of its applications to inquiry
and practice.
Standard 7: Professional Practices. Program completers will cultivate dispositions that will enable them to
meaningfully and ethically participate in communities of practice, as well as professional and scholarly
organizations and networks.
Standard 8: Technology Integration. Program completers will demonstrate the ability to think critically about
issues related to technology and the implications for teaching, learning, and equity, as well as develop the
technological competencies to function effectively as a learner, researcher, and instructional leader.
Adoption of these standards has resulted in changes to the curriculum, as well as to the six, targeted key
assessments. Consequently, due to curriculum realignment, modification or revision of key assessments, and
course sequencing (some courses are offered only once per year), some key assessments have only one semester of
data available. The key assessment for one of the research courses was completely redesigned and data collection
for that course began in Fall 2012 semester. Data from the six key assessments have been carefully reviewed, and a
rigorous internal review has been conducted, resulting in several recommendations for improvement to the doctoral
program. In particular, the four areas described below have been targeted for improvement.
The first area targeted for improvement is the doctoral dissertation proposal. Although 100% of the candidates
achieved the standards measured by this key assessment, further analysis regarding those candidates who scored
Met with Weakness was merited. As the key assessment data reveals, several candidates struggled with review of
literature, methodology, and scholarly writing. Therefore, in order to increase candidates’ attainment of a Target
score for the dissertation proposal defense, several measures have been instituted over the past year:
1. A doctoral student handbook was developed and implemented in the Fall 2011 by the new coordinator of the
doctoral program. In addition to procedural information, the doctoral handbook includes an overview of each
assessment, including the proposal defense. The manual is available in hard copy and online on the University
website.
2. A new doctoral dissertation proposal rubric, aligned to the new C&I standards, was developed and added to the
doctoral student manual to help candidates know and work towards specifically developing the knowledge, skills,
and dispositions required by each standard.
3. Increased opportunities for students to develop their writing and research skills have been incorporated into the
core and research courses. In EDCI 8300 – Advanced Research Methods in Education, an enhanced Research
Literature Review project (see Key Assessment #1) was developed to strengthen doctoral students’ research and
writing skills. To further enhance their scholarly writing ability and research skills, a Curriculum Conference
Proposal (see Key Assessment #2) paper was developed for
EDCI 8320 - Advanced Curriculum: Instructional Design and Development to get doctoral students to use
guidelines from the Curriculum Studies Division (B) of the American Educational Research
Association to develop a conference proposal based on appropriate forms of curriculum inquiry for submission to a
national educational conference focusing on curriculum. These two major key assessments should help to
strengthen doctoral students’ scholarly writing and research skills in preparation for their doctoral dissertation
proposals.
The second major area targeted for improvement was the Doctoral Comprehensive Exam. While data from this key
assessment indicated that 100% of test takers passed all three sections of the doctoral comprehensive exam, several
candidates (71%) met the research (inquiry) question with weakness. One possible explanation for the low
performance on the research question is that the question writers and raters were not the same individuals. Based
on these results, we recognize the need for writers and raters to be familiar with the content, students, and
expectations. Furthermore, in earlier cohorts, faculty teaching the research and C&I core courses tended to lean
more heavily toward qualitative methods, resulting in a majority of students preferring qualitative over quantitative
methods in their research papers and dissertations. This imbalance has since been addressed with the hiring of
additional research faculty, who have a strong background in quantitative methodologies. Recent and future
cohorts will receive a more balanced coverage of qualitative and quantitative methods through their research and
content courses. It is expected that as the improvements take effect, the performance on the research (inquiry)
question will improve.
The third area targeted for improvement in the doctoral program is the doctoral dissertation. Although key
assessment data obtained on the doctoral dissertation reflected a Target rate of 100% on the standards addressed in
this assessment, it has been observed that a significant majority of doctoral dissertations submitted over the past 3
years have relied heavily or almost exclusively on qualitative methods. While qualitative research studies are
appropriate for educational research, the lack of quantitative studies was a concern. As indicated above with the
issues regarding the research question on the comprehensive exam, steps have been taken to provide a more
balanced coverage of quantitative and qualitative methods through the research courses as well as in the core
courses. Furthermore, additional doctoral faculty with experience in quantitative research methods have been hired
to teach the research and curriculum courses.
These changes to the faculty line up should provide a much more balanced coverage of research and prepare
doctoral students to use the most appropriate research methodologies based on their research topics. The addition
of two new doctoral specializations, in Educational Technology and Higher
Education Teaching should also help to provide a balance between qualitative and quantitative research studies.
Last, but not least, the new doctoral student handbook will provide students with guidance in the development of
their doctoral dissertation topics. Resources for selecting and using appropriate quantitative and qualitative
research methods will be included in doctoral student handbook as well as a detailed doctoral dissertation rubric,
aligned to the new C&I standards, to assist doctoral students and faculty assessing the quality of the dissertations
being submitted for review.
A fourth area that was raised as a result of a rigorous internal review process, involves the re-evaluation of the Key
Assessment Rubrics. Internal reviewers recommended that for the next assessment cycle, the doctoral program
faculty and coordinator evaluate the benefits of revising the Key Assessment Rubrics so that that standards are
assessed using one portion of the scoring rubric, thereby allow for individual assessment of the different standards.
Doctoral program faculty and coordinator will collaborate throughout this assessment cycle to assess the feasibility
of this recommendation.
In summary, the four recommendations enumerated above will significantly improve the quality of the doctoral
program and serve as a launching board for additional improvements in the coming years. The adoption of new
program standards in curriculum and instruction that align with the College of
Education’s conceptual framework and reflect the highest NCATE standards have already resulted in significant
improvements to the program. The new C&I standards and COE conceptual framework will be included in all
doctoral course syllabi. All course content will be aligned to these standards and all key assessments will monitor
student performance on the standards through newly developed rubrics. The three, targeted areas for improvement
are congruent with the goals and direction the College of Education faculty have set for the future of the program.