1
Doctoral Comprehensive Examination
Policies and Procedures for the Portfolio Assessment System
Measurement, Statistics and Evaluation Program
Department of Human Development and Quantitative Methodology
University of Maryland, College Park
Approved unanimously by vote of the Measurement, Statistics and Evaluation Program on
May 16, 2016
SECTION 1 – The Portfolio Assessment System
This document describes the policies and procedures for the portfolio assessment system for the
doctoral comprehensive examination in the Measurement, Statistics and Evaluation (EDMS)
program in the Human Development and Quantitative Methodology (HDQM) department. The
statements in this document are subject to future revision but as of the above date represent the
official rules and regulations regarding the portfolio assessment system for EDMS.
Objectives of Portfolio Assessment System
The overarching goal of the portfolio system is to provide each student with an opportunity to
demonstrate his/her ability or readiness to successfully engage in professional activities
encountered in typical employment tracks for which the EDMS program is training graduates.
These include, but are not limited to, positions as a tenure-track academic, psychometrician,
research scientist, quantitative methodologist for an organization, and measurement specialist for
policy work in governmental agencies.
Passing the program portfolio assessment is required to advance to candidacy in the doctoral
program. Failure to pass this assessment leads to termination from the doctoral program. If the
student fails either of the two evaluation components during the second attempt, the student will
be terminated from the doctoral program.
Evaluation of Portfolio
Students are required to demonstrate competence for each of the components as described in
Section 2 of this document. This evaluation is implemented using holistic evaluations of the work
products submitted for each of the portfolio components. The evaluation of the portfolio involves
two steps:
1. Review of portfolio documents, and
2. Interview with student following satisfactory portfolio review.
2
Both of these steps in the evaluation are conducted by the Examination Committee of the EDMS
program. The examination committee includes the student’s advisor along with the other regular
members of the committee in the year in which the portfolio is submitted. In the case where the
advisor is already a regular member of the committee, the committee will select a fourth faculty
member from the EDMS program who will be added to the committee for that student. All voting
by the committee will be by majority rule (i.e., two out of three members of the examination
committee have to vote to “pass” the student); the student’s advisor does not have a formal vote
and is not allowed to be present during the voting process.
Each examination committee member will independently evaluate the portfolio. The committee
will subsequently discuss the quality of the components. Logistically, after an evaluation of the
portfolio document by the committee, and in the case of a majority vote to “pass” the student, the
student will then move on to the oral defense. If the vote is a “fail” the process starts a second and
final time. The student will be given clear instructions on how to re-submit the portfolio at a later
date, within the time line specified by the graduate school for advancement to candidacy.
During the oral examination, the student will be interviewed by the committee to evaluate his/her
competence in each of the targeted competency areas. This interview is not to last more than two
hours. During the interview, all members of the augmented examination committee may
participate regarding the portfolio, including any matter that, in the judgment of the committee, is
germane to the purpose for which the portfolio is submitted. The focus is to be on the content of
the submission, and on establishing the expertise of the student to engage in satisfactory work of
which each facet is indicative.
After the interview, the committee will take a second vote according to the majority rule. If this
vote results in a “pass” for the interview component of the evaluation, the student will have
“passed” the doctoral comprehensive examination. If the vote is a “fail” the process starts a
second and final time. The student will be given clear instructions on how to re-submit the
portfolio at a later date, within the time line specified by the graduate school for advancement to
candidacy, followed by a second interview at that time should the portfolio review be satisfactory.
The following actions remain the prerogative of the examination committee at all times:
1. The committee may ask for additional evidence documenting the claim for
primary responsibility for the work;
2. The committee may ask for clarification regarding the nature and extent of the
work;
3. The committee may request multiple clarifications for portfolio components;
4. Based on a student’s responses to questions during the oral examination component, the
committee reserves the right to ask for multiple follow-up tasks to be satisfactorily
completed addressing the committee’s concerns.
3
SECTION 2 - Composition of the Portfolio
The portfolio consists of a cover page and table of contents (front matter) and the following nine
components, which are described in more detail below:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Personal statement
Curriculum vitae
An internal, EDMS-generated program of study
Literature review / synthesis paper
Data analysis paper
Simulation paper
Research presentation
Critique of a paper or book
Research proposal
Importantly, the work that is submitted in fulfillment of these components has to be done
independently by the student, except in the accepted ways noted below. Prior to the formal
submission of the portfolio to the examination committee, the student’s advisor should work with
the student on compiling the portfolio to meet the expectations set out in this document.
In general, the context in which a student has acquired the requisite skills that underlie the
creation of the portfolio components is less important than his/her contribution to them. In other
words, a student can submit original or revised work that was done as part of a class project, an
independent study, or project work at his/her place of employment.
In cases of scholarly papers, it is permissible to have multiple authors on a paper; however, the
student must have had primary responsibility for the design, implementation, and write-up of the
project that is documented in the portfolio component. The advisor takes primary responsibility for
verifying with the student that the work on each component is theirs.
A single project may address more than one portfolio component, but the criteria for each
component must all be met by that single component. Students must provide adequate
justification when meeting the requirements stipulated for multiple components using one piece
of work.
The following describes the nine components of the portfolio submission and the dimensions
along they will be holistically evaluated.
Formatting Requirements
All of the following portfolio components (including any requested additional written supplements
from the committee) should adhere to the latest version of the style guide of the American
Psychological Association (APA) whenever possible; at the time of this writing, the current APA
style guide is version six. This includes pagination, the setting of margins, spacing for text, tables,
and figures, the formatting of reference lists, the use of headers, and other considerations in line
with typical expectations for professional publications in the field of measurement, statistics and
4
evaluation.
General Standards for Writing
Each of the documents submitted for a component of the portfolio needs to meet minimum
competency criteria, which are referred to as an “effective style of communication” in the
following. Students are encouraged to consult the APA style guide, professional standards, as well
as examples of peer-reviewed work in journals in measurement, statistics and evaluation for
descriptions and exemplars of what this means.
The following considerations are generally important for creating an effective style of
communication:
a)
b)
c)
d)
e)
f)
g)
h)
language is grammatically correct;
language is linguistically appropriate for formal academic American English;
text segments are locally cohesive and globally coherent;
the organization is logical and supports the general line of reasoning;
figures and tables are used only when necessary to support the general line of reasoning;
supporting evidence for each claim is provided when such claims are made;
work by other scholars is cited appropriately; and
there is no evidence of plagiarism of any kind.
What follows is a description of each of the portfolio components.
5
Cover Page and Table of Contents
The portfolio must include a cover page, for which a template is included at the end of this
document. The cover page must be signed by the student and the advisor, certifying that the
contents of the portfolio have been examined by the advisor and meet all portfolio requirements
regarding the student’s contribution to all work contained therein. The second page is a table of
contents (ToCs) which refers the reader of the portfolio to specific page numbers corresponding to
the beginning of each component.
I.
Personal Statement
The student needs to write a personal statement, usually not to exceed four double-spaced pages,
that includes the following elements:
a) a brief summary of his/her trajectory through the EDMS program and any relevant
professional experience in the field;
b) a brief description of his/her professional plans upon completion of the program;
c) a brief description of each portfolio component and the context in which it was
developed; and
d) a critical self-assessment of the perceived areas of strength and weakness relevant to
those targeted by the portfolio.
II.
Curriculum vitae (CV)
The student needs to include a current CV that is dated and signed. It must include his/her
educational post-secondary background, professional presentations and publications, awards and
citations, and other work that is relevant to the education received in the EDMS program and the
targeted career of the student.
III. Program of Study
The student needs to submit a current EDMS form for the approved program of doctoral study,
which needs to include the credit hours for each course as well as the grades received for each
course that has been completed. In the case of incomplete grades, a description of why the
incomplete was received and when this work will be completed needs to be provided. A template
for this form appears at the end of this document.
IV. Literature Review / Synthesis Paper
The student needs to submit a literature review / synthesis paper that focuses on a current issue in
measurement, statistics and evaluation. It needs to provide a clear and focused review of the stateof-the-art of the methodological work in the identified area, identify key themes of this work and
open research questions, and discuss areas for future research that are suggested by the gaps in
this work. It cannot be a simple annotated bibliography or a point-by-point summary of individual
sources; rather, it has to be a critical synthesis.
6
The review will typically be 20-30 double-spaced pages in length. It can be original or revised
work from a course, an independent study, a summer internship, or project work from a place of
employment.
In addition to demonstrating the skills necessary to locate relevant research, extract key pieces of
information, and synthesize these pieces into a coherent narrative, the paper needs to meet the
following criteria:
a)
b)
c)
d)
e)
f)
g)
the research questions that guided the review / synthesis are clearly articulated;
the review is objective;
the review is comprehensive, yet focused;
the methods for locating relevant resources are clearly articulated;
the ways in which information was extracted from the sources are clearly articulated;
the limitations of the literature are clearly and thoughtfully addressed;
the interpretations drawn from the synthesis are appropriate given the methodology and
limitations of the literature; and
h) the paper is written effectively for communicating these ideas.
V.
Data Analysis Paper
The student needs to submit an applied data analysis paper, which addresses a research question
of practical relevance for a clearly identified stakeholder group in a clearly identified context. The
analysis does not have to use the most complex / advanced / sophisticated methodology to which
the student has been exposed; rather, it should use a suite of data-analytic tools that provide a
comprehensive set of robust empirical evidence for the evidentiary narrative that the analysis is
designed to support.
Part of the task is cleaning the data and preparing it for analysis as well as organizing and
summarizing the results in a way that meets the goals set for the analysis and satisfies the
analytical objectives. The code for performing the analyses should be submitted as an appendix
as part of the paper.
The student must indicate the source of the real data. The data set may be supplied by a faculty
member, but must be in the form of original data from the field. The paper can be original or
revised work from a course, an independent study, a summer internship, or project work from a
place of employment.
In addition to demonstrating the skills necessary to identify, clean up, and analyze a data set with
appropriate measurement tools, the paper needs to meet the following criteria:
a) the research questions that guided analysis are clearly articulated and of practical
relevance;
b) the selection of the data is appropriate for answering the research questions, which
includes any exclusion rules for cleaning the data;
c) the preparation of the data is appropriate for the analyses that are conducted;
7
d) the data-analytic methods are appropriate for the structure of the data and for
answering the research questions;
e) the analyses are implemented appropriately;
f) the interpretations that are drawn from output are statistically accurate;
g) the assumptions of the methods and the robustness of the results are investigated
comprehensively, as appropriate;
h) the limitations of the work are clearly and thoughtfully addressed; and
i) the paper is written effectively for communicating these ideas.
VI. Simulation Paper
The student needs to submit a simulation paper that is 15-30 double-spaced pages in length in
which he/she addresses a set of research questions that are of relevance to the field of
measurement, statistics, or evaluation with a simulation study.
The work needs to demonstrate that the student is capable of performing original programming in
a commonly accepted programming language or environment such as SAS, R, WinBUGS, SPSS,
Matlab, or a more fundamental language like C++, Java, Python or Fortran.
The submission for this component needs to include the original code that was used in the
programming as well as a certification that this code was indeed written exclusively, or at least
predominantly, by the student. If the code is too extensive, a link to a webpage can be provided.
In addition to demonstrating the skills necessary to implement a simulation study, the paper
needs to demonstrate that the study meets the following criteria:
a)
b)
c)
d)
the research questions are clearly articulated;
the design of the simulation study is appropriate for answering the research questions;
the outcome measures are appropriate for answering the research questions;
the methods for analyzing the resulting data are appropriate for answering the research
questions;
e) the interpretations drawn from the results are appropriate given the design and limitations
of the study;
f) the limitations of the work are clearly and thoughtfully addressed; and
g) the paper is written effectively for communicating these ideas.
VII. Research Presentation
The student must submit a professional presentation that he/she prepared for a professional forum
such as a regional, national, or international peer-reviewed conference (e.g., IMPS or the annual
meetings of AERA and NCME). It could also have been given to a group of professionals at the
University of Maryland at large, the College of Education, the HDQM department, the EDMS
program, or as part of an informal brown-bag session. However, the presentation slides must have
been prepared for a formal presentation and not as a handout for a roundtable or as a poster in a
poster session.
8
The presentation must be of typical length of presentations at professional meetings (i.e., 10 to 15
minutes). A script of the presentation need not be provided. A single presentation cannot be
submitted as part of a portfolio by multiple students in the program; only one student can have
primary responsibility for its development and presentation. The presentation must be a
professionally accepted format such as PowerPointTM format (PPT) or an equivalent product (e.g.,
LaTex).
The presentation must be accompanied by a summary statement that is about two double-spaced
pages in length, which describes the audience for the presentation, the context in which it was
given, the length of time for the presentation, whether any discussions followed the presentations,
and whether the presentation led to, or was based on, a peer- reviewed publication.
Apart from meeting the above requirements, the work that is discussed in the presentation needs
to meet the following criteria similar to those required for the simulation and applied papers:
a)
b)
c)
d)
e)
the research questions are clearly articulated;
the design of the study is appropriate for answering the research questions;
the outcome measures are appropriate for answering the research questions;
the methods for analyzing the data are appropriate for answering the research questions;
the interpretations drawn from the results are appropriate given the design and limitations
of the study;
f) the limitations of the work are clearly and thoughtfully addressed;
g) the text on the slides is written effectively for communicating these ideas; and
h) the layout of the slides facilitates comprehension of the material (e.g., it contains only
necessary key information, the font was large enough, graphics and tables supplemented
text).
VIII. Critique of a Paper or Book
The student needs to submit a double-spaced 4-10 page critique of a peer-reviewed journal
article, technical report, a book chapter, or a book. The reviewed work can focus on
methodological, applied, or instructional issues, but the review must address relevant issues
centered on measurement, statistics, or evaluation.
The critique needs to meet the following criteria:
a) be objective;
b) identify strengths and weaknesses of the reviewed work;
c) explain the significance of the measurement, statistics, or evaluation issues that were
explored in this work;
d) provide recommendations for revisions of the work; and
e) be written effectively for communicating these ideas.
9
IX. Research Proposal
The student must submit an original research proposal that has been submitted or will be
submitted to a conference, fellowship application, or grant application. The length shall be
consistent with the conference or funding agency’s requirement for such a submission. The call
for proposal should be attached and the submission requirements should be highlighted.
Examples of proposals include:
a)
b)
c)
d)
e)
a proposal submission to a conference;
a proposal to secure money from a granting agency;
a proposal for the evaluation, data analysis, or methodological approach for a project;
a proposal for a personal training grant funding; or
a proposal for some other professional purpose similar in importance to these examples.
The document may be written with some help solicited from others, but it must be clear that the
student took the primary responsibility for the intellectual ideas and creation process. The
proposal must, in the judgment of the exams committee, have a realistic chance of acceptance,
were it to be submitted for publication. If it has already been submitted for peer-review, a
negative peer-review does not automatically preclude a positive evaluation as part of the portfolio
assessment system. Similarly, a positive peer-review does not necessarily guarantee a positive
evaluation as part of the portfolio assessment.
In addition to representing a persuasive argument for of the proposed research, the document
needs to meet the following additional criteria:
a)
b)
c)
d)
e)
the research questions / objectives of the project are clearly articulated;
the methodology or the study plan is appropriate for achieving these objectives;
the methodology or the study plan is realistic/actionable;
the implications of the work for research and/or practice are clearly articulated; and
the paper is written effectively for communicating these ideas.
1
Doctoral Comprehensive Examination
Policies and Procedures for the Portfolio Assessment System
Measurement, Statistics and Evaluation
Department of Human Development and Quantitative Methodology
University of Maryland, College Park
Appendix A: Frequently Asked Questions
1. When do I have to complete these tasks?
The tasks must be completed by the time the portfolio is submitted to the Program Examination
Committee for review. The specific work may have been conducted at any time prior to that,
including prior to admission to the program, but within the ten year requirement for recency set
by the Graduate School. For example, if prior to admission you held a position in which one or
more of these facets was a part of your job, or if your advisor asked you to do a real data
analysis in your second year of graduate school and your work meets the requirements for this
portfolio, that product can be submitted.
2. Do I have to do the product submitted all by myself?
No, but in your cover page you will have to certify that you had not only primary responsibility
for the project, but in fact you actually performed most, if not all, the work being presented. It
is your skill at each task that is being evaluated, not your ability to organize or work with
others.
3. May I submit a single project to meet multiple sections of the portfolio?
Yes. The purpose of the portfolio is to demonstrate and establish your skill in each area. If you
were the primary person responsible, for example, for writing an excellent review and then
conducting a significant real data analysis, all as part of a single project, then that would
demonstrate your skill in multiple areas and the examination committee will accept the
submission to satisfy each facet. Remember, the purpose of the portfolio is to provide you an
opportunity to demonstrate a set of specific skills.
4. Do I have to demonstrate skill in all of these areas, or can I do an exemplary job on one to
counterbalance a less ambitious job on some other?
Each of these facets was identified as a significant area of skill in which all quantitative
methods students must be proficient and therefore you must demonstrate an acceptable level of
skill in every area.
5. How good is good enough?
The Program Examination Committee members are going to evaluate your work from the
perspective of a new employer in a prestigious organization. If they judge that your work on
2
each and every facet would meet the expectations (e.g., of a testing company, a university, a
journal editor, a book editor, a state assessment office), then that work will be accepted as
meeting the expectations of the quantitative methods program.
6 . Do I have to submit the portfolio with all the facets included?
Yes, the portfolio should be complete when it is submitted to the Program Examination
Committee. Your advisor may be comfortable examining individual components of the portfolio
at separate times prior to submission, but the Program Examination Committee wants the
material to be complete.
7. What if I fail some requirements in the portfolio?
You may be asked to clarify what you submitted if the committee judges that the issue(s) they
identify might be the result of insufficient clarity. Such clarifications do not constitute a
resubmission. You will be asked to resubmit after you have, in the opinion of your advisor,
satisfactorily completed that facet.
8. How many times do I have to provide remedial work? Or do I have another opportunity to
provide remedial work to the component I failed?
You have the opportunity to submit your material twice. If the material is still not satisfactory,
the Program Examination Committee may decide that you have failed the Doctoral
Comprehensive Examination.
9. How sophisticated does the data analysis have to be?
It is assumed that you are capable of doing advanced analyses if you passed the relevant courses
with a grade of B or better. The data analysis facet primarily has to do with your ability to meet
the specified needs of a typical contract, including cleaning the data and outlining and
completing the analyses needed to meet that request and answer the questions that were raised.
10. Must my portfolio represent everything that I have learned and everything that I must learn to
earn my degree?
No. The EDMS program recognizes that you have already taken a number of courses to satisfy
the requirements for a program of study approved by your advisor and shared with the
committee. The program also recognizes that you have passed either the doctoral preliminary
examination or the master’s comprehensive examination that demonstrates an integrated
knowledge of materials judged appropriate at that level of your work toward the degree. The
program also recognizes that passing the portfolio is another stage in your doctoral program and
will be followed by your completing a dissertation. That dissertation will allow you to
demonstrate a series of additional skills warranting the award of a Ph.D.
3
Sample Cover Sheet
Doctoral Comprehensive Examination
Portfolio Assessment
Measurement, Statistics and Evaluation Program
Department of Human Development and Quantitative Methodology
University of Maryland, College Park
I pledge on my honor that I have not received any unauthorized assistance
on this examination and the work contained herein is entirely my own.
Name: ______________________________
Date: ________________________
(Student signature)
Name: ______________________________
(Advisor signature)
Date: ________________________
4
PROGRAM OF STUDY
Name: ________________________________
UID: ___________________________
Advisor: ______________________________
List courses in chronological order, starting with earliest credits earned.
The following list should include the courses that the student might bring in and count towards
his/her degree from a program outside EDMS as well as the courses that student took here at UMD
as part of the degree program.
Semester/Year
Course Number
Course Title
Grade
Credits
5
Semester/Year
Course Number
Course Title
Grade
Total
Credits