Pamela Russo, M.D., M.P.H.
Senior Program Officer, Young Epidemiology Scholars Program
The Robert Wood Johnson Foundation
Diane Tsukamaki
Director, National Recognition and Scholarship Program
The CollegeBoard
Dear Pam and Diane:
First, congratulations on a job very well done, and thanks for a great YES weekend. It
was a fantastic learning opportunity for all of the YES Scholars, and the parents and
chaperones as well. In particular, we would like you to know that we strongly support the
stated objective of the Robert Wood Johnson Foundation and the CollegeBoard: that the
Competition should be educational for the participants, in addition to its competitive function
of winnowing out the best projects.
Per our discussion on Monday, it does seem that the one major improvement that
could be made in the program, that would greatly enhance the learning of the Scholars, would
be the provision of feedback from the judges to the Scholars. As we understand it, the
Foundation and the CollegeBoard have three areas of objection to the provision of feedback:
1.) The threat of litigation, 2.) Identification of the National Finalists prior to the formal
announcement at dinner on Saturday, and 3.) Providing undue advantage to contestants
(current juniors) who compete a second year (as seniors). A fourth area of objection might be
concern about increasing the workload on judges to the point where it would affect
recruitment of judges.
While there are certainly potential pitfalls in this day of seemingly omnipresent
litigation, perhaps a way can be found to provide feedback to the Scholars without exposing
the judges, the Foundation, or the College Board to increased risk; without prejudicing the
national finalists in any way; without unduly increasing the workload on the judges; and
without providing undue advantage to Scholars who compete a second year. Would it be
possible, for example, to develop a form/checklist, (or perhaps a separate form/checklist for
each of the major research areas: e.g. mathematical model development, questionnairebased research, etc.) where the strengths and the areas needing improvement for each
presentation could be rapidly delineated? To distribute the workload, perhaps each of the
regional judges (who are not going to also be national judges) could choose two of the
semifinalist projects to comment upon (based, perhaps, upon the congruence of the individual
projects to their own areas of expertise), using the form/checklist. Alternatively, if your
counsel feels that any direct feedback from the judges poses too much of a litigation risk, then
perhaps an additional experienced researcher could sit in on the presentations and be
responsible for providing feedback based upon his/her impressions of the Scholar's paper,
presentation and Q&A responses, independent of the judges. Additionally, language similar
to that used in the Miss America system, wherein as a condition of entry, contestants affirm
that all decisions of the judges are final and agree that they will not bring suit against the
judges, the sponsors and/or the organization, could be incorporated into the application. An
arbitration clause could even be inserted, if desired by your counsel (although that may be
excessive.)
The checklist could either be given to the respective Scholars after Saturday's dinner,
or could even be mailed to the respective Scholars after the YES weekend, so that, in either
event, everyone would be treated equally. Neither of these procedures would give any hint as
to who had been selected as a National Finalist prior to the announcements at dinner
Saturday. The judges filling out the forms could either be anonymous, or the judge could
provide her/his contact information, and agree to carry on a limited discussion with the
Scholar regarding any subsequent questions the Scholar might have about the
recommendations. The latter approach (limited additional feedback) is certainly preferable,
as it would allow the Scholars to clarify any points of interpretation they weren't sure they
understood in the feedback form.
Regarding the issue of giving further competitive advantage to repeat Scholars,
certainly any Scholar who has experienced the Competition once is going to have an
advantage over the "newbies". We can understand that the Foundation and the
CollegeBoard don't wish to do anything to exacerbate the situation. Perhaps a way to obviate
the concern about creating unfair advantage through giving feedback would be to post all of
the regional finalist's papers online, along with the related comments -- especially if they were
from the "additional independent researcher" suggested above. That way, new first-time
Scholars would have the benefit of the work others have done as well as the feedback others
received, and it would help to place everyone on a similar footing (although the Competition
experience itself is so powerful we don't know how one could compensate for it in terms of
Scholars who are returning for a second year vs. those who are new to the Competition....).
Upon further reflection, perhaps the best way to mitigate the “experiential advantage”
of returning Scholars would be to make each year's papers, PowerPoint presentations, and
associated feedback available online. This is probably the most cost-effective way of
“leveling the playing field” for newcomers.
Again, congratulations on a job very well done, and, on behalf of ourselves and our
respective Scholars, many thanks for a great YES weekend.
Very truly yours,
Mark, Judy and Rob
Rob Gillio, M.D. <rgillio@paonline.com>
Pulmonary and Critical Care Medicine (ret.)
Chairman and CEO, Innerlink
Judy Stone, M.D. <jstone12345@yahoo.com
Internal Medicine and Infectious Diseases
Mark Skinner, Ed.D. <mskinner@hereintown.net>
Sample Feedback Form
Questionnaire-based Research
A note to the Judges: Please provide as much information to the contestants as possible on this form.
In particular, please try to provide as much detail as possible regarding suggested improvements.
A. Study Design
1. Sampling design: ________________________________
(Adequacy of randomization strategy, stratification, etc.)
2. Appropriateness of test statistic:___________________
3. Are potential confounding variables considered/accounted for?
___________________________________________________
___________________________________________________
4. Instrumentation issues: _____________________________
(Reliability, validity, .......)
B. Methodology
1. Sample size: ______________________________________
2. ..........................................
C. Results / Discussion / Conclusions / Recommendations
1. ...............................
D. Presentation
1. Communication effectiveness of slides _____________________
2. Points to be added / deleted to / from presentation
_______________________________________________________
3. Remarks regarding Q/A session:
_______________________________________________________
E. Any issues raised in paper (as opposed to PPT Presentation / Q&A session)
Sample Feedback Form
Mathematical Modeling Research
A note to the Judges: Please provide as much information to the contestants as possible on this form.
In particular, please try to provide as much detail as possible regarding suggested improvements.
A. Adequacy of model:
1. Variables: __________________________________
2. Algorithms: __________________________________
3. Assumptions: _________________________________
B. Results / Discussion / Conclusions / Recommendations
1. ...............................
C. Presentation
1. Communication effectiveness of slides _____________________
2. Points to be added / deleted to / from presentation
_______________________________________________________
3. Remarks regarding Q/A session:
_______________________________________________________
D. Any issues raised in paper (as opposed to PPT Presentation / Q&A session)