New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
History
The SBA first became interested in computerized exam taking (CET) in February 2002 as the
result of student inquiries as to why the Law School does not allow CET. Seeking to determine the
rationale behind this policy, the SBA learned from the administration that, at one time, students were
allowed to use laptops for in-class exams. Under that system, a program was loaded onto the student’s
laptop, which disabled all functions except word processing. Because of problems with the software’s
technical support, NYU Law eventually returned to the traditional practice of handwritten exams.
Having looked at schools which use some form of CET, the SBA at that time determined that the
technical hurdles the Law School had faced in the past could, most likely, be overcome. If it was
determined that overwhelming support for CET existed within the student body, the SBA resolved to
strongly suggest that the administration move towards implementing a CET program. However, the SBA,
in initial discussion about the issue, was itself clearly split as to whether implementation of CET would be
beneficial to the student body. Because of this situation the SBA decided to survey the student body to try
to find "the center of gravity" - where most people were leaning on the issue. The SBA intended the
results of this survey to enable it to come to an internal consensus about the type of recommendation it
would then make to the administration.
The survey was distributed on February 20, 2002 and consisted of seven questions designed to
determine class year, laptop ownership, degree of preference for different CET choices, and overall
preference between the current system and implementation of CET. On the reverse of the survey was a
statement of purpose and excerpts of the longer supporting and opposing arguments separately
distributed. The results of the survey indicated that the split among the members of the SBA on this issue
reflected those held by the student body at large. On the basis of these results, the SBA determined that it
was not possible at that time to reach a consensus that would have it push in favor of a change in the
status quo of written exams with the option to use mechanical typewriters. However, the SBA believed
that the results clearly indicate that CET was an important issue to the study body and therefore
recommended that the Administration look carefully at the needs and expectations of incoming students,
in order to draw its own, independent, conclusions on this issue.
Specifically, the SBA suggested that the Administration take all necessary steps to further
understand the nature of student’s desires for CET. Suggestions of how to accomplish this included:
1. Conducting a more detailed poll of students that provides more explicit detail about students’
interest and trepidation about CET and allows open commentary for them to voice their concerns
and desires;
2. Holding a community discussion chaired jointly by the students, faculty, and administration to
discuss the results of any such survey;
3. Facilitating additional research into the logistical ease or difficulty of developing a CET plan
along with more direct consideration of the problems and benefits experienced at other
institutions;
4. Determining, with continued vigilance, if matriculating students are interested in such a policy
and if they would like to see the nature of exam taking changed. This could be accomplished by a
paired survey issued with the student’s admission packet (upon acceptance of NYU’s offer) and a
follow up survey taken after the student has his/her first series of exams during the 1L year; and
5. Monitoring developments at other top 5 schools to determine which schools are adopting CET
and why.
The SBA does not know which, if any, of these steps were taken.
History
1
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
In September 2002, the Administration expressed to the SBA its interest in CET, specifically, in a
program called ExamSoft, and indicated that the Law School had already contracted with the owners of
ExamSoft in order to be able to test it with the incoming 1L class. In October 2002, the SBA again met
with the Administration regarding ExamSoft and learned that the Administration was aware that the
software only worked with Windows-based laptop computers. Regarding the test of this software on
incoming 1Ls, the SBA was unsure how it would proceed and how success would be defined.
Throughout Spring 2003, the SBA engaged in several discussions with the Administration regarding
ExamSoft's limitations regarding Macintosh and about loaner computers, however, none of these meeting
resulted in any clear resolution. In fact, the SBA did not learn the exact terms of the Administration’s
proposed ExamSoft policy until April 30, 2003, a mere two days before the beginning of Spring
Examination Period, when a group email containing the new ExamSoft policy was sent to all rising 2Ls
and 3Ls.
At the beginning of Fall Semester 2003, the SBA saw that the ExamSoft policy was at the forefront of
student concerns and accordingly devoted much of its time in meetings with the Administration to this
issue. Particularly, the SBA was concerned that:
1. There had been no report by the Administration regarding the “pilot test” on 1L students last year.
The SBA became even more concerned when it learned that there had been no simulation of inclass exams; last year, all in-class exams were handwritten and only on take-home exams was
there the possibility of using ExamSoft.
2. There was a serious lack of communication between the Administration and students regarding
the new ExamSoft policy.
3. There were significant technical issues raised by students who had used ExamSoft that hadn’t
been addressed by the Administration.
As a result of the SBA bringing up these issues, the Administration agreed to hold an ExamSoft Town
Hall Meeting to discuss and hopefully alleviate some of the student concerns. At the Town Hall Meeting,
which was held on October 27, 2003, many students took issue with the new ExamSoft policy. As a
result of this meeting, the Administration decided to solicit formal input from the SBA regarding the new
ExamSoft Policy.
The proposed policy embodied in SBA Bill 03-04, “A Resolution Regarding the Law School’s
ExamSoft Policy,” is the formal SBA response to the Administration’s request. With less than one week
to work in, the SBA brainstormed nine possible exam taking policies, which included the previous status
quo, the new ExamSoft policy, and seven other variations, published them for all students to see, and
solicited student comments. The SBA also considered the overriding goals of the Administration –
retaining exam integrity – and the professors – control over the examination environment and a
preference for typewritten exams.
The SBA believes that its proposed policy more than meets the Administration’s and professors’
goals while alleviating the many concerns of students. As is the case with most policies, this is a
compromise, therefore the SBA decided to attach the Explanatory Notes below to explain in further detail
each student concern and how these compromises were achieved.
History
2
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
Explanatory Notes
First Whereas:
The SBA has found that there are three instances where application of the Law School’s current
ExamSoft policy results in the perception of inequity amongst students. They are: (1) lack of sufficient
notice for returning 2Ls and 3Ls; (2) exam results; and (3) the Law School’s proposed solution.
1. Lack of sufficient notice for returning 2Ls and 3Ls:
The switch from Blue Books, the standard for decades, to computer-based testing is a seismic
shift in exam practice and law school policies. A change of this magnitude, with all of its
financial and test-taking implications, requires notice and a phase-in period. However, all
returning second and third year students matriculated without notice that the Law School
would be switching to computer-based testing. While arguably this may not be enough to
form a reliance interest per se, as the law school did not represent to any particular type of
exam technology, the Law School did, at least implicitly, represent to a fair exam taking
policy. The only agreement among students currently is that the present administration plan
will result in test taking results that will be substantially and inappropriately different
between the two groups of students because of the technology differences, and that these
technology differences are being imposed essentially without choice or notice (or the choice
to expend an extra $2,000 to re-establish an even playing field)
Until April 2003, there was no notice that in-class exams would be administered via
ExamSoft and require certain equipment. It is this “switching in mid stream without notice”
that is the cause of the perceived unfairness. Students relied up to that point on the
administration (both Law School and University) and purchased equipment that is noncompliant with the new requirements. In the future, when all students will have had notice
prior to matriculation as to the technology requirements, this particular fairness issue will
disappear.
2. Exam Results
Implementation of an exam taking method which allows two different modalities – computerbased and handwritten – results in perceived inequitable exam results. Student concerns
include:
1) Conscious bias by professors against handwritten exams
2) Unconscious bias by professors against handwritten exams
3) Discrepancy in word count limitations
4) Discrepancy in editing abilities inherent in word processing:
a. Faster (or slower) pace for typing
b. Mass delete, cut and paste abilities
c. Spell check
5) Advantage for handwritten exams in classes requiring tables, graphs, and other examrequired answers not supported by ExamSoft (e.g. transfers, class of 2006 and
L.L.M.s required to use ExamSoft are disadvantaged compared to those allowed to
handwrite).
Explanatory Notes
3
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
Despite administration efforts to alleviate concern over some of these specific issues through
simple statements that exam results will not be inequitable, no studies have been referenced
by the administration to support its statements, nor have the results of any beta-tests.
Conversely, common sense and in-class statements by professors to the effect that they much
prefer typed exams has instilled in students the perception that they will be disadvantaged if
they choose to handwrite.
3. The Administration’s proposed solution
The Administration’s proposed solution is to allow any student to purchase an ExamSoftcompatible laptop. The reaction of the impacted student body is this is a financially
inequitable solution because:
1) Until October 21, 2003, the law school via the university computer store were still
endorsing the sale of MacIntosh computers and desktop PCs and based on this
recommendation students bought non-ExamSoft compatible computers and do not
believe they should be forced to purchase a second computer to achieve parity with
other exam takers.
2) To remedy the situation, students are forced to either purchase an entirely new
computer system, with estimated prices of $1,850 (plus tax) at the NYU computer
store, or to purchase a wireless card and other upgrades to make an existing laptop
compliant (cost variable).
3) The option of additional loan support, while appreciated, does not address the longterm financial burden on the significant part of the student body that will be pursuing
legal careers in public interest practice.
The number of students without an ExamSoft-capable laptop is unknown, as there have been
no surveys of the student body by the administration that the SBA is aware of. However,
regardless of the results of any such survey, the inequity still exists for those without the
compliant equipment.
Second Whereas:
The SBA has received numerous emails from students concerning technological issues with the roll-out of
ExamSoft. The SBA has heard from the Administration many times that it is confident that not a single
test will be lost as a result of the ExamSoft software itself, however, there is still student concern
regarding this issue that has not been answered. For example, one current 2L emailed the SBA the
following:
“I was nearing the end of my exam. I suddenly got a notice saying the program could not save my
document and would be shutting itself down.
It turned out okay, because I'm relatively computer savvy, and because the professor didn't lock out all my
other programs. I was able to copy the full contents of my exam into MS Word and save it there. I
immediately emailed a copy of that document (with about 45 minutes left in the exam, but I wanted certain
proof that it all happened before, not after the time limit) to the professor's admin assistant. Then I
proceeded to complete the exam in MS word, and emailed the completed version, about 10 minutes before
the end of the exam, to the admin assistant.
Ultimately, ExamSoft lost my whole exam--they were not able to recover it. They said I had an error they
had never seen before. They accepted my MS word version, so I wasn't penalized, but I had to spend about
Explanatory Notes
4
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
5 hours after the exam on the phone with their tech support people.
Ultimately, their advice was to never again use that particular computer with ExamSoft.”
Aside from repeating that no exam had ever been lost (as indeed, this student’s email makes clear, the
exam wasn’t lost because it was copied into MS Word), the Administration has not effectively addressed
and alleviated student concern regarding these issues.
The substantial number of remaining technology issues do not relate to the ExamSoft software itself but
rather relate to the Law School’s IT system and its ability to handle 1,800 students taking tests on laptops
over a two week period. A sampling of issues not yet addressed to student satisfaction include:
1. What to do if a laptop freezes during the exam?
2. What kind of technical support will be available before, during and after the exam?
3. What should be done if there are Internet connectivity problems?
Third Whereas:
The implementation of ExamSoft is an Information Technology Process Change. IT process change
initiatives are common and widespread across both corporate America and American educational
establishments. The SBA recognizes that primary responsibility for policy decisions lies with the
Administration. However, as the SBA Constitution mandates, the SBA’s role is to “provide an effective
medium for the expression of students’ views” and to “maintain an effective liaison between students, the
administration, and the faculty of the Law School.” To those ends, the SBA believes it to be its duty to
note that some of the basic practices which are common to all IT process change programs were
apparently not followed in the ExamSoft implementation. The basic process is as follows:
1. Project Working Committee formed of impacted stakeholders/users. They write the following
documents, which are presented (after approval by the project manager and key stakeholders –
i.e. administration) to the developers:
a. Process map
b. User requirements
c. Use cases
2. Designers build prototypes (non-functioning dummy screens), validated by PWC by working
through the use cases and comparing against the user requirements.
3. After validation by PWC, hard-coding.
4. Beta Testing in non-live environment
5. Bug fix
6. Pilot Testing in a live environment of representative user groups
7. Bug fix
8. Phased roll-out.
Other common practices include:
1. Best Practices research of other similarly situated institutions that have implemented (or
attempted to implement) the process change proposed. To the knowledge of the SBA, the
Administration did not compile, examine, or make available to the student body any detailed
studies of the results of the process change (both IT technical issues and impacts on the
substantive issues of exam taking and results) at other law schools where computer based
examinations and/or ExamSoft were implemented. The SBA was particularly concerned by
reports from transfer students from Harvard and information from the SBA president of Cornell
Law School that ExamSoft was attempted and dropped due to poor results. While the SBA is
Explanatory Notes
5
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
2.
3.
4.
5.
happy to share this information with the Administration at this point in time, the larger issue is
that this information was not actively solicited and analyzed by the Administration at the
beginning of their process, to our knowledge.
Project Working Committee (PWC) of all impacted stakeholders formed to review proposals,
processes, and technologies. When this was suggested by SBA as a result of the survey
conducted in February 2002, it was not acted upon by the Administration. The SBA made this
suggestion again, with several members volunteering to serve on the focus group; again, the
Administration did not act on this and the comment on this proposal was “its not their [the
volunteering students’] job.” This response is indicative of the Administration’s misperception
about IT process change: it may be, in the end, a policy decision by the Administration but the
process of finding the appropriate decision is not a decision solely of “best technology” but of
“best process with new technology to meet the needs of the stakeholder.” Finding a new process
necessarily involves involving all impacted stakeholders. Discussing ExamSoft with SBA during
and around regularly scheduled Dean’s Meetings is not a substitute for a formal PWC: a processfocused stakeholder review and input of the proposed process change and new technology.
Beta testing, which involves parallel implementation of the new process and technology in a faux
live environment. Beta tests usually test capabilities of the system over all possible use scenarios.
This, to the SBA’s knowledge, was never done.
Pilot testing to demonstrate the effectiveness of the new IT process change in scenarios mirroring
or representative of full-implementation. The pilot test in this case was not representative: the
pilot test was of 1L take-home open book open hard-drive exams where the option of using nonExamSoft computers was available. To the extent that this mirrors current proposed practice for
all take-homes, it was an appropriate pilot (although to the knowledge of the SBA, no systematic
feedback from the pilot test group was gathered, as is normal in IT process changes). The current
Administration policy, which requires ExamSoft for in-class closed hard-drive exams, was not
tested in the pilot. Nor was the dual-modality (e.g. some students on ExamSoft, some students
handwriting) tested to examine differences in grading results.
Clear, constant, focused, and repetitive communication. IT process change programs should
constantly, with the guidance of both the administration and the focus group, be distributing
pertinent information to all impacted stakeholders as well as soliciting input. While the SBA
respects and agrees that the decision is ultimately the Administration’s, the SBA believes that
clear information flowing in both directions would have resulted in better implementation of this
IT process change. Examples of lack of communication include:
a. Failure of administration to publicly address the issues discovered in February 2002 SBA
survey.
b. Failure to provide notice and guidance on IT purchases to JD Class of 2005 despite the
apparent fact that ExamSoft, and its requirements, were known prior to August, 2002.
c. Lack of proper public announcement of new exam taking policy: The SBA believes that
a single email prior to the beginning of exams is an inadequate method of communication
which did not provide an opportunity for response. Preferred methods of communication
for IT process change include:
i. Article or advertisement in school newspaper
ii. Announcements in the Docket.
iii. Flyers
iv. Communication in conjunction and with assistance of focus group (did not exist)
and/or with SBA (not informed prior to decision being announced).
v. Opportunity for notice and comment.
vi. Dissemination in plain language of requirements and FAQ for the changed
process (FAQs are frequently discovered and answered by the beta testing and
Explanatory Notes
6
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
focus group processes, as well as from best practices research, three elements
missing from the ExamSoft process change).
The SBA believes that a better process roll-out would have caused the following results:
1. Identification of the level of concern over the dual-modality exam taking issue. This would have
led to a change in the phased roll-out of the project or of other alternatives, such as Law School
provided loaner laptops or private sector provided rentals for the examinations.
2. Avoidance of purchase of non-ExamSoft compliant equipment by the JD Class of 2005 and any
members of the JD Class of 2004 that purchased computers after August, 2002 (or whatever the
date was at which ExamSoft technical requirements were known). This would have reduced the
number of students impacted by the exam taking equity issue.
3. Clear and prominent explanation of computer requirements would have reduced the non-English
language Windows issue for foreign LL.Ms.
4. Technical or process solutions for exams that require non-text answers or that have non-text
questions, such as graphs and charts, could have been addressed with adequate time to find a
resolution and reduce concern.
In the limited time the SBA had to reach a formal response on the new ExamSoft policy, it was able to
publicly post nine different proposed policies and to solicit feedback from students on each one. While
not entirely satisfactory due to limited time during which to solicit feedback, the SBA still received scores
of emails detailing student satisfaction or dissatisfaction with the current ExamSoft policy and proposed
alternatives. Each and every email received impacted the final form of SBA Bill 03-04.
Resolution Section 1:
For the following reasons, the SBA believes that the use of ExamSoft should continue to be an option for
all students:
1. Although there are still many technical issues that need to be clarified, the Administration has
assured the student body that there are no technical issues that would prevent the use of
ExamSoft. The SBA, taking this assurance at face value, has no reason to believe that ExamSoft
should not be rolled-out due to technical reasons, although there is still much that needs to be
done in terms of communication between the Administration and the student body on this issue.
2. The SBA also believes that a solution preventing the use of ExamSoft would also perceived by
students as inequitable. Preventing the use of laptops for in-class exams would put students who
purchased ExamSoft compatible computers who otherwise would not have made this purchase in
a position financially worse-off than they would have been without the administration’s efforts to
implement ExamSoft.
This section requests that ExamSoft be optional for all students, not just returning 2Ls and 3Ls. Reasons
for this include:
1. Many visiting and transfer students, as well as L.L.M.s, have indicated to the SBA that they think
it is inequitable that others in their classes get to handwrite while they are forced to use
computers.
2. Tax L.L.M.s in particular have complained that they will be disadvantaged in taking their exams
if forced to use computers, due to the nature of the tax exams.
Resolution Section 2:
Explanatory Notes
7
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
This section requests loaner ExamSoft-equipped laptops be made available to any student who requests
one for a minimum fee. The SBA believes that providing this alternative to any student who so requests
will alleviate much of the student concern that the choice of handwriting vs. buying an expensive laptop
computer is no choice at all. The SBA sought and received two quotes from companies that rent laptop
computers. For a Pentium III Laptop running Windows 2000 and equipped with a wireless internet card,
the quotes were between $13 and $15 per day. The SBA suggests that the Law School coordinate with a
vendor to make laptops available to students for a minimal fee of perhaps $20 per exam.
Resolution Section 3:
This section attempts to honor professors’ preference for typewritten exams while alleviating the concerns
of students that there may be conscious or unconscious bias in favor of typewritten exams which may be
reflected in grades. The Law School currently allows students to have their handwritten exams typed up
only when requested by professors. This proposal would extend this to require that all handwritten exams
be transcribed prior to the professor receiving them for grading. The SBA suggests that the Law School
coordinate a support staff that would transcribe handwritten exams for a minimal fee of perhaps $20 per
exam.
Resolution Section 4:
This section requests that students using laptop computers and students handwriting be separated into
different rooms during in-class exams. Many students who intend to handwrite have expressed concern to
the SBA regarding distraction from the sound of typing on computers during exams. The SBA fails to see
any reason why this would not be possible.
Resolution Section 5:
This section requests that an electronic and hard-copy FAQ be distributed to all students by November 18,
2003. Many students’ concerns, as referenced above, focus on lack of information provided to them by
the Administration. The SBA believes that a FAQ addressing the policy, implementation, and technical
issues would go a long way in alleviating these student concerns.
Resolution Section 6:
This section requests that an emergency plan be presented to students in electronic as well as hard-copy
form by November 18, 2003. The SBA believes this is necessary because, as the experience of the sevenweek classes show, sometimes unexpected things happen. Providing students with information so that
they know what alternate exam taking modality will be utilized in case of the unexpected will help
students feel confident in the exam taking process.
Resolution Section 7:
Explanatory Notes
8
New York University Student Bar Association
Bill No. 03-04
A Resolution Regarding the Law School’s ExamSoft Policy
The SBA has raised several of these issues with the Administration repeatedly, and has yet to receive a
written answer. This section formally requests the Administration provide a written response to these
issues by November 18, 2003.
Resolution Section 8:
The Administration referred to such a memo at the ExamSoft Town Hall Meeting. This section simply
requests that the Administration detail in it student concerns and make it public to the student body by
November 18, 2003.
Resolution Section 9:
This section requests that the certification period for ExamSoft be extended through November 26, 2003,
to give students the opportunity to read all of the information requested in Sections 5 through 8 prior to
deciding whether to use ExamSoft, and thus having to certify.
These Explanatory Notes accompany SBA Bill No. 03-04 and were approved by the Student Bar
Association Board of Governors on November 4, 2003 by a vote of 14 to 3.
_____________________
Nick Kujawa
SBA President
Explanatory Notes
9