CIS 475: EVALUATION OF COMPUTER APPLICATIONS
Prerequisites: A course in probability and statistics, or a course
in social science research methods.
Fall 2007 Syllabus
Contact Information:
Instructor:
Kathleen Higginbotham
Office:
Virtual
Office Hours: Monday-Friday 10 am – 11 am or anytime you see me in WebBoard
Online Hours: Monday-Friday 10 am - 11 am
IM:
kathleen_higginbotham@yahoo.com
Web Site:
http://web.njit.edu/~kh27
Telephone:
(985) 878-1578 (voicemail available)
E-mail:
kh27@njit.edu
Course Description: . Methods for identifying usability problems and for testing the relative
merits of alternative designs for interactive systems. Following a review of usability heuristics,
students read journal articles about and practice five different methods: semi-structured
interviews, protocol analysis, cognitive walkthroughs, user surveys, and controlled experiments.
Course Objectives:
1. Learn to evaluate computer applications, including human and organizational aspects of
performance, in a scientific and objective way.
2. Develop skills to:
1. Carry out a case study.
2. Design a questionnaire.
3. Do interviews.
4. Carry out a controlled experiment.
3. The course will introduce:
1. Methods of analyzing data.
2. How to analyze an information system:
1. Website.
2. Groupware.
3. Legacy organizational software.
3. How to identify usability problems.
4. How to test the relative merits of alternative designs.
4. Students will read journal articles about, discuss and practice the following types of
methods:
1. Semi-structured interviews.
2. User surveys.
3. Experiments.
Readings:
1. Text book: H. Russell Barnard, Social Research Methods, 2000 , 1st edition. Sage
Publications, ISBN: 9780761914037.
2. A number of recent articles for each week. These articles will contain examples of
different research methods being applied to analyze and improve software usability. The
articles are chosen from ACM and other publications. These publications are available
online to NJIT students through the digital library facilities. As a student you can have
access by downloading the Cisco Systems VPN client. I will include instructions in
WebBoard on how to find it and, also, set up a conference where we can deal with
problems encountered with its use.
Assessment
Course assessment (subject to modification) is 70% course work and 30% exams, broken down as
follows:
Assessment
Assignment 1: Research Design
Assignment 2: Protocol Analysis
Final Project.
Group Presentation
Class Participation
Mid-Term
Final Exam
TOTAL
Allocation
10%
15%
20%
10%
15%
15%
15%
100%
Graded Out Of
100
100
100
100
100
100
100
Course Outline:
* Assignment 1. Research Design
* Assignment 2. Protocol Analysis
* Final Project
* Group Presentation— This is a presentation of the final document. It is due one week after the
final project. The objective of this assignment is to give the group a time to
synthesize the various individual work that has created the final project and to
present the project in a powerpoint presentation. Since this is a distance course
and we will not be presenting our projects in a classroom, I will expect that a
voiceover narrative will be provided with each presentation. I will provide
instructions in a WebBoard conference on how to do this and entertain any
questions you may have on the mechanics.
* Class Participation—This class is presented in a virtual format making you a member of a
virtual community.. My philosophy of education is simple: “If you can say it,
then you own it.” Another way to put it might be, “I want you to find your voice
on this subject.” As such, I will expect you to contribute to the Webboard
discussions and activities. This is 15% of your grade. For this purpose, the
grading week will run from Sunday morning through Saturday night. A
topic will be set up for each reading assignment in the Readings
Conference or the Lectures Conference. You are required to have in each
topic, one comment on the material and at least one response to someone
else in the previous week’s postings. This is 50% of the participation
grade, the other 50% will involve the level of your participation in these
conferences. I don’t want to orchestrate this participation, but for those
new to WebBoard, I want to clarify expectations. For a perfect
participation grade, I would expect to see the following types of
interactions:
Message: You make a good point, but I wonder if this might also have a
bearing on XXX. You know, XXX seems to be related to YYY
and it even seems you can’t have one without considering the
other.
Message Response: No, I think they have very little interaction. I think
it is more likely that ZZZ is involved here. It says in the article
by ABC, that. . . .
Needless to say this participation level can take many forms: personal
opinion, quotes from the book, quotes from the readings, etc. The more
interaction, the more you find your voice, the better your participation
grade. I will post your participation grade weekly. This is to give you
immediate feedback and to help you prepare for the test. Answers to test
questions will involve this same level of familiarity with the material and
the ability to verbalize it (on paper). My job is to help you become
articulate on this subject.
* Mid-Term
* Final Exam
The following Matrix represents a breakdown of the the course requirements:
#
Lesson
Part I.
Introduction
1. L1--Information
1/14
to
1/20
2.
1/21
to
1/27
Reading
Text: PART ONE: BACKGROUND TO RESEARCH
systems research.
 Chapter 1--About Social Science
Qualitative vs.
quantitative methods,
Posting: Initial Posting in Discussion on W1 (C1 & L1)
Epistemology,
Rationalism,
Empiricism,
Assignment 1: Research Design--Begin
positivism,
interpretivism,
cognitive approach;
L2—
Text: PART ONE: BACKGROUND TO RESEARCH
Scientific method
 Chapter 2--The Foundations of Social Research
concepts:
 Chapter 3--Preparing for Research
Variables
measurement
unit of analysis
Posting:
validity
 Response to W1 posting(C1 & L1)
reliability
 Initial Posting on W2 (C2, C3 & L2 & [Goodwin
hypotheses
causal relationships
Ethics of social research Assignment 1: Research Design--Continue
Article:
Nancy C. Goodwin Functionality and Usability
1987])
CACM March 1987, 229-233.
http://portal.acm.org/citation.cfm?id=214758&coll=portal&dl=ACM
(Remember: You will need VPN client to access, see WebBoard)
3.
1/28
to
2/3
Part II Qualitative
Research
L3-Interviewing:
Unstructured interviews
Semi-structured
interviews
PART THREE: DATA COLLECTION
 Chapter 6—Interviewing
Posting:
 Response to W2 posting (C2, C3 & L2)
 Initial Posting on W3 (C6 & L3)
Assignment 1: Research Design--Continue
4. L4--Participant
2/4
to
2/10
Observation:
Observation,
Protocol analysis,
Case and field studies
Article:
L. Wood, “Semi-structured interviewing for user-centered design”, Interactions,
March-April 1997, p. 48-61
http://portal.acm.org/citation.cfm?id=245134
(VPN accesss)
PART THREE: DATA COLLECTION
 Chapter 9—Participant Observation
Posting:
 Response to W3 posting (C6& L3)
 Initial Posting on W3 (C9 & L4)
Assignment 1: Research Design--Continue
5. continue
2/11
to
2/17
L4--Participant
Observation:
Observation,
Protocol analysis,
Case and field studies
Article:
Maloney-Krichmar and Preece, “A multilevel analysis of sociability, usability and
community dynamics in an online health community,” ACM TOCHI, 12, 2 ( June
2005), pp. 201- 232.
http://portal.acm.org/citation.cfm?id=1067860.1067864&coll=GUIDE&dl=GUIDE&
CFID=858940&CFTOKEN=93703067
(VPN Access)
PART THREE: DATA COLLECTION
 Chapter 10— Direct and Indirect Observation
Posting:
 Response to W4 posting (C9 & L4 & Article [Maloney-Krichmar 2005])
 Initial Posting on W5 (C10 & Articles [Boren 2000] & [Holzinger 2005])
Assignment 2: Protocol Analysis--Begin
Articles:
Boren, M. Ted and Ramey, Judith. “Thinking Aloud: Reconciling theory and
practice.” IEEE Transactions on Professional Communication, 43, 3, Sept. 2000, 261278.
http: //ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/47/18778/00867942.pdf
(VPN Access)
Skim: Hozinger, A. “Usability engineering methods for software developers.”
Communications of the ACM 48, 1 (January 2005), 71- 74.
6. Exam 1
http://portal.acm.org/citation.cfm?id=1039541
(VPN Access)
Assignment 2: Protocol Analysis--Continue
Test 1
2/18
to
2/24
PART III
QUANTITATIVE
METHODS: DATA
COLLECTION
Sampling:
Text: PART ONE: BACKGROUND TO RESEARCH
Sampling frame,
 Chapter 5—Sampling
2/25 population,
sampling methods, Posting:
to sample size,
 Response to W5 (C10) posting
central limit
 Initial Posting on W2 (C5 & L7)
3/3 theorem and
confidence
Assignment 2: Protocol Analysis--Continue
intervals.
Article:
Bailey and Pearson. "Development of a Tool for Measuring and Analyzing Computer
User Satisfaction" Management Science, , May 1983, p530-545.
http://www.jstor.org/view/00251909/di012818/01p03944/0
PART THREE: DATA COLLECTION
8.
 Chapter 7— Structured Interviewing
3/4 Questionnaires and
 Chapter 8—Scales and Scaling
surveys:
7.
Question wording,
to format,
3/11
9.
3/12
response rate problem,
survey techniques.
Scales and scaling:
Simple,
complex,
Guttman,
Likert,
semantic differential,
t-testing
unidimensionality
Posting:
 Response to W7 posting (C9 & L4 & Article [Maloney-Krichmar 2005])
 Initial Posting on W8 (C7, C8 & Article [Lederer 1998])
Assignment 2: Protocol Analysis—Due
Final Project: Issued--Begin
Articles:
Lederer A.L. et. al "The role of ease of use, usefulness and attitude in the prediction of
world wide web usage" ACM CPR , undefined 1998, p195-204.
http://portal.acm.org/citation.cfm?id=279211
Spring Break
Could be a good time to work on Final Project.
to
3/18
10. Experimental design Text: PART ONE: BACKGROUND TO RESEARCH

Chapter 4—Research Design
3/19
Posting:
 Response to W8 posting (C7 & C8 & Article [Lederer 1998])
 Initial Posting on W9 (C4 & L9 & Article [Hoadley 1990])
to
3/24
Final Project--Continue
Article:
Ellen Hoadley, “Investigating the effects of color," CACM, 33, 2 (February 1990),
120-125.
http://portal.acm.org/citation.cfm?id=75578&coll=portal&dl=ACM
(VPN Access)
11.
3/25
to
3/31
Part III. DATA
ANALYSIS
Qualitative and
quantitative analysis-Univariate analysis
and correlation:
Raw data,
frequency distributions,
measures of central
tendency and
dispersion, graphing
variables,
z-scores and
chi-square test
Text: PART FOUR: BACKGROUND TO RESEARCH
 Chapter 11—Introduction to Qualitative and Quantitative Analysis
 Chapter 14—Univariate Analysis
Posting:
 Response to W9 posting (C4 & L9 & Article [Hoadley 1990])
 Initial Posting on W10 (C11, C12 & L10 & Article [Fogg 2001])
Final Project--Continue
Article:
12. L11. Bivariate
4/1
to
4/7
Analysis 1:
t-test,
ANOVA,
cross-tabs,
Lambda,
chi-square,
Gamma
B. J. Fogg et. al “What makes Web sites credible?: a report on a large quantitative
study”, Proceedings of the SIGCHI conference on Human factors in computing
systems, p.61-68, March 2001, Seattle, Washington, United States
http://captology.stanford.edu/pdf/p61-fogg.pdf
Text: PART FOUR: BACKGROUND TO RESEARCH
 Chapter 15, pp. 545-576
Posting:
 Response to W10 posting (C11, C12 & L10 & Article [Fogg 2001])
 Initial Posting on W10 (C15 pp. 545-576 & L11 & Article [Xue 2003])
Final Project--Continue
Article:
Xue, Y., Sankar, C.S., and Mbarika, V.W., (2003), "Multimedia and Virtual Teams:
Results of an Experimental Research", American Society for Engineering Education
Southeast Section Conference, Macon, GA, April 2003.
http://www.auburn.edu/research/litee/media/pdfs/eval_conf_papers/2003se_lucky.pdf
13. L12. Bivariate
Plan around Easter Break, nothing is due Friday or Sunday, which are school
Analysis 2: Correlation, holidays, so we should be okay. You normal postings are due by Saturday, so
Spearman’s r,
4/8 Pearson’s r,
plan ahead if you are busy on Saturday. There is no reading for this week. I
hope this makes the holidays of Passover and Easter more enjoyable.
Regression,
to Significance,
Eta
4/14
Text: PART FOUR: BACKGROUND TO RESEARCH
 Chapter 15 pp. 576-612
Posting:
 Response to W11 posting (C15 pp. 545-576 & L11 & Article [Xue 2003])
 Initial Posting on W12 (C15 pp. 576-612 & L12)
Final Project--Continue
14.
4/15
to
4/22
L13. Multi-variate
Analysis:
Partial correlation,
Multiple regression,
Path analysis,
Factor,
Cluster and
Discriminant analysis
Text: PART FOUR: BACKGROUND TO RESEARCH
 Chapter 16—Multi-variate Analysis
Posting:
 Response to W12 posting (C15 pp. 576-612 & L12)
 Initial Posting on W12 (C16 & L13)
Final Project—Due
15. Review &
Final Exam
4/23
Circumstances for Final to be determined at a later date.
to
5/1
Copying/Cheating
All students are expected to follow published guidelines on academic honesty and integrity. You
will find these at: http://www.njit.edu/academics/honorcode.php. You must acquaint yourself
with these policies before submitting any assignments. All written work must be original.
Violations of NJIT policies will be reported to the Dean of Students and may result in failure on a
particular assignment, failure in the course, failure in the course and probation, or failure in the
course and expulsion. Honor Code violations will be pursued immediately and aggressively.
Using "TURNITIN.COM"
A copy of all individual assignments, including the final project, will be required to be submitted
to the "turnitin.com" web site. This is in addition to paper copies that may be supplied to the
professor, or to posting a copy of the assignment for the class, according to detailed instructions
that will be given for assignments. This service checks your work against everything available on
the world wide web and in the archives for all sections of 675 past and present, and gives a report
on exactly where your work is similar to that of other work that can be found. This is an objective
way to help determine whether plagiarism has occurred.
The last thing I want for any of you is that you find “someone else’s voice”. I can assure you that
however good TurnItIn may be, I am better. Google and I will personally sit down with each
assignment and look for sources of exact matches. Make sure if you use material from a source,
that you quote it if exact. If you integrate the material into your work, please cite it. When you
are writing a paper for a project, you are building a case for the premise of your paper. It is much
less work for you if you cite pieces of information from a source than having to actually make a
case for the material. I want you to leave this educational experience being able to feel confident
in talking about the material you have assimilated. I want us to have a great semester.
See you in class!
Kathleen
Disclaimer: I do reserve the right to make any changes I deem necessary.
kmh