Analysis of BABELnot MCQs for
Level 1, 2 Exams
Daryl D’Souza, James Harland, Margaret Hamilton
BABELnot Melbourne, 20 July 2012
Research Background
 Research aim: Analyse BABELnot MCQs in written
exams at Levels 1 and 3. These questions employed in
thinking-out-aloud experiments (ref: ??)
 Research questions: How do the students’ performance
on BABELnot questions correlate with the overall
performance on the exams?
Methodology
The exams
 Programming 1 (Level 1)
 Semester 1, 1st year, CS/SE
 No previous programming (strictly) necessary
 Java language
 20 MCQs, 5 SAQs, 1 3-part themed question
 All three BABELnot Qs used as MCQs
 Programming Techniques (Level 3)
 Typically 2nd year
 Prerequisite Programming 2
 C language
 Two BABELnot Qs used
 one as one part of a seven-part SAQ
 one SAQ on its own
Question 5 P1
boolean testArray(int [] x, int arrayLength)
{
for (int i = 0; i < arrayLength-1; i++)
{
if (x[i] > x[i+1]) return false;
}
return true;
}
If testArray returns true after this code is executed, which of the following is the strongest
statement we can make about the contents of the array x? Assume the subscripts p, p+1 and q
are legal indexes of x.
*A.
B.
C.
D.
x[p] <= x[q] for all p < q.
x[p] <= x[p+1]
x[p] <= x[p+1] where p is an even number (i.e. 0, 2, 4, etc).
No statement can be made, as the last iteration of the loop will attempt to index an element of
the array that does not exist.
E. x[0] < x[arrayLength-1]
Question 11 P1
How many times will the while loop body below be executed?
The code below in the right of the table moves all elements of the array x one place to the
right, with the rightmost element being moved to the leftmost position. The variable length
contains the number of elements in the array x:
Line
number
Left
right
1
int temp = ???
int temp = x[length-1];
2
for (int i ???
for (int i=length-2; i>=0; i--)
3
4
x[i-1] = x[i];
??? = temp;
x[i+1] = x[i];
x[0] = temp;
Consider the partial code provided in the above table, on the left. When the occurrences of
??? are replaced with appropriate code, that code can undo the effect of the code on the right.
That is, when the ??? are replaced appropriately, the code can move all elements of the array
x one place to the left, with the leftmost element being moved to the rightmost position.
The table already shows that line 3 is different in the code on left to the code on the right.
When the ??? on lines 1, 2 and 4 are replaced with appropriate code, which of these lines of
code must be different between the left and the right
A. None of the lines 1, 2 and 4 must be different.
B. Only lines 1 and 4 must be different.
C. Only line 2 must be different.
*D. All of the lines 1, 2 and 4 must be different.
Question 12 P1
Below is incomplete code for a function which returns the minimum
value in the array x. When an appropriate line of code is selected from each box,
the completed code will scan across the array, using the variable minsofar to
remember the best candidate for minimum so far.
int min(int x[], int arrayLength)
{
int minsofar = A) 0
;
B) x[0]
for (int i = 1 ; i < arrayLength; i++)
{
C) minsofar
if ( x[i] < D) x[minsofar]
)
{
minsofar = E) i
;
F) x[i]
}
}
G) minsofar
return H) x[minsofar] ;
}
Which of the following choices of lines from the boxes will produce a correct
version of the function min?
A) Lines ACFG only.
B) Lines ADEH only.
C) Lines BCFG only.
*D) Both lines ADEH and lines BCFG.
Question 1g PT
In one sentence, explain in plain English the purpose of the following piece of code
int fn(int *array, size_t n)
{
size_t i;
for (i = 0; i < n - 1; i++) {
if (array[i] > array[i + 1])
return -1;
}
return 1;
}
.
Question 4 PT
The purpose of the block of code below is to take an array of integers and move all elements of the
array one place to the right, with the rightmost element moving around to the leftmost position.
void shift_right(int *array, int n)
{
int temp = array[n - 1];
int i;
for (i = n - 2; i >= 0; i--)
array[i + 1] = array[i];
array[0] = temp;
}
Write a function shift_left that will move all elements of the array one place to the left, with the
leftmost element moving around to the rightmost position.
void shift_left(int *array, int n)
{
/* . . . */
}
Results—P1
60
50
40
Question 5
30
Question 11
Question 12
20
10
0
Correct %
Answer a %
Answer b %
Answer c %
Answer d %
Answer e %
Results—P1
120
100
80
BABELnot
60
Other MCQ
BABEL
40
20
0
HD
DI
CR
PA
NN
Flop
Results—PT
90
80
100
70
60
80
50
BABELnot
40
BABEL
30
60
1g
40
20
10
1a-f
20
0
HD
DI
CR
PA
NN
Flop
0
HD
DI
CR
PA
NN
Flop
Chi-squared: p = 0.20
90
120
80
100
70
60
80
50
BABELnot 60
Overall
40
40
30
4
2,3,5
20
20
10
0
HD
DI
CR
PA
NN
Chi-squared: p = 0.228
Flop
0
HD
DI
CR
PA
NN
Flop
Discussion
 The only significant correlation was between the
BABELnot questions and BABEL ones on the PT exam
 No significant correlations on P1, possibly due to being
only 3 MCQs and less overall marks than on PT.
 BABELnot questions tended to ‘spread’ the students
out more than other questions (high mode & median,
lower mean)
BABELnot Deliverables
 If Mercy's work contributes to BABELnot, then it must be contributing
(perhaps indirectly) to one of the projects' deliverables ...
•
A formalism for describing learning outcomes, exam
assessment, and the method for mapping between the two, for the
first three programming subjects
•
The learning outcomes of the first three programming subjects,
from at least the six participating universities, re-expressed in the
formalism
•
An archive of exam questions, with meta-tags mapping the
questions to the formalism, serving as examples to other academics
•
Performance data from real students for a subset of the archived
exam questions
Mercy’s work
 Briefly Mercy’s work involves the creation of Facebook sites with
the BABELnot questions posted there. We aim to find out:
 Why students make the errors they do
 What mental models students have about these questions
 What words they are using to explain their answers and/or
difficulties
 Find out in their own words how most students perform while
answering these exam questions
Contribution to BABELnot
 Performance data from real students for a subset of the
archived exam questions
 These are real students enrolled in the programming courses
 They must solve the actual subset of BABELnot questions
 They must explain their steps somehow – to the group, and not to
the instructor alone, as in a pencast.
 A formalism for describing learning outcomes, exam
assessment as in their discussions, students will need to
identify where they are struggling, and this discussion can:
 improve our formalism and language
 identify the problematic learning outcomes eg: can we simply say
one outcome would be to compute a ‘sort’ if we are discussing
objects?