Express the Error Margin in Quiz Formula Question as a % of the Result
Stefan Balaz
suggested this on March 28, 2013 17:04
If a set of solutions includes a wide range of numbers, using a numerical margin error is meaningless. For instance, +/-0.1 means a
large error for small numbers and can be an acceptable error for large numbers. Expressing the error as a percentage of the result
will solve the problem. I saw this idea in the discussion about scientific notation and significant digits but I hope it will be more
visible as a separate request.
37 people like this - Me too!
Comments
Autar Kaw May 20, 2013 08:33
University of South Florida
I second this feature. The ABSOLUTE percentage error tolerance is extremely important for STEM instructors to make quizzes that really serve their
students. There may be some concern - what if the answer is close to zero or zero itself. If the different combinations are currently generated before
hand for a question, and the instructor is able to take a quick look at them, it would not be an issue when the question is deployed. If division by zero
really takes place in any of the combinations or that the criteria of relative error is not met for a combination, the question generator should mention that
the range of inputs is invalid. Please take this request seriously.
I have written a blog to explain this issue: http://autarkaw.wordpress.com/2013/06/08/the-learning-management-sy...
The blog entry is given below.
The Learning Management System Canvas (Instructure) Lacks Key Feature in Quizzes
In Spring 2014, the university I teach at, University of South Florida is migrating from the current learning management system of Blackboard to
Canvas. It has been a welcome change but Canvas lacks a key feature in its quiz option which many STEM (science, technology, engineering and
mathematics) instructors have requested. In fact, I consider the lack of this feature as a bug.
To give you an example, one of the options in making a quiz is called the formula question. This option is attractive to STEM instructors as one can
develop a question whose correct answer is numeric but based on a formula.
For example, one may ask the question what is x/3 and the instructor can choose a range of input values of x (say 1 to 100,000). The quiz option
allows the instructor to develop up to 200 combinations with x being chosen randomly in the chosen range. The quiz option asks for an error margin
for the combinations. But here is the problem – the error margins are based on true error and not on RELATIVE true error. An error margin of +-1 may
be acceptable for a question that asks – “What is 10000/3?” but not for “What is 1/3?”. This issue could be easily resolved by making the error margin
to be a RELATIVE error margin.
The above example is simple as one may argue that one could use an error margin of 0.001 and it would work reasonably well for all possible value of
x, but for problems with many intermediate steps, where students could be carrying varying number of significant digits, the prescribed error margins
can create issues with a correct answer being deemed incorrect.
June 08, 2013 18:05
Ramune Braziunaite June 09, 2013 05:45
Bowling Green State University
I added my vote and tweeted your blog @bramune Good luck!
Autar Kaw June 09, 2013 05:50
University of South Florida
Hello Ramune: Thank you. I got an encouraging reply from the co-founder of Instructure: "Brian Whitmer @whitmer9h @numericalguy saw your
blog, now understand. Agreed that's better, honestly stalling until we address feature in general (e.g. sci. notat.)"
Derek Robinson June 09, 2013 21:38
University of South Florida
I added my vote. Hope this helps
Glen Parker August 22, 2013 06:12
University of South Florida
Bumping this to give it more attention. I've also asked my account rep to bring this up at their next managers meeting. This does seem like a bug
when you consider it in context of small numbers and precision.
April Mixon September 02, 2013 16:32
Clark College
Honestly, Canvas seems great for the soft sciences - it is truly a failure for anyone who wants to put quizzes or assignments online if they are teachign
f2f, or worse entirely online or in the hybrid format if you teach in STEM. The quiz section is an utter disgrace for anyone in the STEM disciplines.
Check out this thread as well - we need STEM folks on here voting like mad for these changes.
https://help.instructure.com/entries/21193413-Scientific-Notation-a...
Autar Kaw September 02, 2013 16:39
University of South Florida
April Mixon, Instructure should consider these options to be bugs, not enhancements. I have been using relative error tolerance as a way to grade
algorithmic answers in Blackboard. The algorithmic quizzes in Blackboard has helped my students to improve performance significantly. Now I am left
hanging dry as we have moved to CANVAS at Univ of South Florida and the Instructure administration at the highest levels does not consider it to be a
priority. Please tweet to Brian Whitmer @whitmer.
April Mixon September 02, 2013 16:47
Clark College
Autar - honestly this issue could be easily resolved using the Cloze style of questioning I posted in the Scientific notation forum. There, in the code that
you write for your question, you simply copy and paste what you want your answers to be and can type in error tolerances - which for us has been
helpful if we change the precision of our numbers - sometimes we need relative error for our stats problem to be 0.001, other times, 0.002. It's the
same type of problem, but the numbers we have them using, and rounding errors that might happen mean that different numbers sets in the same type
of problem might need a little extra cushion. When we give quizzes online, we have a block of 5-6 questions and told Moodle to just pull one. While
we tried to make everything work out - since we are working with experimental data, sometimes it just didn't, so changing the accepted tolerances was
just the easiest way out for us. Unfortunately, with the limited questions I can ask using Canvas, all my quiz questions apparently are going to have to
be multiple choice. I hate multiple choice. For some things it's fine - for every single quiz, it's a failure to the STEM faculty.
Autar Kaw September 02, 2013 16:57
University of South Florida
April Mixon: I agree. The code change from absolute tolerance to relative error tolerance is one line of code! I have developed 270 questions in
Respondus for my whole course and they are worthless now. Multiple choice has only so much value when students are helping each other outside of
the honor code.
Ramune Braziunaite September 03, 2013 05:33
Bowling Green State University
April - Canvas is a failure for social sciences and humanities as well. Among the worst, single forum discussion format which is a step back to the stone
age for online classes in history, communication, languages and many other areas. I added my vote to your call for quizzes, please consider returning
a favor https://help.instructure.com/entries/21941807-Higher-level-Organiza...
Please consider blogging, tweating and reaching out to other universities/schools that are still testing Canvas. Only they have some power to influence
what gets implemented. University administrators love Canvas because it is cheaper. Faculty are screaming about the product that does not work.
Devlin Daley, co-founder and apparently the key recruiter for Canvas has left http://mfeldstein.com/devlin-daley-departure-mean-instructure/. So
Instructure is not responsible for any promises they made to this point. 500 customers are doomed. Let's protect any others who are considering saving
money. It is not worth it.
By the way, after voting for this feature, I scrolled up and noticed my note that I already voted for this feature. I have 0 trust in Instructure, so this is not
surprising. It could be that the voting system does not work because many things don't work, or perhaps instructure is deleting votes on purpose. I will
start monitoring features that I cane about. I suggest you take notes on that as well. By the way, our votes don't matter. Instructure is fixing what they
want not what faculty ask for.
April Mixon September 03, 2013 09:29
Clark College
Thank you Ramune - I am happy also to share my vote - as I too will be using extensive discussion and forum posting as part of my student's hybrid
grade. In fact, I was thinking of moving away from paper homework entirely and asking students to solve problems in the discussion forum setting.
Something that is nearly impossible with the limited tool set I have available in Canvas. But boy, am I sure glad that the programmers spent their
valuable time on programming in that students can post things from Canvas directly to Facebook! I did notice that things they are fixing take about 2
years to do - which is also highly discouraging. The entire state of WA has moved to Canvas as their LMS - here's hoping that a state's worth of
firepower can influence change. I would love the ability to set two due dates for discussions. An initial due date - when is the first post due, and a final
due date - when are all of your replies due. That, to me, also seems to be paramount to keeping students on track. Sadly, for some reason, I cannot
request a feature! I can only vote and reply. I think Canvas knows how many features I want to request and is purposefully keeping me out of the
system :) (kidding...)
Hilary Scharton
September 03, 2013 10:20
Instructure, Inc.
Hi All,
This particular feature is actually one I feel pretty strongly about. As a math (stats and data) nerd myself, I get how not awesome absolute error is in
formula questions. In spite of not having a lot of votes behind this topic, our engineers took an initial look at this task a few weeks ago. While it isn't
anywhere close to a one line code change, it is currently in the queue for UI attention, which is one of the necessary first steps to making a change in
Canvas. As soon as we understand the scope of any UI tweaks, we'll be better able to predict about when we can see this in production.
In the meantime, I know that this doesn't come anywhere close to solving the problem, but I'd like to share that it is possible to put a decimal in the
error margin line (up to the hundredth place). Maybe that will help a little bit while we wait for the full solution.
@Ramune, I can tell you're frustrated with discussions and I have a lot of sympathy for you. I get that it's painful to change how you're used to doing
things--especially when you feel passionate about the way you've done it in the past. I wonder if that frustration has had any impact on your
impression that we don't care what faculty want. I can assure you that we do care--just as passionately as you care about discussions. As I've said to
you before in these forums, you're welcome to reach out to me personally anytime hilary(at)instructure.com. I'm happy to talk about anything that
concerns you, whether it's discussions, the reasons behind Devlin's decision to stop spending his life on airplanes, or possible bugs in
the Zendesk software we use for ticketing and support.
Thank you, everyone, for being part of our open community,
Hilary :)
April Mixon September 03, 2013 13:50
Clark College
Hi Hilary
A few questions - am I to presume (please note I haven't used this quiz problem type yet!) that you can only do error to the hundreds place?
Is there a reason why the system doesn't allow error for any decimal point range?
Am I also to presume that Canvas doesn't understand that from a science standpoint, the answers 0.544 and 0.5440 are not the same thing (significant
figures/precision with values)?
I just noticed that your significant figures for all of your starting x values are 4, but two of your answers have 4 sig figs and 2 of your answers have 3 sig
figs. In Moodle, I had to put answers like 0.5400 and 0.540 into scientific notation in order for Moodle to understand that those two values were
different numbers. It sucked, but it was possible. However, in several of the quiz options, using superscripts isn't possible (nor does it accept HTML or
your Latex code) so I can't use scientific notation to render an answer. Using 5.400E-1 or 5.400 x 10^-1 is truly poor formatting and doesn't appear
professional in the slightest (although I know excel understands the E formatting - it turns right around and can render it to appear professionally
formatted with superscripts for your exponents.
Any comments on getting the full text box available for all quiz types? Which would at least allow for the use of super and sub scripts regardless of
question type?
Thanks, April
Steve Mortensen November 13, 2013 11:34
Instructure, Inc.
Sorry for being a little late on this, but I've got some exciting news! Margins of error can now be input as percentages in Quiz Formula Questions. Take
a look at the 10/26/2013 release notes for more details: https://help.instructure.com/entries/22835025-10-26-13-Canvas-Produ...
I'm going to mark this as done and close it for comments. Thanks for your contributions, everyone!
Topic is closed for comments