Effectiveness of Educational Robotics in the Classroom

• Dr. Bradley S. Barker, 4-H Science and
Technology Specialist
– Email: bbarker@unl.edu
• Dr. Neal Grandgenett, Professor of
Mathematics Education
– Email: ngrandgenett@mail.unomaha.edu
• Dr. Gwen Nugent, Associate Research
Professor, Nebraska Center for Research on
Children, Youth, Families and Schools
– Email: gnugent1@unl.edu

• Provide background of using robots in nonformal education
• The three iterations (pilot study, large scale study, NSF ITEST program) and impact data from the 4-H robotics program.
• Embedded assessment (Reasoning/
Communication)
• Effects of robotics on attitudes
• Summary
• Questions

• To address the shortage of students pursuing careers in science, technology engineering, and mathematics (STEM).
• Goals
– promote youths’ interest in STEM fields (including IT),
– introduce basic STEM skills,
– foster problem solving and inquiry,
– and encourage teamwork

• Fundamental gap in literature – what is the impact of using robotics on STEM learning?
• Most studies are qualitative, examine self-evaluations, project descriptions, and student reactions.
• We set out to examine the impact on STEM learning in a nonformal environment.

• Purpose was to develop assessment instrument and examine impact on learning STEM concepts.
• Developed 24-item multiple choice test one item from each unit in the curriculum.
– Test items reviewed by CMU robotic experts and revised.
• Robotics group met twice a week for six weeks. Pre test prior to intervention and post on last day for both groups.

• Participants
– 32 students in study ages 9-11 median age was 9.0
– 14 students in (9 male,
5 female) experimental group
– 18 students in control group (11 male, 7 female)
– Control group selected by instructor in same grade but not part of the afterschool program.

• Cronbach’s alpha score of 0.86 on posttest.
• LEGO Robotics questions removed
• STEM assessment items alpha was calculated at 0.76
• The assessment instrument seemed to be valid and reliable.
• Used Pell and Jarvis instrument to measure attitudes towards science.

• Observed impact on learning
– No significant difference (t(30) = 11.60, p = .70 on pretest scores between groups (M=7.50, SD = 2.58, control) and M = 7.93, SD=3.71, experimental).
– Significant difference (t(22,17) = 12.93, P < .000 between groups on posttest (M=7.44, SD = 2.98, control) and M = 17.00, SD= .88, experimental).
• No change in attitudes

• Refined the assessment instrument.
• Felt confident students had an increase in
STEM content areas as well as specific robotic concepts.
• Used instrument for larger study.

• Participants
– 121 students ages 7-14 from 6 afterschool programs and 3 4-H clubs.
– 36 youth ages 11-14 acted as a control group from
3 afterschool programs
• Interventions (not concurrent) lasted for 8 weeks. The pretest was administered prior to the intervention and the post test was administered immediately after.

• Impact on learning
– ANCOVA analysis used posttest as dependent and pretest, gender as covariates
– Main effect was significant F(1,141) = 11.04 p=.001
– Posttest (M=10.68, SD = 3.93, control) and (M =
11.09, SD= 3.93, experimental).
– No difference based on gender F(1, 141) = .833 p=.478

• Expanded 4-H robotics program
– Includes the integration of robotics with Geospatial technologies
(GIS, GPS, aerial photography)
– Looking at applications in precision agriculture and natural resources
– Provide career exploration with visits from scientist and engineers.
– New robotic kits

• Open to middle school students (200)
• Year 1 - Pilot camps
• Year 2
– Start with 40-hour summer camp
– 1 day camp for educators and leaders
– Youth then complete 80 hours in clubs and afterschool programs
• Year 3
– 40-hour summer camp
– 80 hours in clubs and after school programs

• Site 1 – Gretna, NE 6 day overnight camp.
– N = 12 ages, 11-14, median age 12.50
– 8 males, 4 females
– Paid to attend the camp
• Site 2 - Grand Island, NE 5 day camp.
– N = 26 ages 11-15, median age 12.00
– 18 males, 8 females
– Offered through CLCs at Barr M.S.
• Modified content examine and added new questions, piloted an embedded assessment

• Site 1 pretest m = 14.5, sd = 4.42 and posttest m = 17.5 sd = 4.89
• Site 2 pretest m = 11.77, sd = 2.99 and posttest m = 16.04, sd 3.68
• Overall significant increase in scores using
ANCOVA analysis using posttest as dependent and pretest as covariate F(1,31) = 21.24, p =
.000.
• KR20 score = .81, Alpha score = .799

• Developed new attitude instrument focusing on robotics
– Modeled after the Motivated Strategies for
Learning Questionnaire (Pintrick, Smith, Garcia, &
McKeachie, 1991)
– Multiple scales focusing on task value, motivation, self-efficacy, problem solving, cooperative learning
• Conducted interviews with selected youth in
Grand Island

• Goal - promote youths’ interest in STEM fields
2.8
2.6
2.4
2.2
2
3.2
3
4-point scale
1 2
Scientist
Engineer
Mathematician
Computer

• Promote youth’s interest in STEM
– Six of the seven Grand Island youth interviewed said the camp made them like math and science more
– Four said it increased their interest in a career in
STEM

• Goal – foster problem solving and inquiry skills
4.6
4.4
4.2
4
3.8
3.6
3.4
3.2
3
I make a plan before I solve a problem
I develop my own ways of solving problems
I like to know how things work
I can draw valid conclusions based on evidence
I used a step by step process to solve problems
Significant increase for “make a plan”
Pre Post

• Interviews
– There was kind of a big task, so it made us break it down into little parts and then get it done step by step.
– For the robot, you have to start with nothing and tell it exactly what to do.
– If your robot did something wrong, you have to figure out after how many steps it was and then you fix the particular problem.

• Goal – Encourage teamwork
4.2
I like to w ork w ith others to complete projects
4
3.8
3.6
I like listening to others w hen trying to decide w hich problem-solving approach to use.
I ask my teammates for help
Significant increase for
“work with others” and
“listening to others”
3.4
Pre Post

• Interviews
– I like the hands on aspect of it. I hate just being stuck in a chair and writing stuff down.
– It was fun when we did challenges.

• Revise the instrument
– Factor analyze the instrument and revise
• Use results to guide instructional improvement
• Initiate new study that measures attitude midway through the camp

Pretest
Measures
Content Tests
Attitude Test
Posttest
Measures
Content Tests
Attitude Test
Envelope Activity
Think-Aloud Interview
Design Tasks
Task Rubrics
Control Groups: Partners and Partner Schools

(mathematical communication/reasoning, and also used to discuss programming instruction)
Steps to Mailing a Letter: Student Directions
Directions: In order to do something well, like playing a sport or a board game, you usually need to know the "procedures" of how to do it. Thus, it is often helpful for you to know the steps in doing some task, before you are able to explain that task or procedure to someone else. For example, if you are given a letter that you need to mail, with an envelope, a stamp, a ruler, and an address of where you want to send the letter, can you describe how you would mail that letter? Or in other words, can you describe the general procedure for mailing a letter?
Let's further pretend that this person who will be mailing the letter is a friendly alien from Outer Space, and that they have absolutely no idea about how to mail a letter here in the United States. List the steps that the alien should use to mail a letter. Use the blank sheets of paper provided to make your list. You have 15 minutes to accomplish this task.

Student #5: Pretest
1.
Tur n the envolope, so the flap is down.
2.
Make sure that the open flap is at the top.
3.
Put the stamp in the top right corner.
4.
Put your address, "the return address," in the top left corner.
5.
Your addes should look kind of li ke this: First Las
111 Cave R
Omaha, NE 6811
6. Put the address you are sending it to in the mi ddle.
7. It is supposed to look something li ke this: First Name Last Name house # and road city, state and zip code
Front
Samm i Bray
23 Merry lane
Omaha, NE 68114
Jack Black
222 Jumper lane
Oakland, NE 68045
Front Back
How to address an envolope
Student #5: Posttest
1.
Front Back How to address and send an envolope, or letter.
1. First make sure the front is facing up.
2. Find a stamp & put it in the top right corner.
3. Put your address in the top left corner.
4. Put the person you're sending it to in the mi ddle of the envolope.
5. Put your letter inside the envolope & close it.
6. Then put it inside the mail box, & put the flag up.
7. And you're good to go.
By the way it should look something li ke this. your address Their address stamp
First Last name road address city, state zip
First name Last name road address city, state, zip code

Student #5: Posttest
1.
Front Back How to address and send an envolope, or letter.
1. First make sure the front is facing up.
2. Find a stamp & put it in the top right corner.
3. Put your address in the top left corner.
4. Put the person you're sending it to in the mi ddle of the envolope.
5. Put your letter inside the envolope & close it.
6. Then put it inside the mail box, & put the flag up.
7. And you're good to go.
By the way it should look something li ke this. your address Their address stamp
First Last name road address city, state zip
First name Last name road address city, state, zip code
Holisitic Scoring Rubric for Envelope Activity
Student ID:____________________________ Pre:___ Post:___
Enu meration of steps
Score:_____
Precision of steps
Score:_____
Looping
Score:_____
Measurement
Score:_____
3
Each step has a unique, logical identifier
Steps are exact and unambiguous
Looping is used appropriately in the steps (e.g. , Ņrepeat steps
3-6Ó)
2
Most steps have a unique, logical identifi er, but some steps have not been enumerated or some enumerated steps contain multiple steps
Steps are mostly exact, but some are not as precise as others
Looping is used, but not accura tely , or looping is im pli ed indirectly (e.g. put name, address, zip under each other)
Relative measure ments are given (e.g., Ņin the top left cornerÓ)
1
Few if any steps have been enumerated
Steps are mostly inexact and potentially confusing
Looping is not used in the steps.
Measure ments are not given
Use of
Examples
Score:_____
Completeness of ins tructions
Score:_____
Exact measure ments are given in standard units
(e.g., Ņ4 inches from the leftÓ)
Extensive written and/or grap hic examples are used to ill ustrate steps to be taken, or a sample envelope is created with addresses and referred to in the narrative.
If foll owed, the steps would lead to the maili ng of a correctly addressed envelope
Some written and/or grap hic examples are used to ill ustrate steps to be taken, or a sample envelope is created, with both return and maili ng address given.
If foll owed, the steps would alm ost lead to the maili ng of a correctly addressed envelope
No written and/or grap hic examples are used to ill ustrate steps to be taken, or just a stamp or partial address is drawn on the sample envelope.
If foll owed, the steps would NOT lead to the maili ng of a correctly addressed envelope
Total Rubric Score:______

Student
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
Student 9
Student 10
Student 11
Student 12
Student 13
Student 14
Student 15
Student 16
Student 17
Student 18
Student 19
Student 20
Student 21
Student 22
Student 23
Mean
Stdev
2.1
1.9
1.0
1.6
1.3
0.6
0.5
0.2
0.5
0.5
1.7
0.7
Experimental Pretest vs. Experimental Posttest:
9.6
2.1
2.4
0.7
2.2
0.7
1.5
0.7
1.6
0.7
1.7
0.8
Probability Value (p <):
2.0
0.7
11.4
2.6
EnumStep PrecStep Looping Measure Examples Completeness T otal
0.02487 0.0249 0.0023 0.37346 0.0127 0.0347383 0.002
sig at 0.05
sig at 0.01
Summary of Pilot Data from Mailing a Letter Activity (Experimental)
2
2
2
2
2
3
2
2
3
2
2
2
2
2
3
2
3
Pretest Data
Enumerat P recision Looping Measure ion of steps
2 of st eps
2 2 ment
2
Use of examples
Completeness of inst ructions
2 2
3
1
2
1
1
1
1
1
2
1
2
1
1
1
2
1
2
2
1
1
1
1
2
2
2
1
1
1
1
2
2
2
2
1
2
2
2
1
1
1
1
1
1
1
1
1
2
1
2
2
2
1
1
1
1
1
1
1
1
1
2
1
2
2
2
2
2
2
2
2
2
2
2
3
1
1
1
1
1
1
1
1
1
1
1
2
1
1
1
2
2
2
2
2
1
1
1
2
1
2
2
1
2
2
1
2
1
1
1
2
3
3
2
3
1 total
9
8
10
8
14
7
11
13
11
11
8
8
10
7
11
10
11
12
11
6
7
8
10
3
2
2
2
3
3
3
2
3
2
2
3
2
3
3
3
3
Posttest Data
Enumerat io P recision n of st eps of st eps
Looping Measurem ent
Use of examples
Completeness of inst ructions total
1
2
2
2
3
1
2
1
2
1
2
2
2
1
1
1
2
1
1
1
2
1
2
2
3
3
2
2
3
1
1
1
1
1
2
2
12
8
9
8
13
10
2
2
1
3
3
3
2
1
2
1
1
1
2
3
1
1
1
1
2
3
2
1
1
1
1
1
1
2
1
2
1
2
2
3
3
8
11
7
11
12
15
15
3
2
2
2
3
3
2
2
3
2
2
1
1
1
2
1
3
2
1
1
2
2
1
1
2
3
2
1
1
2
1
2
1
2
2
2
3
2
1
2
3
2
2
2
3
3
3
2
2
2
9
10
14
11
11
11
15
15
16
11
Mean
Stdev
2.1
0.6
1.9
0.5
1.0
0.2
1.6
0.5
1.3
0.5
1.7
0.7
Experimental Pretest vs. Experimental Posttest: sig at 0.05
Probability Value (p <): sig at 0.01
9.6
2.1
2.4
0.7
2.2
0.7
1.5
0.7
1.6
0.7
1.7
0.8
2.0
0.7
11.4
2.6
EnumSt ep P recSt ep Looping Measure Examples Completeness T otal
0.02487 0.0249 0.0023 0.37346 0.0127 0.0347383
0.002

Are there any similarities between the envelope activity and programming?
“You have to do the steps very carefully”
Are there any differences between the envelope activity and programming?
“It is easier to see the result when you tell a robot”
“Both big things need to be broken down into little steps”
“It is more fun to write steps on the computer than with pencils”
“I had to tell a thing that knows nothing how to do something”
“I am better at thinking than a robot…they just do what they are told”

• Robotics seems to have a promising potential impact on academic achievement
• Robotics impact on attitudes is difficult to measure; current results suggest that the impacts are limited to specific areas.
• More research is needed into long term effects

• Robotics seems to have a potential impact on learning of robotics concepts and principles based on pre to post test scores
• More research is needed into long term effects
• Intervention seems to increase interest in STEM but it is difficult to measure directly and mathematical interest is particularly challenging
• Embedded assessment is a promising area of investigation for robotics activities

“We have not succeeded in answering all of your problems. The answers we have found only serve to raise a whole set of new questions. In some ways, we feel we are as confused as ever, but we believe we are confused on a higher level and about more important things.”
Omni Magazine, 1992