>> Roy Zimmermann: Welcome, everybody online and face-to-face. I'm Roy Zimmerman. I'm the
director of education and scholarly communication at Microsoft Research. I am delighted today to
present Maria Langworthy here to share with us some very interesting research. Before I turn the
podium and microphone over to Maria, I think it's interesting to discuss a typical point of confusion in
education technology. And it's a language function more than anything, and it's different from the
technology industry. In technology and education, we often confuse updates with upgrades, and those
things mean different things in the education space than they do in the technology space. So, an update
in the education world is let's add computers and infrastructure to a school and modernize that
capacity. And that doesn't suggest necessarily an improvement in the quality of what's happening
within the confines of the education structure, whether it's bricks and mortar or virtual or whatever. It
just means that there are new tools that are being introduced to modernize whatever the facilities are.
But if you don't include teacher training on those kinds of devices, if you don't include a plan for
maintenance of those devices, if you don't include ways to make sure that students know how to use
those devices in effective ways, then you're really just modernizing without improving. Upgrade means
that you're doing both the improvement of the infrastructure but also improving the overall quality to
the education ecosystem, and I think that Maria's research will help us better understand why it is
important to understand the difference between an upgrade to education versus just an update to
education. So with that, I'll turn the floor over to Maria.
>> Maria Langworthy: Thank you, Roy. And first let me also thank Roy for setting this up for me. He's
known me for a while, and we're like, "Does anybody at MSR know about this project?" And very few
people outside of Anthony Salcedo's group kind of do very much, and it's big global research, so why not
share it? But before I get started, since we're such a small group and we've got an hour and a half, if you
guys all have an hour and a half, let's have it as a dialogue more than as a monologue and chat just a bit
about who we are all of us and where our interests in this kind of lies. And are there very many people
online yet?
>> Roy Zimmermann: I have got two.
>> Maria Langworthy: Okay. Then we'll probably have to just have them listen. I've got the mikes
turned on so whoever it is online out there in the sphere can hear at least who we are. I'll start off. I'm
Maria. I am the chief research officer for a guy named Michael Fullan, who's a global policy adviser on
education. He's out of Toronto, but he works with Australia, he was part of Tony Blair's turnaround of
the U.K. education system, and we're working on some big projects together in ten countries. But
before that, I mostly ran the global research project for Microsoft, and for the last couple of years
before that I was an FTE here, running global research in the corporate market research group, so
around government relations, citizenship programs, things like that. I'm a Ph.D. in sociology and not indepth technological expertise. So that leads me to what I'll talk about today. Do you want to just go
around and say a little bit about who you are and what your interest in sort of education is? And if you
could give a sentence or so about what your -- what the main things when you think about the future of
education and learning that you think are sort of, you know, the dominant things that are going to
change. Easy, yeah.
>>: Dan Robbins, program manager of strategic prototyping, creating demonstration space in the EBC
around the future of education. And I would -- so the question was what's going to change?
>> Maria Langworthy: Yeah. What do you see as sort of the major things that are going to shift in
education and learning? But mostly in formal education in schools in K through 12 in particular, if you
have any ideas about that.
>>: If I put on my negative hat, I would say that there will be more haves and have-nots, and there will
be more bifurcation and more aspiration. If I put on my positive hat, I'll say I would hope that all this
stuff around common core will work itself out and that it will be a happy world where people
understand the difference between pedagogy, curriculum, and lesson plans, and we'll empower
teachers again.
>> Maria Langworthy: Mm-hmm. Yeah.
>>: I'm [indiscernible]. I'm a Ph.D. student at Georgia Tech. First-year Ph.D. student, so I just started in
the field of education and technology, just took a course and started understanding about the field.
Don't have much background in education apart from my own education, which has been
[indiscernible]. I won't comment on the future of education, but my interest in this field is about
[indiscernible] learning, so how it happens in K-12 is something I would like to study further, but right
now I'm doing it in the context of undergraduate education in Georgia Tech itself, so computer science
courses, per se. So we're focusing on those courses and how [indiscernible] technology could facilitate
better learning of computer science concepts.
>> Maria Langworthy: Very cool.
>>: Eric [indiscernible]. I'm actually an intern at Microsoft Research, but I'm normally a Ph.D. student at
Carnegie Mellon. And I'm actually here working on human analytics, which is completely unrelated to
education but I've never worked in education back home.
>> Maria Langworthy: Not at all. It could be vastly related, yeah.
>>: Yeah, yeah. I would like it to be more related. And I also sort of more subscribe to the more cynical
view of the future of education, but what I would love to see it move into is more where technology
means more of a sort of a catalyst or accelerator that assists the teacher in instructing in the traditional
setting, rather than being sort of the central focus as it tends to be.
>> Maria Langworthy: Cool, cool. Are you familiar with the Glass Labs initiative?
>>: Yes.
>>: Hi. I'm Jonathan Grudin. I'm here at MSR. I've been working the last few years on K-12 education,
mostly with [indiscernible] Gupta talking about [indiscernible] a lot, and I'm generally very optimistic
about the future of K-12 education. The one drawback being the withdrawal of state support for
professional development in teachers. That's a problem, but I think that the changes that I see here,
one is, as Dan mentioned, common core. I think, as that takes hold. Then I think the technology side,
one-to-one going to -- as schools go to one device per student, I think it's going to be a huge change.
>> Maria Langworthy: Do you work mostly focused on the United States or globally?
>>: Well, I'm focused most of the schools that you have spent time in and visited are in the U.S. but I've
been looking at and working with some of our essentially I've been looking at one-to-one deployments
and wherever they court.
>> Maria Langworthy: Interesting. I'd love to talk to you more about that.
>>: Lucy Vanderwende. I'm in the natural language processing group. I am interested in education
from the natural language processing point of view. We process text. As learners, we read a lot of text,
and I'm interested in developing tools to support learning from text so that we could support reasoning
over that text, because we all read so much. It's very hard to remember all the things we need to. It's
down deep in our memory, but how do I help people bring it up so that they can piece together the
information that they're being given? And then specifically to support self-learning, because, you know,
as long as we are in school, the teacher or professors is going to give us quizzes, and kind of test our
knowledge, give us problems that test our knowledge, but how do I do that outside of the traditional
settings so that I can support lifelong learning? Then personally I have an interest because I have two
kids. They're 14 and 11 1/2, and they both are at schools where they have a device per kid, so I'm
watching this. It's very interesting. So there's good stuff. There's stuff that's really challenging for
them.
>> Maria Langworthy: That's right.
>>: It actually has to do with the amount that they can read on a screen is smaller than having a book
open, and for people with processing difficulties, it's actually kind of an interesting challenge.
>> Maria Langworthy: Interesting, yeah.
>>: Igor Labutov. I work with Lucy. I'm also a Ph.D. student at Cornell. So, I work on similar problems in
sort of -- I recently started in this direction, looking specifically at sort of developing tools for scaling
personalized learning, sort of, to large classrooms, in a sense personalizing it to sort of ad hoc
classrooms, to people who want to learn on their own. And sort of I want to look at it sort of in a holistic
way, to look at it in the spectrum of curational material to assessment all way sort of. And this is the
master loop that Sumit and Lucy have kind of been a part of, and this summer I'll be working on
question generation with Lucy and Sumit and how to sort of look at text and produce interesting, high
level questions that would sort of allow people for purposes of assessment and at the same time kind of
thinking, to get people to think at a high level. It's more than just factual questions. So that's one of the
directions I'm interested in.
>>: I'll say that virtually, Sumit is sitting right here. So in addition to the mastery loop, we're developed
a tool called Power Creating, where it is very much can I amplify the teacher's ability to create lots of
answers all at the same time? So the teacher is first and foremost, but we're building tools that will
allow them to kind of do so more efficiently and effectively.
>> Maria Langworthy: Does Anthony's team know about that?
>>: Yeah, yeah.
>> Maria Langworthy: Very cool. Very cool and very related to some of the stuff I will be talking about
in terms of the shift in what students are supposed to be producing as they learn. So very cool. As I go
through some of what we've been working on -- please. Roy, do you want to say anything? I think most
people in here probably know who you are, but do you have any particular research interests, and
what's your take on the future of learning?
>> Roy Zimmermann: I think my take is close to Igor's. I think that the potential personalization is the
most exciting thing emerging in education, but it's also one of the scariest because the way twenty years
ago or fifteen years ago when I was doing my Ph.D. research, the mythology was you put a computer or
a device in everyone's hands and we've democratized learning, right? And we clearly know that that's
not the case. In fact, there can be as many, if not more, harmful repercussions from that approach as
beneficial. And I think the potential for personalization is the most exciting thing around what's
happening in technology and education, but it's always the most dangerous because we know that there
are already efforts out there that are sort of superficial, goes at personalization and if they manage the
sell themselves as the solution I think that we're just going to be right where we were fifteen, twenty
years ago with computers as the solution.
>> Maria Langworthy: And it's a lot about content delivery rather than these higher-order skills, and it
also has the potential to increase the inequality, because those people who are already driven can go so
far.
>>: The challenge is how do you personalize the content that people are learning as well as the skills
that you're going to talk about to make them successful in the 21st century workforce?
>> Maria Langworthy: Exactly. Exactly. Cool. So as I talk through some of this stuff, let's have, again, a
dialogue, and anything I say, feel free to question both the content, whether it makes sense or not, but
feel free to also question kind of the underlying theory and the theory of action behind it. So, I'm going
to talk about a research project that we did but put it in the context of sort of some facts that you may
or may not already be quite aware of. The larger picture. This whole project and my work is more at
the whole system, the social level. And what this is showing us is the unemployment gap between
youth between 15 to 24-year-olds and the adult population in lot of different countries. You see youth
unemployment in places like Spain up above 50 percent. You see there's a great McKinsey study on the
path from education to employment and how our education systems are not preparing people, even in
higher education, for the leap into the workplace. The skills aren't there that are needed by the
workplace today. Is this kind of data pretty familiar to everybody? This kind of gap? Good. I'll show
you something new. So this is when you talk to -- I've been talking to the last couple of years to a lot of
people working in ministries of education very much focused on K-12 all around the word. And this is
the problem that has got them really freaked out is that their education systems no longer connect with
the realities of life and work today. They're preparing kids, in a way, for a life that no longer exists. And
all surveys of employers, private sector, public sector, are saying that the people who are coming out of
the education system don't have those skills that they need for their work. Another really interesting
data point is how engaged students are in school at different age levels, so kindergarten up through
twelfth grade. And when they start school -- anybody have young kids? When they start school, it's
pretty much, you know, drinking from a fire hose and they're pretty excited about it. As they get -master basic skills and stuff, it gets harder and harder to engage them, and then they hit the nadir of
puberty and it all goes to crap.
>>: Is this the Western view or?
>> Maria Langworthy: This is from Canada. But these data patterns have been repeated in various
studies in the United States and in Europe, and the same patterns hold, that pretty much the peak of
disengagement happens in lower secondary school, right, and that you have basically, you know, well
below 40 percent of your students in eighth, ninth, tenth grades, really cognitively disengaged from the
learning experience, and again, part of this could be probably be attributed to puberty, but there's a lot
of trends that happen at this age range that really, really force teachers and students to kind of be
pushed mentally, if not physically, out of the schooling experience. Standardized tests start to hit, a lot
of pressure. There's much more pressure preparing kids for standardized exams for college, entrance
exams, and the packaging of education and learning becomes much more tightly confined and
constrained at this age grade -- at this age ranges. And this is also where I see dropouts starting to
happen so you might see a little bit of a bump but that's because some people have already left. So
really a crisis, you know -- yeah?
>>: This may be irrelevant, but having kids in that age range, it's also where the content of what they're
learning at school has almost nothing to do with what their parents are doing or what their parents can
even talk about. So it's kind of interesting in that respect that suddenly ->> Maria Langworthy: Exactly. And by this time -- they didn't have devices -- they're really awake to all
of the learning opportunities outside of school, all the content resources, and they really start to
question whether the teacher needs to be the source of knowledge any more and whether, you know,
whether what that teacher is saying at the front of the class is the whole story.
>>: How was the engagement measured?
>> Maria Langworthy: Generally these kinds of studies -- they're called student engagement surveys,
and they're a series of items on a survey that ask various dimensions of engagement, and they've been
tested over the last 20 or 30 years. They're pretty, you know, robust.
>>: Has someone also measured the interest in their future or what they want to do or, you know, their
active interest in thinking about what they want to do at or around this time, in ninth and tenth? It
leads to probably disengagement because the material is sort of not satisfying their curiosity as to what
they want to become and is not helping them with that.
>> Maria Langworthy: There's a little bit of research emerging. A friend of mine, Russ Quaglia, he's got
an institute in Maine, but he's doing work internationally looking at student aspirations and student
voice. And where he's going into schools and working with school leaders, with teachers, to give
students greater voice in the everyday of schools but also in the classroom, you see engagement starting
to increase very dramatically. You see dropouts. You see retention. You see people showing up at
school every day. People showing up at school every day, both students and teachers, is a huge
problem. I was at a conference with some of the district leaders from Federal Way in the spring on a
Friday morning. It was sunny and the weather prediction was that it was going to be nice, warm
weather. And in their high schools they had about 30 percent of their teachers call in sick. I mean, you
know, yeah. Other interests, you know. So it's this boringness of school really hits, and it hits both
teachers and students at this age. And I've done research here at the International School in Bellevue,
and the kids will say, you know, "I want to learn about stuff that is connected to what I want to do
outside of school to my future career."
>>: This meme of relevancy is not a new meme. It seems like it comes back every few decades.
>> Maria Langworthy: Do you think?
>>: Yeah. I mean --
>> Maria Langworthy: That's where I would question ->>: Even if you go to someone like Rousseau who says put them out in the woods because that's
relevant to life. I mean, this is not new.
>> Maria Langworthy: Yeah but ->>: So how is this a new dialogue about relevancy? Part of is the advent of digital, but part of it is the
shift in our economies. And, you know, kids used to, 60, 70, 100 years ago, kids all left school once they
had basic reading and writing because it wasn't relevant beyond that because they were going to go be
farmers the rest of their lives, you know. Then we shifted into the twentieth century, where the mass of
middle class jobs, manufacturing, very standardized. They needed basic skills and the development of
basic skills, high levels of literacy and numeracy, were what were required by our society and our
economies for functioning. And, you know, the real focus was on students receiving content and being
able to demonstrate that they could master that content, the basic textbook model. But the teacher
and the textbook combined, their function was to deliver that information. In the 21st century, our
economies are much more based on our ability to produce knowledge, knowledge-based products, IP.
Microsoft is the heart of that, right, about new ideas, new programs, services, et cetera, but all ideagenerated, and that requires different skill sets, right? That requires a different orientation to what
knowledge is, a different epistemological stance about is knowledge something that is concrete,
packaged, or is it something that's malleable that we can create? So here, what the shift is moving
towards is the development of these kinds of what are call higher-order skills, 21st century skills. We're
now starting to call them deeper learning competencies and the creation of knowledge-based products.
How many of your kids, if they're in school, in high school, are really having to produce things? Movies,
videos, ideas, real-world problem-solving? It's this shift, yeah. Did you have a question? Yeah, but it's
early stages.
And the quality of these projects and how they're assessed and how legitimate they're perceived to be
in the eyes of all stakeholders is very much in flux right now, but that's kind of the context in which we
started this. Yeah?
>>: I'm sure you're aware of this, but implicit in that change is that even for other skills, like if you're in
crafts or even if you're in farming -- one of my nieces is in farming -- it's amazing how much they want -they need to know about. They need these higher-order skills about marketing, about figuring out what
sorts of problem they're going to be solving next ->> Maria Langworthy: They need innovation.
>>: -- what products they'll be selling next year.
>> Maria Langworthy: Yeah, the digital world opens up innovation in every single work domain, right?
>>: And requires those skills, even if you're not particularly focused on producing a digital product.
>> Maria Langworthy: That's right. Absolutely. I mean, you could talk about law, about people who
work in government and create social programs. All of those are, you know, it's not just technical IP.
Yeah, exactly. But the assumption has always been, over the last hundred years, that the people who
get these skills get them in college and that college is the place where you really build these types of
classes. And so the big rush -- and you see the income gaps between college graduates and high school
graduates. The reality of it is in the United States we don't have enough seats in universities to possibly
fulfill getting all those students who need these skills to wait for them to go to college. There's a huge
gap between the number of people who want to go to college, the number of people applying, and the
actual number of seats in our universities.
So, in that sense -- and problem is not going to be solved because of the cost of higher education for 20
or 50 years, so we've got a generation there. Like 70 percent of secondary students in Australia go on to
college, and they have the seat, but they've been working on it a long time.
>>: I think MOOC tide spells for all of that demand and lack of ->> Maria Langworthy: It has. I mean, in a way it's going to be a necessity to have more MOOC-type
orientations, but that means -- I mean, the research I'm seeing on MOOCs is that they have incredible
dropout rates. 90, 95 percent of people who start them leave and don't finish the courses. So MOOCs,
yes, but we've got to still figure out ways of making MOOCs more engaging, making them relevant, and
legitimate in terms of credentialing and badging, but also figuring out how to give people the skills to
stay engaged in MOOCs regardless of whether they have a personal hand to hold or a teacher who
shows up every day because it is about scale.
But this whole project and my whole work is about we don't have to wait until college. We don't have
to wait for MOOCs. A lot of these skills -- and I see it all around the world. If you design the learning
experience in a certain kind if way, students can get these skills at that -- in that secondary school range.
I have seen students demonstrating these skills as early as second grade, you know, really powerful
collaboration skills, really strong self-regulation skills, so it's really about how we construct the learning
experience, I think, as much as it is, you know, technology or the tail end of the picture.
This project was kind of emerged from this recognition that, because of these changes in economies and
societies, we need to think about whole system renewal, whole education systems. Yes, there's huge
opportunities emerging for informal learning, learning outside of the school, but the reality is and the
inequality issue is very much based on we've got to get a hundred percent of populations engaged in a
different kind of learning to make sure we don't have, you know, the French Revolution of the 21st
century on our hands in about twenty years.
So, I will also say contextualizing this, what we set out to do -- well, I'll tell you. You're all inside the loop
of Microsoft. So Microsoft, Anthony's team, came to me and they at first wanted to do a program
evaluation of their Partners in Learning group, and they wanted to do that for thought leadership and
education leaders, policy makers all around the world. And I looked at it and I said, you know, We can
do a program evaluation and say, "Look at the impact all our programs have on teachers all around the
world," and those policy makers will look at it and say, "What nice marketing materials" and toss it out,
and it wouldn't have the impact they wanted.
And they were willing to spend a couple million dollars on this research project. I said instead, let's
make it much more of a public research project, and let's go out and ask them and engage them from
the beginning, these policy makers, in what questions they're interested in and how they wanted to -what was really at the forefront of their minds.
So that's how the project got started, but it was always and always has been a project that's really
designed around engagement with and repositioning Microsoft as a company in the education space as
a company that's really thinking deeply about the problems of education and on the innovation cusp of
that. So what we heard from those people was concern about, you know, they've been investing in
technology for 20, 30 years. There hasn't been any clear impact on educational outcomes based on that
investment. That's one question they had. The second question was really about how do they
transform their education systems to develop these kinds of 21st century skills in their students. That
was the major thing. So what we designed was this ITL, innovative teaching and learning research
project. We partnered with SRI International in designing the whole thing, but it really had three
essential questions. If teaching practices change and become more innovative, will students actually
develop these kinds of 21st century skills? Can we get evidence around that? And then what kind of
school and education system factors can drive higher adoption of innovative teaching practices? Really
looking at it holistically, systematically, from the perspective, kind of, of the big education policy maker.
It was sponsored initially by Microsoft Partners in Learning. So as part of this, the way we designed it,
we designed a bunch of different research methods to really capture data on a lot of different variables
at the policy level, the types of professional development programs offered by teachers, teachers' and
schools' attitudes, the school culture, the ICT infrastructure, information communications technology,
infrastructure in schools. And we really worked hard to define this concept of what we meant by
innovative teaching practice. It wasn't just the use of technology by teachers. We looked at how
technology was used by teachers and by students and looked at whether it was being used in very basic
ways versus higher-level ways that develop these kinds of skills. And I'll get into that in more depth.
But we also looking at what kind of pedagogical practices teachers were being used and whether they
were developing these kinds of skills, problem solving, et cetera. And we based all this on a literature
review of some of the factors pedagogically that have been shown through prior research to have a
significant impact on student learning outcomes. This was mostly developed by SRI International, and
we defined these skills for life and work today the kinds of skills that students -- employers are saying
that they need to see, you know. This is kind of an uber set of these 21st century skills. Are you guys
hearing about this in public dialogues around education today in schools and stuff? Okay. Varying
degrees of familiarity with what these things are. So we started out the ITL project in 8 countries,
partnering with some kind of national partners in each country so that we could make sure that there
would be a tie-in back to policy. Went from Brunei and Senegal and Australia, Indonesia, Russia, all over
the place. We didn't have partners in the United States. We did have one. The CEO of the foundation
that was going to fund it changed roles and they got a new CEO and dropped out halfway through, so
unfortunately, the U.S. was not a big participant in it. But it was very much a partnership, and in each
country we had a research partner, usually a university that specialized in this stuff, that carried out the
research methods in each country. And those methods that we designed are now being used in
upwards of 45 countries, and I'll talk more about that throughout later on.
But we developed ->>: I’m sorry. It says "The Schools Network." Is that -- I know that there's a small group of schools
called "The Schools." Is it The Schools?
>> Maria Langworthy: No. There was an entity at that time called SSAT, Special Schools and Academies
Trust, that represented a whole bunch of the academies in the U.K. It then changed its name to The
Schools Network, and so that's what they are. Then they went bankrupt when Cameron became prime
minister, and now they're reconstituting themselves and they're SSAT again, but they don't have very
much funding. So it's, you know, these kinds of projects -- it's very interesting. Like negotiating
partnerships with universities and doing a tender, an open tender bid, for who's going to win the
project.
In Mexico it was particularly interesting, and no, we could not do the best friend of the guy who was
funding, you know, and it was a real imbroglio. In Brunei it was super interesting. You basically met
once with the minister, and he said yes, and you got a check. But it was a very interesting experience,
just that piece of it.
>>: This was not in collaboration with the Bill and Melinda Gates.
>> Maria Langworthy: No, no. This was still at a time when there was really fine lines between the
foundation and the company, so all this was purely Microsoft. In fact, I've only presented this once at
the Gates Foundation.
>>: [indiscernible]
>> Maria Langworthy: No, I can't remember their names, sorry. Off the top of my head. So we
developed a set of research methods to investigate, kind of, these patterns at all layers of the system,
from doing interviews with policy makers, school leaders, teachers. We created a set of surveys for
really having teachers reflect on their own innovative teaching practices in a way we could quantify.
There were a lot of classroom observations. These were all carried out by our local research partners,
and our SRI teams and I would meet with those international research teams twice a year for a week or
so to really delve into those research methods, training everybody in the methods, and then each year
share the results out and refine the methods as well.
These learning activity rubrics and student work rubrics were one of the most powerful things that we
came up with in the program.
They were rubrics for defining what those 21st century skills were and whether the 21st century skills
were designed into learning activities or lessons and the degree to which they were and whether -- the
degree to which they showed up in student work products, okay, whether it was writing, what
presentation, et cetera.
We did focus groups. You can see some of the numbers involved. So what did we find? First, the
question of were innovative teaching practices associated with students' demonstration of these kinds
of skills? So we analyzed the learning activities and we collected, you know, about a thousand learning
activities or lesson plans or unit plans from all these countries, and we trained a group of teachers in
each country to code those lessons, those rubrics, on a 1 to 4 scale, 1 meaning there was no opportunity
in the lesson at all for, say, kids to practice collaboration. A 4 meaning there was a very rich and
complex way, and these rubrics really provided a framework for saying, you know, what is a higher level
of collaboration? It requires shared decision making, a shared work product. It means working on
substantive decisions together.
I use the analogy of kids working together to put up posters on related themes versus three or four
thousand developers working together to produce Windows 8 over, you know, three to four years.
That's a different level of collaboration.
And most of the collaboration experienced by kids in schools way over here and so these rubrics kind of
got to capture that at that conceptual level. So these were powerful tools in and of themselves, and
what we see here, and these were taken from the more innovative schools in our sample or the
innovative schools that were selected by our research teams to represent sort of the forefront of
innovation, and on average most of these learning activities didn't get above a two so this were very
weak opportunities for students to deadly weapon develop these skills in the apologies that they were
experiencing.
This is interesting because this is showing the rubrics' coding for the learning activity activities in relation
to the coding of the student work that was produced. So for every learning activity we collected, we
also collected six samples of student work that was done in response to those learning activities and
coded those on the same parallel set of rubrics. So what you see is that when learning activities are
designed in a way that asks for students to develop these skills of collaboration, of self-regulation, of
knowledge construction, student work can and does demonstrate these skills. 85 percent of the scores
of the student work were explained by the content of what was asked to do in the learning activities. So
this suggests that it is the design of the learning activities that is really going to determine whether or
not students develop these skills, you know, whether it's a MOOC or a, you know, sixth grade class on
geometry and studying what a diameter means, is the way that lesson is designed. Does it call on kids to
collaborate? Does it call on them to apply their learning to a real-world situation for an authentic
audience?
So these kinds of questions were embedded in this. What we heard from teachers is that they
understood that these kinds of skills were needed. They didn't have a clear understanding of what
those skills really meant, and they didn't understand how to apply them in their classrooms tomorrow,
which is every teacher's problem: What to do with my kids tomorrow. And so it was -- they weren't
being provided with the definition of what these skills were, but, you know, you can see here almost all
of the examples are down here with very low levels of innovation. So why was this not happening here?
There's a host of different explanations here. One of which was around them simply note really
understanding what these things were and how to embed them in the design of their own learning
activities with their students. But we also see it in how teachers were asking students to use
technology. When technology existed in classrooms, teachers were largely asking students to use the
technology in ways that were very basic uses of technology: Asking them to practice routine skills, using
it for content delivery. To take tests, turn in homework, in ways that really did not teenage advantage
of the potential for technology for kids to create, to collaborate.
>>: What age -- is this just all of the work?
>> Maria Langworthy: All of the data from students is between students that are ages 11 to 13 with a
few 14-year-olds, but that was -- it was targeting this kind of lower secondary area. And so what you
see here, you know, is there are a few one-to-one environments in this, but most places most around
the world, you know, you have a three-to-one ratio or a five-to-one ratio, even today in Northern
Europe, of students to computers. Any other questions?
>>: So fifteen percent means is that like fifteen percent of their use of ICT? Or.
>> Maria Langworthy: Yeah so teachers the question one kind of how often do you ask your students to
use technology in these kinds of ways and they could check multiple things and so, you know, fist, they
weren't asking them to use technology very much. The highest category was 36 percent asking them to
do searches on technology. You see a hijack amount of technology in schools that is not used. Of
computers sitting on the sidelines.
>>: So 15 percent means that only one in 7 teachers ever asked them to analyze data of information.
>> Maria Langworthy: Yeah, yes ->>: Is that what it means or it means of the times they asked them to use ICT, 15 percent of those.>> Maria Langworthy: Yeah. No, it was that, sorry. It was, yeah, when they asked. How they asked ->>: So it could have been they asked once a year and 15 percent of that ->> Maria Langworthy: Yeah. In fact, OECD has done some research, and it was about 2010 I think, the
data, and it was part of PISA. But they asked them sort of what's the frequency with which students
actually use computers in schools? And it was -- in Northern Europe it was like one to two hours a week
that they actually used the technology in schools because it's just not part of the process yet. I mean,
this is changing and it's changing very rapidly and it's changing very rapidly in one-to-one contexts but ->>: I wonder if it would be interesting to compare this chart against how students you technology at
their homes or outside the school.
>> Maria Langworthy: Yeah.
>>: And if you see there is a shift or their using it in the same pattern, so it's maybe something that the
school is sort of ->> Maria Langworthy: I have seen research from South Korea that shows that when students use
technology for learning purposes outside of the school, not just for social or for content consumption, it
has a pretty dramatic impact on their learning outcomes but, you know, it's making that leap, getting
kids to use it for learning purposes outside of school.
>>: Does this include tutoring centers or just.
>> Maria Langworthy: This?
>>: Yes.
>> Maria Langworthy: No, this is just ->>: You said South Korea so I think of the tutors center.
>> Maria Langworthy: Right, right. I don't know.
>>: As part of the educational experience than the actual schools.
>> Maria Langworthy: Yeah, yeah could be. But what was funny about this is, so you would think this
would differ, right, a lot between someplace like Finland that's one of the top scoring countries in the
world on international tests of students versus Senegal, right? And I was sitting in Dakar with the
research teams and there was a guy from Senegal describing his observations on classrooms where the
teacher stood at the front of the class with the computer at a podium and projected things on screen.
So it was the only substantive difference between that and him writing things on the blackboard was,
you know, the technology device. There was no change in the pedagogical approach. And the guy from
Finland leaned over and he said, "That's exactly what I saw in Finland," so while there's obvious
differences in the density of technology in schools in those two places, the patterns were pretty similar.
>>: What you just said was very interesting, that there's some evidence that if they used it for schoolrelated purposes outside of school that it made a difference. I'm wondering whether that was
orthogonal to one-to-one, because if you don't have one-to-one, then it's very hard for the teachers to
assign anything outside of school.
>> Maria Langworthy: That's right.
>>: So on the one hand, that difference could mostly come from one-to-one. On the other hand, it
could come from other situations. But then you've got the confound that the kids who are not assigned,
they're doing it despite the fact that it's not assigned, so they're probably stronger motivated.
>> Maria Langworthy: The students in the first place, yeah. It's very hard to break out.
>>: If it is in one-to-one schools that that's happening and it's then it could be more -- so anyway I'd be
interested.
>> Maria Langworthy: It's very hard in those situations to break out the socioeconomic variables in the
larger context.
>>: Yeah but I'd be interested to learn more about that.
>> Maria Langworthy: I mean the one-to-one and the equity issue is huge, you know, about and how it's
handled and BYOD often increasing the equity gap and perceptions, et cetera. It's a big issue. Another
thing that we saw about why this kind of more innovative teaching wasn't happening very much was
actually in the language that teachers and school leaders used, and in classroom observations this was
backed up. They would talk about 21st century skills. They would be using technology. They would use
the word "innovation," but they never talked about pedagogy and broadened how they were teaching
and what the learning experience, how that could change. And we saw this over and over and over
again, that they weren't really redesigning learning. They weren't applying this stuff in existing patterns
and existing processes of teaching and learning, so it was in their mental constructs about what teaching
and learning really, really was. It wasn't changing the qualitative experience very much.
So then we turned to, okay, there were schools that had slightly more innovative teaching practices and
those that had slightly less. So what were the variables ->>: Could you please explain a little more of what you mean by more innovative practices?
>> Maria Langworthy: yes. We when we defined innovative teaching practices it was those three
categories of how they were using technology, whether they were extending the learning beyond the
classroom, so to speak, and whether they were using more student-centered pedagogies that develop
these kinds of skills, so that's loosely what we mean by innovative teaching practices and what later on
I'll talk to as new pedagogies. We have to -- you know, there is a branding, marketing world, so we have
to change the terminology every three years. So there's new terms.
The next question was: What were the school- and system-level factors that were related to higher
levels of innovative teaching? Because we'd established that innovative teaching can produce more of
these kinds of skills. One thing that we found was that there was a lot more variation in innovative
teaching practices within schools than across schools. Even though we had sampled for having a set of
innovative schools and a set of other schools that were similar socioeconomically but didn't have -- that
weren't known for innovative teaching practices, and even between those we found more variation
within school on teaching practices than between schools.
And this is pretty standard, even in the United States, on other variables, that there's a huge amount
more variation within a school on teacher quality than there is between schools. So in every school,
there's going to be some very innovative teachers, and there's going to be some very laggard teachers,
and a lot in the middle.
>>: Does that suggest that the administrator in the school and the school district itself fairly neutral
variable in what's happening from an innovative perspective in the schools? It's really just up to the
teachers?
>> Maria Langworthy: Well, it does now, but what we'll see is that there's a huge impact when the
leadership, when the system, when the policy starts to support more innovative teaching, it starts to
pull everybody up.
Right now what you see was innovative practices rather than innovative schools, right? You go into very
innovative schools and you still see some teachers that are hugely resistant. And depending on the
context set by the leadership, and there's one variable, it turns out, that is the most powerful variable in
pushing people up that innovation scale in the school and that is the level of collaboration between
teachers in a school. What?
>>: -- or across schools?
>> Maria Langworthy: Within schools, because if you have the most innovative teachers across a lot of
different schools collaborating with each other, they get more and more and more innovative, but that
doesn't pull along the masses of teachers behind them necessarily. They become outcasts, and we see
that a lot actually in the Partners in Learning program, that teachers who are very innovative that exist
within a noninnovative environment become very much outcasts.
I had one teacher -- I don't know few guys have heard of Partners in Learning. The Partners in Learning
program is an international program that Microsoft has for teachers in schools all around the world, the
most innovative ones, and you see some teachers that come from very innovative schools that have a
lot of support. I talked to a teacher at the global forum in Barcelona this year who doesn't live in an
environment like that, and her pay got docked for her coming to this global forum. She had to take, you
know, a leave of absence basically and not get paid for coming and participating. Very much -- and, you
know, her principal says, "You keep doing this kind of stuff and you're, you know, on the way out."
>>: [indiscernible]
>> Maria Langworthy: Yeah, this was in Texas.
>>: Wow, okay. So no, not America, thanks.
>> Maria Langworthy: Yeah [laughter]
>> Maria Langworthy: I used to live in Texas. It was not in Austin, I will say that [laughter]. But what
you see here are schools where there's low frequency of teacher collaboration and high frequency of
collaboration, and we looked across a lot of school variables, all the ones that we gathered and this was
-- the level of collaboration between teachers in a school about teaching, not just them sitting around at
the lunch table. It's them collaborating on teaching, looking at student work together, talking about
how students are doing, talking about any kind of new teaching practices that they are -- and other
research supports this. Michael Fullan's work is all around this collaboration at the leadership level. I
mean, whenever you're trying to do something new, it's a risky proposition and denying it by yourself is
much harder than trying to do it in a context where other people are fairly supportive and collaborating
with you even if it goes wrong, which is it does in a lot of cases, but if you're in a context where other
teachers and your school leadership is supporting this, it gives you back against any kind of resistance by
students and especially by parents. Parents are the most conservative force in education in most of the
world. All teachers need somebody to defend them in parents come in and say," What are you doing?
I'm concerned about these standardized tests." Yeah?
>>: Is there any breakdown of this -- when the collaboration is within the subject matter or across
subject matter?
>> Maria Langworthy: We didn't study that, but I will say we did study levels of innovation by subject
domain, and math teachers are by far the least innovative, very much the least innovative teachers out
there. This is globally.
>>: There's a dark lining to this because the technology is going to enable that \[inaudible
00:00]\[indiscernible] it's going to enable good teachers in different schools to interact a lot more and
reduce their inclination or time or to work with other teachers in the same school, which we've seen at
the university level too, so.
>> Maria Langworthy: It's the same pattern you see with students. Good students, you know, work on
stuff independently, get in contact, collaborate online. Great teachers do the same thing. The teachers
we have in the Partners in Learning program, amazing, gives me shivers. These are just fantastic
teachers. That's why the context and the structure and the policy and programs, whether they support
teacher collaboration, whether they, you know, encourage innovation across the board, whether they
measure it, becomes a huge factor in the degree to which this kind of stuff spreads beyond just those
top two ->>: What?
>> Maria Langworthy: Whether they measure 21st century skills, whether they measure innovative
teaching, these kinds of things because -- and you'll see those in a second. Teachers right now say
they're not recognized if they do this in most schools.
>>: As opposed to Microsoft where collaboration is well known to be highly rewarded [laughter].
>> Maria Langworthy: Exactly.
>>: Teamwork [laughter] [cross-talk]
>> Maria Langworthy: Yeah now there's the [indiscernible] variable. Will it change? [laughter] But by
and far, when we looked at the system level, teachers and school leaders said these were the biggest
barriers to change, and those barriers were lack of professional development programs that really
supported teachers in shifting their practices and the way those professional development were
managed and designed and that teacher appraisals really did not include anything about whether the
teachers were using technology, whether the teachers were developing 21st century skills, those things
so it is really out there doing it on your own in most cases around the world for teachers doing this. This
is all starting to change, but this is what we heard then. But the number one variable is the way we
measure success of learn outcomes. It's the kind of standardized tests that focus only on literacy and
numeracy, only on pure content mastery. And this is a teacher or a school leader in England who said,
you know, my success is judged by examiners at Ofsted in England.
In A lot of systems around the world, they have school examiners that come in, they look at their scores,
they publish their scores on these standardized tests by what parents what. What children want is not
necessarily what I would call innovation because we're all working off models of how we learned, of
what school looked like, and those tend to be the biggest barriers to change.
I can't say enough about how our tests are run, how our assessments is -- not that those shouldn't be
done, but how they were done and the importance placed on them is driving our education systems in a
lot of places around the world.
Another piece is the type of professional development teachers participate in and are offered. What
you see here is teachers reporting on the type of professional development they did over the last two
years relative to their reported innovative teaching practices.
So teachers who participate in certain types of professional development tends of much higher levels of
innovative teaching practices. This is innovative teaching practice, right, which was measured in our
surveys based on their self-report of their own teaching practices. These are the types of professional
development. We controlled for the amount of time they spent in professional development.
What you see here is that these passive, traditional kinds of learning, listening to a lecture, you know,
those tend to produce not very big innovations in teaching practices. When teachers actually go out and
practice a new teaching method and conduct research on those teaching practices -- in places like
Finland, teachers are required, as part of their contracts, to do research on the impact of their teaching
practices on their students and to report publish, present those at conferences. Places like Singapore,
these are the places with the highest levels of teaching professional in the world, and they treat
teaching as a learning experience for teachers. They teach it -- I mean it's a profession that requires a
lot of reflection on what's working and that pushes innovation some too.
>>: Something is that seems little buried there. When you say planned lesson or unit, I mean, that
could include all of these things. I don't understand what that means.
>> Maria Langworthy: Yeah, it's one of the limitations of surveys with really standard categories on it.
But I think what people are meaning here is just the practice of taking the contents and how you're
going to put it into a package for your students.
>>: Because if you practice a new teaching method, it's part of planning a lesson, right? I don't
understand.
>> Maria Langworthy: Yeah.
>>: They don't seem separate to me.
>> Maria Langworthy: Yeah. No, it is a little bit messy, but in general, we're thinking a traditional
lesson. And I forget whether that response category on the survey had a little bit of a richer description.
>>: That could include collaboration, that could include research.
>> Maria Langworthy: Yeah, if it's done collaboratively, yeah, especially. But this piece in particular,
about the impact of teachers reflecting on their own practice and looking at it, is very much
corroborated by other research. There's some work by John Hattie that looked at 800 meta-analyses of
the teaching strategies that had the most impact on student learning outcomes over the last 30 years.
And he's produced a series of books that are all around the theme of visible learning, which is about
teachers researching and reflecting on the impact of their teaching practice on individual student's
growth.
I'll speak a little bit about that, but these were all coming out at the same time, suggesting it's that
metacognitive drive of are we making progress? and why is that progress happening? That’s really
essential. I mean, it's a far cry from teachers standing and delivering the content in a talk, kids taking
the test, and if students didn't get it, it's their fault. This is a saying there's got to be something more to
that picture, a lot more to that picture.
>>: It'd be interesting to see how that's changing at least in the U.S. now that it's 2011, because there's
just this huge common core -- because I mean a lot of schools just totally ->> Maria Langworthy: Yeah.
>>: And ICT is potentially just being dropped by the way. I mean, even though it's potentially, obviously,
a part of it, but it's not a priority because they're so worried about ->> Maria Langworthy: Common core.
>>: Right.
>> Maria Langworthy: I mean if you look at the survey, I think it was Gallup did a survey recently, the
number one reason in the United States why districts were investing in one-to-one was so that they
could run the common core assessments, you know [laughter]. It wasn't to change pedagogy. It wasn't
to give students access to more learning opportunities or content. It was so that they could all take the
exams.
>>: The usual practice in using what they're going to take the exam with before they take it, but yeah,
you're right. They can't rotate them through the lab and testing would know and so.
>> Maria Langworthy: Yeah, but on the other hand -- so this guy John Hattie, who is all about this visible
learning and teachers reflecting on their own practice. So I get Education Week, which is one of the
largest magazines for education in the United States, comes out weekly, and John Hattie is on the front
cover of like two a month out of that. And his visible learning kind of professional development
programs are taking off across the United States. So, you know, whether that becomes measured in
formal policy is not -- it's not there yet, but certainly it's, you know, becoming part of the conversation.
How do you put all this together in a big picture? There's no one silver bullet to shifting teaching and
learning practices, changing our schools. What this research and there's nothing -- if you look across
education research, there's nothing hugely new in this picture. A lot of findings have been found in
individual pieces of research in individuals countries. What this did was pull it all together saying, you
know, we've got to have ubiquitous technology for kids that's inevitable. But use it in a way that
supports their creative and their collaborative problem solving knowledge creation.
You've got to develop professional development programs for teachers that help them make this leap,
especially through collaborating. We have to have leadership that recognizes, supports, and helps
create a common vision of what this innovation looks like. And most importantly, we have to have a
system at policy level, assessments and appraisals and ways of measuring schools, teachers, students,
that take this stuff into account, that don't just measure the basics, because it's not ever going to be
taken seriously systemwide until you have some measures of this stuff. Questions? It's a much bigger
picture, kind of, at the systemic level than looking at, you know, how you can use individual technologies
to support growth. But the shift does very much support this idea of a mastery loop. The kind of
learning that kids are going to be doing is not just mastery of content, but it's mastery of these kinds of
skills.
The exciting part here is actually not the research, at least in my view. It's what we did next. So I was
doing all this, and at later stages, the Pearson Foundation became involved with the work as well and
became a sponsor. And most of the countries that joined, they funded the research in their country
themselves. So Microsoft kind of paid at the global level, but individual countries contributes.
So it really was a partnership, but at the same time, I was working with Partners in Learning, with some
of the leaders over there, and they were saying they had programs for schools especially, for teachers
internationally, which they wanted to start to make use of some of this.
So a colleague of mine over there, [indiscernible], a Finnish woman, now back in Finland. She said, "Why
don't we start using some of these methods with the schools in my program, the innovative schools?"
So the first year we just translated all the surveys into all the languages of the schools in her program -- I
think it was about 25 languages -- and ran those surveys in those schools and created individual
schooling reports measuring the degree of innovative teaching based on this method in their school.
Those schools came back and said," Oh, my gosh. This is incredibly powerful. It helps us understand
what innovative teaching really looks like on the ground in a way that's much more complex than just
using technology, and it's a powerful vehicle, having this report and this measure."
So based on that, Kati decided she was going to fund us creating a Partners in Learning school research
program which put all of this survey online for any school in the world to use for free and with an
automated reporting system. It was translated into about 34 languages, and it's taken off.
There's over a hundred thousand teachers all around the world who've taken these surveys in the last
couple of years. It's a big piece of the program, and each school gets, you know, their own reports.
There's about a 16-page report. I was showing this at the ministry of education in Sweden, and so this
one is in Swedish, but a mechanism for helping to develop that common language, that common picture
of -- do you speak Swedish? [laughter] I don't, so I have no idea of the quality, but usually we had
someone from the country review the translations.
The other piece of it though, because this is one piece of a component for shifting schools, shifting
systems, is some way of measuring teaching practices that incorporates this stuff.
Another piece was we needed professional learning, the professional development programs for
teachers. And we went back to -- I mean, we went back to this piece that's all about the design of the
learning activity that is going to shift both the practice and the learning outcomes for students.
And we went back to this piece and said, okay, they've got to have an opportunity to practice, to embed,
to collaborate and to research. And so we listened to that and we listened to all those teachers we
trained to code the student work and to code the learning activities. We heard all around the world
over a four-year time span that that experience of coding the work was the most powerful professional
learning experience that those teachers who did it had ever done.
So we kept hearing this and kept hearing this and kept hearing this, so the final year of the project we
transformed it from a purely research-based project into a professional development program for
teachers called 21st Century Learning Design that used the rubrics that we had designed to measure
innovative teaching and student outcomes on these skills, used those as the basis for a professional
learning program for teachers where they would come in and code some learning activities that were
sample learning activities and code student work and then start to collaboratively redefine and redesign
those learning activities for their own classes, for their own students. And we've seen this -- and part of
this emerged because teachers and schools all around the world were getting a hold of the rubrics -they're under Creative Commons -- And they would come back with these stories of how they were
meeting monthly to review their lesson plans and change them.
>>: The samples collected, was it written of what students produced, or was it observations of what
students were doing? What do you mean by?
>> Maria Langworthy: The student work? So in these types projects -- and this is a very important piece
of it -- there's no standardized way, usually, of the student -- it's like an essay. The students may get an
assignment to write an essay, but the content of that essay is going to be very, very diverse and openended, right?
No one student is going to write the same way. In the same way, the student work products for these
kinds of projects are very, very diverse, and the student voice in designing the projects are very deep in
that. So they might design a project that is solving a problem in Uganda. Another group might do a
water cleaning project in their own backyard. So there's no standardized way of assessing them.
That's why this kind of coding structure creates a framework for looking at a lot of diverse kinds of work
and looking at it by a specific set of categories that -- and those categories were those 21st century
skills.
>>: My question specifically is: What data were they coding? So what was the data that they collected
that they were coding?
>> Maria Langworthy: Right. We worked on different ways, but what we did was collect a set of
examples or exemplars of learning activities and student work that we would use in these professional
learning programs to give them -- because you can read the rubrics and you're like, "Interesting, kind of
makes some sense," but until you start applying them to real examples, they don't go, "Aha." And what
we've seen all around the world, it's only when teachers sit down with a real example and start to
interpret it and start to debate. I think this is a 3 on knowledge construction. I think this one is a 4 on
real-world problem-solving and innovation and why. That's when the aha moment goes off in a
teacher's head and they start to say, oh, if this lesson was redesigned to ask kids to do this, then it would
get to a higher level on the rubric. Is that so we created a set of examples and they would start with
that and in a very non-- because those weren't their examples those examples were not all great
examples. Some of them were very bad because we wanted them to see the difference between a 2
and a 4.
But then as part of professional development, then they start to bring in their own learning lessons and
their own students' work and start to apply that.
>>: I think -- are the samples -- is it a mix between like video or.
>> Maria Langworthy: It could be any of those things, right.
>>: It is any medium?
>> Maria Langworthy: Yeah.
>>: Or would they produce observation of when the students were working and specifically the
question is whether the these projects or the design activities were meant for individual learning, or was
it group activities where students were working together?
>> Maria Langworthy: Very good, very good.
So all of this was collected pretty much manually. That's the way schools today work. So one, it would
be a range for any one assignment or learning activity. There would be kind of a standard group. There
might be a lot of diversity in that, but learning activity by learning activity, you could have a huge range
of types of student work products, very qualitative, very open-ended. And that's because of the
problems with assessment of this kind of work. What was your other question in there? Okay.
Yeah again all of this not using technology to support the data collection or the assessment or any of it.
>>: How much of the value to the teacher in improving their teaching practices was on seeing the data
versus generating that data by doing the assessment themselves? And the reason I ask is you've seen
people like that who are working a lot on automatically generated assessments. There's tons of work
around automatically generating an assessment, and my worry is if you just show teachers the data,
then ->> Maria Langworthy: They can't make the interpretation. Yeah.
>>: I don't know if that's important or not. Do you think it's the process of actually going through the
student work?
>> Maria Langworthy: Absolutely I do think it is.
>>: So then what is the impact on this huge body of work on automatic assessment?
>> Maria Langworthy: It depends, and there are some cases in which it could be highly valued. If you
want teachers and students to really be able to do this stuff, these complex higher order skills, they have
to experience doing it and knowing what it means, so it can't be fully automated. Once they understand
in some depth, it can help, but I'll -- there's more after this that I want to go into because it addresses
this question very directly.
>>: That's why Sumit and Chuck and I designed the power grading to be teacher-centric, because as part
of -- we give them an assist for the assessing of the student work, because we also give them a place to
write feedback to the students. So that's one way in which the teacher then basically moves into tutor
mode, but they can give that feedback at scale, so if a group of students all make the same mistake, I
only have to write, kind of, the tutorial why that's wrong once, and I can reuse that.
But also it's very important and studies show that if the teacher understands the misunderstandings of
the class, they themselves go into tutoring mode in subsequent classes so you get a tutoring effect there
you don't in a classroom setting?
>> Maria Langworthy: Yeah, and that is critical to be able to -- and that fits in with the John Hattie
model of visible learning, of people being able to look at the data that's being produced, the student
work that's being produced, and look at where the teaching methods used are working or they're not, so
you can say, okay, across that group of 100 students, this set of 50 seem to get it and their work product
seems to be scoring pretty high on an automatic thing. This other group gets it halfway, and this other
group doesn't get it at all. We need the figure out what to do with this group. We need to figure out
what to do with this group, and we need to figure out what's the next challenge for this group.
>>: Is the benefit from the teacher doing the scoring themselves, or is it from looking at the data?
>> Maria Langworthy: Well, there's two things. There's one, teachers can't make this leap, especially on
the higher order skills front. If we were just talking about content knowledge that teachers already
know, it would probably be pretty safe, yeah. It would be pretty safe to use a kind of automated scoring
system because they know the answer s.
What we find in this kind of stuff is this the way that teachers learn it, by doing, you know, and so they
have to do the assessment. And also part of the teacher's role is to design the learning experience in a
way that makes it relevant and connects to their students in that moment, and that cannot be fully
automated right now.
Teachers have to know their students, have to have a relationship with them, know their context
outside of school to be able to do that, and students have to have a voice in that too. So there's
elements of that that, at least right now, would be very hard to automate.
That's why we're saying in some of the new work like what Michael Fullan and I wrote, is that the core
piece of the new pedagogy is a learning point between the teacher and the student and that's personal.
And that is what more than anything else addresses the equity gap. It's because only when those
personal -- there's Paul Tuft's work on grit and stuff about how do you get kids to persevere? And it
turns out that personal relationship, that personal knowledge is essential.
>>: I found online 21CLD learning activity rubrics and student work group. Are those?
>> Maria Langworthy: Those are the ones issue yeah. Exactly. Good you found them yeah.
So, yeah, basically in this professional development program, they have some example ones that we
bring in. And then they start to bring in their own learning activities and they collaboratively debate. Is
this lesson a 2 or a 3 or a 4? And this is the simple version of it. You'll see the rubrics are like 8 to 10
pages apiece for each of these because they give a real description. They're very rich.
But it provides, you know, conceptual framework for really looking at the depth and rigor of the lesson
in terms of these 21st century skills, and it really relies on teachers working together. You don't realize
until you really go into schools how isolated most teachers are. They spend all day by themselves with
20 or 30 kids or 50 kids, and they don’t' -- if you've ever been in meetings with teachers, you'll notice
pretty quickly that teachers' collaboration skills with other teachers are pretty low level. But what we
do in this professional learning program is teachers are working together, looking at students' work,
examining their own lesson plans in relation to these 21st century skills. It gets really exciting. And the
only reason I really believe in this is because it has just organically taken off, the use of these rubrics. I
hear stories from all over the world, not supported by Microsoft. In Australia they took it upon
themselves to rewrite the rubrics to integrate with the new Australian national curriculum.
There's programs are emerging in teacher colleges around this stuff in Denmark and Portugal and
Australia, so there's something to this, basically. And it's, you know, it's still part of the Partners in
Learning program, but it's also, you know, taking off in all kinds of schools in all kinds of ways all around
the world. I've seen in South Africa, a school that has made these -- they adapt the rubrics to their own
context but they've made them part of the report cards that go home with schools twice a year and can
create a framework for discussion with teachers and student around these kinds of things, really
integrating them.
And they've also, the rubrics, have become the basis for the judging of innovative teaching projects at
the Microsoft Partners in Learning global forum and pulls together about 800 educators from about 80
countries each year based on their innovative teaching. This year it was in Barcelona, and so these
rubrics have become the basis, and the judges are people usually from ministries of education all around
the world so they get exposed to the rubrics as well. It's very exciting.
What's next after this? Let me take just a minute because this has all been sort of the last four years or
so. So what are we doing now with all of this? Well, there's this big recognition that even with
professional development, even with some early stage measures of teaching, it's really not enough to
shift whole systems. You still see innovation in pockets of teachers in pockets of schools. You don't see
it systemwide. So this is what we've started about a year. We really kicked it off last June, a new
project, in which Microsoft is a partner from the private sector. It's called New Pedagogies for Deep
Learning. New pedagogies is innovative teaching, deep learning is 21st century skills. The new names
for them, but we have this guy who's an expert in creating whole system change.
>>: Deep learning is just such a buzzword.
>> Maria Langworthy: I know, yeah. The whole machine learning, yeah, I know. I found that out about
a month and a half ago. I was like, really? I'm working on deep learning too [laughter]. I think we mean
something different by it, yeah.
In fact, my husband's ex-boss, who left Microsoft, has a start-up that's called deep learning [laughter]. I
was just like, whoa.
So this is a global partnership, bringing together clusters of schools, private sector, big education
partners, EOCD, et cetera, and a lot of great advisers like this guy John Hattie, to work on deep learning.
And when we're talking about deep learning, it is these 21st century skills, but it's not just that. It's also
creating the dispositions that will allow students to use these.
There's a different orientation, both epistemologically and toward doing things in the real world. You
don't just receive information and practice using, say, new concepts. You've got to go out and do it in
the real world, and technology enables that, that really gives kids the oomph, the disposition to create
their own value streams. Nobody's going to give them a job in the future. They've got to create their
own value. And that's what this deep learning stuff, and this is in a book that's right here, look. Kind of
a large white paper that Pearson sponsored us to do.
But here we're forming partnerships with ten countries, two clusters in the United States. Each country
is bringing 100 schools into the project, so a thousand schools around the world working on this
together, really working on how do we mobilize and foster new pedagogy for deep learning across all
schools in all systems? We talked about there have to be measures, there have to be policies, there
have to be programs, there has to be leadership, that really supports and embraces that stuff holistically
to transform whole systems. So that's what we're trying to do with this project. It's not -- it's not
research. It's mobilization. Its implementation of these ideas and this is kind of a logic model, but
because it's not a research project, it's much more focused over here on the inputs, these learn
progressions are the definitions, the rubrics of what these 21st century or deep learning competences
are.
We're also doing a lot of professional learning to develop teachers' pedagogy capacities, change
leadership, partnerships, building partnerships, you know, across countries to get the ideas of the
hundred schools and the evidence really used, but looking again across the whole system.
And my job in all this has been to develop a measurement strategy so that we're measuring not just
student learning outcomes on these skills. We're also measuring the learning partnerships between
teachers and students, the tasks, or the learning activities that teachers are designing. We're measuring
all of those.
And we're measuring the conditions within schools, the level of student engagement, the levels of
family engagements with and how those shift as we shift these things.
>>: Have you tried to contact anyone in India and have you found partners or people who are interested
in working with you as collaborators on this? Because I haven't seen India in any of the previous ->> Maria Langworthy: Yeah, yeah, yeah. I think somebody -- I'm not in charge of the partnership piece
of this. I think somebody talked once with India, but no. We did talk initially with India, but there
wasn't, you know, it's -- you'll notice in the ITL research that the United States wasn't there either.
>>: -- China, India, and I see a lot [indiscernible] countries I was like no. For the technology to -- I mean
I'm interested the technology and ->> Maria Langworthy: Yeah no, it's -- I I'd love to have them involved. I would have loved to have them
involved in the former project.
They are also incredibly complex, huge, finding the right partner amid that. And a lot of those
partnerships happened by chance, you know, like Mexico. He saw me speak at an event and he was the
sub minister of education, and he said, "We want to do this."
The same thing happened with Brunei, and it was just boom, boom, boom.
So we didn't like set out on open tender globally to say come join us on this. With this project we did
submit. We did write a white paper saying -- an invitation to partner and that provoked most of
response and we tried to send it out internationally but we didn't have, you know, massive amounts of
marketing dollars.
But when we talk about the new pedagogies in this project, we come back to that John Hattie work a
little bit too and it's not just about the design of the learning activity. What his research showed was
that -- have you guys all heard about teachers as guide on the side? So Hattie's work on looking across, I
think it's two or three hundred, different kinds of instructional strategies, and he categorized some of
them into teacher as activator, teacher as facilitator, where a teacher is much more doing inductive or
students are controlling, where teacher's role is much more passive. And what he found was that
teacher as activator, where there's a strong student-teacher relationship, where there's a focus on
reciprocal teaching, where students are teaching each other, where students are teaching teachers, et
cetera, where there's a huge formative feedback loop, metacognition, teacher clarity, this is has a much
higher impact on student learn outcomes.
Hugely higher impact. So for this book I was doing interviews with teachers that I knew to be kind of at
the cutting edge of some of this stuff and just loosely listening to them, and they kept talking about the
changed relationship with their students that was a result of changing their practice. They knew their
students better. They knew who they were personally, and we built that into this model of what we
mean by new pedagogies. It's not just the redesign of the learning tasks. It's the building of these
relationships with students and the ubiquitous use of technology to enable both the learning
partnerships and the deep learning tasks.
>>: Offline maybe but the one before, where does this come from?
>> Maria Langworthy: It's a guy named John Hattie. He's a researcher. He was in New Zealand. Now
he's at the University of Melbourne.
>>: I'd just be interested in tracking down a reference for this at some point.
>> Maria Langworthy: Visible learning?
>>: So I could understand better the difference between activator and facilitator.
>> Maria Langworthy: Yeah I would highly recommend visible learning. I'd highly recommend we bring
John Hattie into MR. He's done more research on this stuff happen he's a very ->>: Visible learn is a book?
>> Maria Langworthy: Yeah. It's a book. It's a series of books, actually, and now a professional learning
program, but the original piece was this analysis of the impact of different instructional stages that was
based on 800 different meta-analyses of studies that have been collected globally on the impacts of
student outcome. It's an amazing piece of work.
But he's very much a pessimist about technology and about technology having an impact on learning
and we debate this all the time but he's starting to come around a little bit but a lot of it is about how
it's been used in the past, right? How technology has used in teaching and learning.
But a brilliant guy, yeah. So that's informed, sort of, the second level development of this idea of new
pedagogies that is at the core of this new project and the things that we'll be measuring. And this is a
kind of example of what one of the kinds of tasks look like that incorporated all of these different
elements. It was a fifth and sixth grade science class that had been studying water, the hydrosphere all
around the world.
It had a bunch of different missions that the teachers designed, and the last mission was to demonstrate
their learning. And the kids got to choose however, wherever in the world they wanted to demonstrate
their learning about the hydrosphere.
>>: What age?
>> Maria Langworthy: Fifth and sixth grade students, and this is in Birmingham, Michigan. And the kids,
they did some research. They said, okay, there's this project that is started by a guy in the United States
working on water purification and increasing the rate of development of communities in Zambia. And
so they introduced themselves to the leader of the project -- it was called Project Hope -- and they
Skyped with him. And they talked it through and decided the best thing that they could do would be to
fund this village getting an ecodome toilet.
So then they Skyped with providers of ecodome toilets in Africa, got cost quotes, figured out. And then
they started fund-raising. They needed to have $2,500 to buy an ecodome toilet for this village in
Zambia. And they did recycling. They designed chairs and auctioned them online. They made these
bracelets from recycled gift cards. This is all in the last three weeks of school last year.
They failed a lot. They didn't make money the way they thought they were going to make it. They had
tremendous energy. They got the attention of all the other students and teachers in the school, going
why are these kids so excited?
You know, they're not -- and all of this was tied to their common core curricular goals around science, et
cetera, but they did manage to raise the $2,500. They have a video of the whole thing. They've Skyped
with the village.
They've sent the ecodome toilet, and then the principal got so -- it was his aha moment. We're doing
this teaching and learning in this new way kids. Get entrepreneurial skills, they get problem solving
skills. They do all kinds of things beyond learning the content.
And this year they've taken on buying a tractor for the whole school. They raised that amount of money
-- for the community. And then, I think it was a Skype in the classroom day about a month ago, they
were awarded a matching grant from Skype to buy a second tractor for the village. And they have it all
on video and it's a fantastic story. But you see, these kinds of tasks when designed well, and they have
all kinds of different rubrics for assisting the student work for peer collaboration, et cetera, they become
very, very rich and these kids will never forget this project and it creates a different disposition in them
about what they can do and what learning is about.
>>: That's your classroom work.
>> Maria Langworthy: That's right, yeah, and it was using pretty basic technology tools in a way. But so
this is what, you know, we're aiming for with this project is shifting how kids are using technology and
shifting why they're using technology.
I'm going to summarize real quick. Like the old pedagogies, you had teaches really focused on their own
content knowledge of their subject. Pedagogical capacity was somewhat important but secondary, and
tech use was really layering on top to support the content knowledge, and students were measured on
their mastery of that content.
In the new pedagogies, what we're seeing is ubiquitous technology that surrounds everything that
teachers and students do. The teachers and students discovering and mastering content knowledge
together, not as the teacher as the source of that knowledge, maybe as an arbiter of it, but students
with just as much voice in the kinds of knowledge they want to discover. The kinds of information, the
kinds of content, the actual concept.
Teachers mostly valued on their pedagogy capacity. How well can they build those relationships? How
well can they design learning activities and deep learning tasks that really deeply engage their students
and launch them? And then how well do students do on performing tasks? How well can they
demonstrate their knowledge and apply it and use it to do real things in the world, you know, solving
problems with water in Zambia, those kinds of performance tasks?
And the question for us is how can we use technology not just for this piece but for this piece and this
piece, and how could you do it for the assessment of all that as well and the measurement of that?
These are, I think, at the early stages of the work, and that's where, you know, there's, I think, a huge
potential for Microsoft to play a role. All of this stuff could be plugged into Office 365 on the back end in
a very powerful way, but I haven't heard of anyone yet who's working on that [laughter]. That's it. So,
that's it. We've -- I never thought we would make it to an hour and a half here, but I thank you guys
very much, and I'm happy to continue talking about all this stuff. Thanks. [applause]