Running head: CRITICAL REVIEW 3
Critical Review of Research #3
Tammi Kolski
EDU 800
CRITICAL REVIEW 3
2
Critical Review of Research #3
1. I found the authors (Gamrat, Zimmerman, Dudek, & Peck, 2014) to clearly state a specific
reason for conducting their research: to demonstrate with implementation of digital badging
educational technologies, that teachers’ professional development (PD) could be personalized
and customized in accordance with their workplace learning settings. More specifically, Gamrat
et al. investigated “how teachers could use a digital badging system to personalize their
experience in Teacher Learning Journals (TLJ) to meet their PD goals by providing learner
customization of their content choices and decision making in the level of assessments utilized”
(p. 1145).
2. As teachers must engage in PD to expand their skills and knowledge (as well as to remain
compliant to state regulatory credentialing and certification systems) use of an online PD
badging system could meet this need. An increasing number of educators are seeking online
sources to meet their professional development needs (Learning, 2010). Badging, as a form of
microcredentialing, could replace the outdated “one-size-fits all model of PD” (Gamrat et al.,
2014, p. 1138). Badging can be both incentivize learning and tie performance and achievement
to observable tasks, activities and skills as well as allow for customization to better meet the
needs of professional educators with a wide range of expertise and experience. “Badges offer a
socially constructed and valued encapsulation of experiences through a variety of rich metadata
that offers transparency and depth into the learning and achievements of the learners which can
then be reviewed by others” (Gamrat et al., 2014, p. 1136).
Hickey and Soylu suggest that digital badges can capture metadata and can act like a
portfolio, documenting workplace learning and artifacts that can be shared with fellow
colleagues and supervisors, or as repositories of content or reflection (as cited by Gamrat et al.,
CRITICAL REVIEW 3
3
2014, p. 1137). Through the use of digital badges, teachers can electronically display, share and
revisit activities completed towards their PD. This would also allow their administrators to gain a
better understanding of what skills the teacher sees needed to best complete their work, as well
as allow other teachers to view and offer feedback on the PD decisions made by their colleague.
As studied by Abramovich, Schunn, & Higashi (2013) the role digital badges is found to impact
both intrinsic and extrinsic motivation. Certain types of extrinsic motivation combined
synergistically with intrinsic motivation, can lead to high levels of employee satisfaction and
performance (Amabile, 1993).
3. I think this problem is researchable, I am just not convinced this study offered an effective
method for data collection and analysis. This being an exploratory study, the authors had a
limited amount of resources to draw inferences from. However, just because there isn’t much
research conducted does not preclude there is not a key problem to be explored. While the
authors identified the need to answer the questions of “how should PD be designed and what
types of interactions should the PD support?” (p.1137), I found them to fall short in answering
these questions.
Gamrat et al. (2014) approached the collective case study data with a theory-driven
thematic analysis to investigate how teachers personalized their use of TLJ to support their PD
goals. A thematic analysis is a “categorizing strategy for qualitative data. It helps researchers
move their analysis from a broad reading of the data towards discovering patterns and
developing themes” (Boyatzie, 1998). As an additional assessment method, Gamrat et al.
(2014) “applied personalization as an analytical framework to analyze teachers’ experiences” (p.
1146). An analytical framework allows for the collection and organization of analyzed patterns,
techniques, and the expertise of others who have solved similar modeling problems
CRITICAL REVIEW 3
4
(Lieberman, 2007). Through personalization the authors construed the uniqueness of each
participant’s situation that allowed for individualized decision making and unique customization
of their experience. It can be difficult to effectively measure outcomes based on independent,
personalized commentary not being ‘fact based’ measureable. While the authors utilized a
coding system, this seemed to fall short in clearly distinguishing trends or patterns of outcome
content. As evident by the lack of cohesion between thematic codes and sub-codes used for data
analysis found in Table 2 (Gamrat et al, 2014, 1142).
4. Jabareen (2009) identifies “the concepts that constitute a conceptual framework support one
another, articulate their respective phenomena, and establish a framework-specific philosophy”
(p. 51). The main features therefore of a conceptual framework identifies the key factors, or
variables, and infers relationships among them. Gamrat et al. (2014) built a foundation for their
study around two components of the Kearney, Schuck, Burden, and Aubusson (2012) model:
decision making for personal interest and customization for workplace opportunities and
constraints. Decision making being defined as “learning related to personal self-development
and customization as related to addressing learning related to workplace affordances and
constraints” (Gamrat et al., 2014, p. 1139). The authors focus on decision making and
customization was supported by a literature review of professional learning, where “professional
learning is widely believed to be more effective when it is based on self-development and workbased learning” (Stoll, Bolam, McMahon, Wallace, & Thomas, 2006, p. 232). Again referencing
the Kearney, Schuck, Burden, and Aubusson (2012) framework of mobile learning, Gamrat et al.
adapted personalization to refer to a “learners’ choice related to just enough, just-in-time, justfor-me educational opportunities where learners get to create their own tailored learning journey”
(p. 1139).
CRITICAL REVIEW 3
5
Gamrat et al. (2014) expanded the customization concept into learner empowerment and
community of practice; utilizing both components to strengthen the foundation of their study. As
found in the research of Brooks and Young (2013), learner empowerment is connected to a
motivational process that increases personal initiation, persistence to complete a task, and
feelings of self-efficacy. This allowed learners to choose what they needed to meet their
workplace goals and adapt material to achieve these goals (McLoughlin and Lee, 2008).
Customization also allowed for a community of practice to develop. As stated by Mackey and
Evans (2011) in their research, “participants became brokers and conduits between the online
learning community and their own community of practice” (p. 12). Through a community of
practice in peer teaching and structured learning situations, school-based and industrial-based
workplace learning took place (Fuller & Unwin, 2002).
An aspect I did not find Gamrat et al. (2014) to address in their conceptual framework, yet
they emphasized its importance in their conclusions, was the importance of having an expert
practitioner providing feedback to the teacher’s activity logs. I liked the analogy of one
participant who identified the expert feedback to feel like she had a personal coach. Especially
in online learning, feeling connected and supported, as would be obtained from use of a mentor
or coach, is vital to successful virtual learning (Dorner, 2012).
5. Being an exploratory study, little research had been conducted to determine how personalization
through badging can enhance online teacher PD. To analyze workplace learning within the TLJ
project, Gamrat et al. (2014) “adapted a theoretical framework from the educational technology
personalization literature provided by Kearney, Schuck, Burden and Aubusson (2012)” (p.
1139). Furthermore, in McLoughlin and Lee’s (2008) discussion of learning as a means of
participation, the authors describe decision making as “central to personally meaningful learning
CRITICAL REVIEW 3
6
and allow[ing] learners to decide how to best meet their goals. In this way, teachers make
responsive decisions to focus on their personal needs and expertise” (p. 1139). Regarding
customization, it can be applied to workplace communities as well individuals. In a case study
analysis of workplace learning research studies, Fuller, Hodkinson, H., Hodkinson, P. and Unwin
(2005) examined learning communities through the lens of communities of practice and
concluded individuals’ experiences, interests and professional responsibilities varied with regard
to the need for training and their participation in teaching others.
I did not find the inclusion of Tynjälä, Häkkinen, and Hämäläinen’s (2014) work on
social media tools in supporting rationale for capturing personalized experiences in the
workplace to be relevant to the problem being investigated for this study.
Gamrat et al. (2014) focused heavily in their research on the Teacher Learning Journeys
(TLJ) as an approach that allowed teachers to customize their PD experience to their workplace
and make decisions about what PD they need based on their expertise and interests (p. 1136).
Even throughout the Findings and Discussion sections of their article, their focus remained
strong in discussing TLJ. Yet the independent variable being measured in this study was the
digital badging technology and not the TLJ as a PD tool.
6. I did find the literature review to conclude with a summary that connected the literature to the
problem being investigated. The authors posited that “one can design technologically enhanced
workplace tools that can enhance how learners customize navigation and track personally
relevant PD” (Gamrat et al., 2014, p. 1140). They found the use of TLJ to “empower
customization when learners modify their PD experience to reflect constraints and opportunities
in their local or regional workplace” (p. 1140).
CRITICAL REVIEW 3
7
7. Gamrat et al. (2014) offered clarity and appropriateness in formulating the focus of their
research. The ways in which “personalization through badges supported workplace professional
learning by providing learner customization and decision making was investigated” (p. 1145).
Gamrat et al. conducted a collective case study as an inquiry into personalized learning
supported by the TLJ tool and digital badging system. The case study methodology allowed the
authors the ability to examine professional development as a whole, yet giving an in-depth look
at the teacher’s experiences from multiple perspectives.
8. Using a theoretical framework that focused on decision making and customization as a part of
personalization, Gamrat et al. (2014) conducted a “theory-driven thematic analysis on teachers’
TLJ artifacts (goal statements, interviews and reflective activity logs)” (p. 1136). “TLJ was built
as an online PD website that used digital badging to provide and capture real-time, ongoing, and
work-embedded support through the use of online teacher PD” (Dede, Ketelhut, Whitehouse,
Breit, & McCloskey, 2009, p. 9). TLJ was designed to support teachers in a personalized
PD that allowed the participants to set their own learning goals, to select learning activities that
aligned with these goals, and to fulfill education requirements set by their local districts and
states as well as their workplace. The use of the TLJ tool was presented as intentional in its
choosing and provided an appropriate instrument in fulfilling the research suppositions.
The research questions were examined across all 36 teachers in the study as well as with an
in-depth study of eight teachers’ experiences. The researchers explored how a teacher using the
TLJ’s badging system was afforded personalized PD. In addition, having used a collective case
study method, they compared and contrasted the teacher’s experiences amongst each other. A
collective case study method “will allow the researcher to analyze within each setting and across
settings” (Baxter & Jack, 2008, p. 550). Randolph (2007) described a case study as “an
CRITICAL REVIEW 3
8
empirical inquiry that investigates a contemporary phenomenon within its real life context. The
goals of a case study are to develop an in-depth understanding of a case …to gain insight into the
interaction between the phenomenon and the case.” All of these characteristics are consistent
with the current study. Being an explorative study, the case study method is a “useful design
when not much is known about an issue of phenomenon” (“Types of Research Design”, n.d.)
such as this study was.
9. Data was collected from 36 teachers who completed 154 PD activities over a 3-month period.
Participants were teachers at all levels of their careers, who had science teaching responsibilities,
and they were recruited via email rosters from partnering educational organizations. What I
found Gamrat et al. (2014) to not disclose, was who the partnering educational organizations are.
I assume they are reputable teaching affiliated organizations, but to not identity the sources that
participants were drawn from is an error in their reporting. Without knowing more about the
partnering educational organizations, it could also result in selection bias if the organizations
drawn from held a particular position about online learning, benefits of professional
development, use of the TLJ tool, etc. According to methodologists concerned with selection
bias, ‘this approach to choosing cases leaves these scholars vulnerable to systematic and
potentially serious, error. Selection bias will lead quantitative analysts to underestimate the
strength of causal effects” (Collier & Mahoney, 1996, pp. 57-63).
Since the participants were recruited via email, the study excludes in its sampling
methods those teachers who did not have emails or internet access for emails. Considering the
authors (Gamrat et al., 2014) noted a lack of internet access as a limitation to their study, this
could also impede the possibilities of teachers from participation.
CRITICAL REVIEW 3
9
To understand the in-depth professional learning experience, the authors (Gamrat et al.,
2014) utilized a collective case study method that focused on eight of the 36 teachers. Teachers
were excluded as participants if they did not complete all research tasks (although this did not
exclude them from participating in the TLJ project), including interviews, a goal statement and at
least one PD activity. It is not identified in the article if any teachers were excluded from
participating. While the teacher population used fulfills the personalized component of the
study, it also presents as a challenge for the results to be generalized to other populations; in
particular where only eight teachers are queried in detail. As reported by Myers (2000), “small
qualitative studies are not generalizable in the traditional sense”. Eight participants seems to be a
small population for a qualitative study
10. Using TLJ as a PD tool, teachers made personally relevant decisions when composing their
learning goals. Providing teachers the ability to make decisions about their assessment provided
them the flexibility to personalize their PD to align with their existing expertise and with what
was needed for their workplace.
Stamps and badges were awarded by the TLJ mentors when the teachers demonstrated the
specific skills or content covered in each PD activity. Stamps represented a lower level of
achievement (i.e. general activities such as writing a short reflective statement) while badges
represented a higher level of achievement (i.e. focused, high-level activities such as developing a
work plan on how they would integrate new skills or knowledge into their job). While the
research results show that stamps were achieved six times more often than badges were earned, it
is not stated that a teacher needed to complete a pre-determined number of stamps in order to
achieve one badge. At least that was my experience in a badge pilot program where I both
administered as well as received a badge when the established criteria for such was fulfilled.
CRITICAL REVIEW 3
10
Analytical memos are one of the most important methods used for analyzing qualitative data
and for developing unique ideas and insights about the topic being studied (“Guidelines for
Analytical Memo “, n.d.). After coding each teacher’s data, Gamrat et al. (2014) were wise to
create analytical memos to more deeply understand each teacher with regard to their
personalized, professional learning experience.
Four sources of data were identified as being collected in this study: system generated reports
for each stamp or badge awarded, and only on the eight specially designate teachers, for each,
their PD activity logs, purpose statements, and pre- and post-TLJ interviews (Gamrat et al,
2014). Other than stating numerically the system generated reports for each stamp or badge
rewarded, the authors do not show aggregate data on the remaining data collection aspects
(which also was specific to the specially designated eight teachers). Not having a global
representation of what the data collected corresponded to in the authors’ findings I see to
compromising the integrity of the data collection procedure.
11. Regarding reliability and validity, Randolph (2007) describe each as “Reliability is the extent to
which measurements are consistent, over time, and under different conditions. Validity is the
extent to which an instrument or design is measuring what it intends to measure” (p. 37). This
paper demonstrates no attempt on the authors part to establish or demonstrate the reliability or
validity of their instruments used. As much as the authors seemed to favor the TLJ instrument,
they offer no supporting documentation on its reliability or validity.
The authors used expert practitioners (senior educators) – also referred to as TLJ mentors
– to review the teachers reflective activity logs. This offered a degree of verification that the
data collected from the teachers is consistent with other data collected from the logs. The
antidotal comments of individual teachers during the interviews extrapolated into conclusions as
CRITICAL REVIEW 3
11
being effective by the authors did not seem to demonstrate that quality measures were being
utilized in this study. The paper does use several quotes from participants, but from such a small
sample, it is difficult to determine the extent to which this is representative of the entirety of
responses. In this way, the quotes may be useful to emphasize and illustrate key ideas, and
contribute to validity, but only as a supportive method.
Consistently across all measures, the teachers completed more stamps than badges. The
researchers posited this provided teachers the ability to make personalized PD decisions about
their assessment that would align with their existing and desired expertise needed for their
workplace. I question that allowing the participants this much control over one of the
independent variables (stamps and badges) would affect the reliability of the conclusions drawn
referencing the overall effectiveness of the badging system.
12. To be omitted as instructed by the instructor.
13. Data was collected from all 36 teachers between June and August 2012. It is not identified what
information was retrieved from the system generated records when a participant was awarded a
stamp or a badge and how that information was integrated into the results. Neither were the
compiled results of the eight case study teachers pre- and post-TLJ intervention interviews
reported. The authors fail to identify how many of the teachers may have started work towards a
stamp or badge but did not finish the criteria required.
Another suspected conceptual limitation of the results was being able to effectively
measure summatively the independent, personalized commentary. Through personalization the
authors construed the uniqueness of each participant’s situation that allowed for individualized
decision making and unique customization of their experience. While the authors utilized a
CRITICAL REVIEW 3
12
coding system, this seemed to fall short in clearly distinguishing trends or patterns of outcome
content.
The authors acknowledge a limitation to their research was all but one of the 36 research
participants were female. While the National Center for Education Statistics (“Fast Facts”, n.d.)
in 2011–12 report 76 percent of public school teachers were female, this still seems like a gender
imbalance for the purpose of exact and equal separation of men and women in research.
14. There is an error in the article when it states “In our dataset, teachers typically made the decision
to seek the stamp microcredential (the lower achievement) rather than a badge (the higher level)
as shown in Table 2” (p. 1142). The stamp versus badge achievements by content area is
actually shown in Table 1.
Throughout the conclusion of this article, wording such as “very responsive”, “has
proven much more effective”, or “the data supported” are used. However, there was no
statistical inferences made, no ANOVA, and no hypothesis testing (t-values or p-values)
reported. Understandably the use of thematic analysis on qualitative data is valuable in terms of
improving the rigors of the analytical steps for validating that which does not reflect the
researcher’s impressions of the data (Alhojailan, 2012). In addition, case studies rely primarily
on qualitative methods, although quantitative data may be used to support emergent themes
(Creswell, 2003). What this study failed to include was substantial quantitative data that would
support their use of language as noted above.
Consistent with the desire for PD to be customized, the commentary by the majority of the
teachers during the interviews after their TLJ experience, supported individuation among the
teachers in finding the PD they needed to meet their professional learning goals. Gamrat et al.’s
(2014) analysis suggests that the reflective purpose statements provided by most teachers
CRITICAL REVIEW 3
13
supported them in identifying and managing their own learning goals. “This type of organic
needs assessment has proven much more effective than traditional PD and even other types of
online training” (p. 1144). In addition, customized assessment and online mentoring supported
personalized professional learning. I found the use of conversation with the expert practitioners
to support the teachers in customizing their PD in ways that were sensitive to regional and local
regulations affecting their workplaces.
15. Gamrat et al. (2014) stayed true to relating their results to the studies theoretical base. Their
research findings suggested “possible design principles for badges and flexible PD to advance
the use of personalized educational technologies in workplace learning settings” (p. 1138).
Regarding the two primary components to Gamrat et al.’s (2014) conceptual framework,
decision making and customization, the authors concluded the following advantages of using
TLJ for professional development purposes:

Choosing level of assessment and content exploration allowed for personalization

Teachers used TJL system flexibility to set personalized PD goals (designing flexibility
online PD using a digital badging system)

Decision making in the form of initial goal setting helped enhance productivity and was
beneficial in teachers making future decisions on how to approach their
professional learning that led to the fulfillment of the teacher’s goals.

Expert feedback on reflections increased teachers’ customization to their workplace
(expert mentors provided key support for customization)

Overall implications for technologically enhanced workplace learning using badging.
(pp. 1142 – 1146)
CRITICAL REVIEW 3
14
As indicated previously, my critique is the above advantages speak to the TLJ tool with limited
content about badging, which is supposed to be the core variable of their research.
Fuller and Unwin (2002) found that workers across employment sectors effectively used
formal educational opportunities. The digital badge design implications from this study was
utilized as a possible formal PD tool to deliver content that could be customized and assessed
using targeted microcredentials. Archiving and sharing PD artifacts for further learning and
workplace advancement was also found to be helpful and appreciated by the teachers in this
study.
16. The significance of this study was to understand how personalized learning/PD activities can
support teachers and other professionals in a variety of workplace settings. Allowing individuals
to customize their PD training could delineate companies from paying for their employees to
attend conferences, in-services, and other expensive technological investments. Use of online
PD opportunities would also allow individuals to be empowered by their own decisions (such as
on the types of TLJ activities to engage in) that are most relevant to their workplace
responsibilities. It would extend workplace learning beyond on-the-job apprenticeships to
include personalized independent study. By utilizing TLJ to customize PD content to their own
workplace requirements, this evidence suggests digital badging could be one potential solution
for personalization in workplace learning to meet the needs of workers and employers. As
evidenced by the teachers’ eagerness to share resources with colleagues, Gamrat et al. (2014)
concluded that online PD affords that ease of sharing.
This research provides an example of digital badges as a credentialing system to assess
personalized PD. Future research was recommended to understand how individual
characteristic’s effect technology-enhanced workplace learning as well as additional research
CRITICAL REVIEW 3
15
being warranted in the areas of the design of digital badging systems and its applications to
learning in professional contexts. Having participated in a pilot program using badges to
acknowledge successful completion of tasks, it was not a motivating or rewarding system for me.
Future research to identify the motivational benefits of digital badging, whether in the academicbased workplace environment or the industrial-based workplace environment, would be
warranted to supplement other research looking to replicate the focus of this study.
CRITICAL REVIEW 3
16
References
Abramovich, S., Schunn, C. & Higashi, R. M. (2013). Are badges useful in education?: It
depends upon the type of badge and expertise of learner. Educational Technology
Research and Development, 61, 2, 217–232. doi: 10.1007/s11423-013-9289-2
Alhojailan, M. I. (2012). Thematic analysis: A critical review of its process and evaluation.
West East Journal of Social Sciences, 1(1), 39-47.
Amabile, T. M. (1993). Motivational synergy: Toward new conceptualizations of intrinsic and
extrinsic motivation in the workplace. Human Resource Management Review, 3(3), 185201.
Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and
implementation for novice researchers. The Qualitative Report, 13(4), 544-559.
Boyatzie, R. E. (1998). Transforming qualitative information: Thematic analysis and code
development. Thousand Oaks, London, & New Delhi: SAGE Publications.
Brooks, C., & Young, S. (2011). Are choice-making opportunities needed in the classroom?
Using self-determination theory to consider student motivation and learner
empowerment. International Journal of Teaching and Learning in Higher Education,
23(1), 48-59.
Creswell, J. W. (2003). Research design: Qualitative, quantitative and mixed methods
approaches (2nd ed.). Thousand Oaks: Sage.
Dede, C., Ketelhut, D. J., Whitehouse, P., Breit, L. & McCloskey, E. M. (2009). A research
agenda for online teacher professional development. Journal of Teacher Education,
60(1), 8–19. doi: 10.1177/0022487108327554
Dorner, H. (2012). Effects of online mentoring in computer-supported collaborative learning
CRITICAL REVIEW 3
17
environments: Mentor presence and cognitive engagement. The American journal of
distance education, 26(3), 157-171.
Fast Facts. (n.d.). Retrieved from http://nces.ed.gov/fastfacts/display.asp?id=28
Fuller, A., Hodkinson, H., Hodkinson, P. & Unwin, L. (2005). Learning as peripheral
participation in communities of practice: a reassessment of key concepts in workplace
learning. British Educational Research Journal, 31, 1, 49–68.
doi: 10.1080/0141192052000310029.
Fuller, A. & Unwin, L. (2002). Developing pedagogies for the contemporary workplace. In
Working to learn: Transforming learning in the workplace (pp. 95–111).
Gamrat, C., Zimmerman, H. T., Dudek, J., & Peck, K. (2014). Personalized workplace learning:
An exploratory study on digital badging within a teacher professional development
program. British Journal of Educational Technology, 45(6), 1136-1148.
Guidelines for Analytical Memo #1: Social media, agency, and languaging. (n.d.). Retrieved
from https://canvas.instructure.com/courses/777899/pages/guidelines-for-analyticalmemo-number-1
Jabareen, Y. (2009). Building a conceptual framework: Philosophy, definitions, and procedure.
International Journal of Qualitative Methods, 8, 49-62.
Kearney, M., Schuck, S., Burden, K. & Aubusson, P. (2012). Viewing mobile learning from a
pedagogical perspective. Research in Learning Technology, 20(14406), 1–17.
doi: 10.3402/rlt.v20i0/14406
Learning, D. (2010). Innovative teacher professional development. International Review of
Research in Open and Distance Learning, 11, 81-95.
Lieberman, B. (2007, April 17). Applying an analytical framework. Retrieved from
CRITICAL REVIEW 3
18
http://www.ibm.com/developerworks/library/ar-anframe/
Mackey, J., & Evans, T. (2011). Interconnecting networks of practice for professional learning.
International Review of Research in Open and Distance Learning, 12(3), 1-18.
McLoughlin, C. & Lee, M. J. (2008). The three p’s of pedagogy for the networked society:
Personalization, participation, and productivity. International Journal of Teaching and
Learning in Higher Education, 20(1), 10–27.
Myers, M. (2000). Qualitative research and the generalizability question: Standing firm with
Proteus. The Qualitative Report, 4(3/4). Retrieved from:
http://www.nova.edu/ssss/QR/QR4-3/myers.html
Randolph, J. J. (2007). Types of research approaches in educational technology research and
development. In Multidisciplinary methods in educational technology research and
development (p. 53). HAMK Publications.
Stoll, L., Bolam, R., McMahon, A., Wallace, M. & Thomas, S. (2006). Professional learning
communities: A review of the literature. Journal of Educational Change, 7(4), 221–258.
doi: 10.1007/s10833-006-0001-8.
Types of Research Designs - Organizing Your Social Sciences Research Paper - Research
Guides at University of Southern California. (n.d.). Retrieved from
http://libguides.usc.edu/writingguide/researchdesigns
Tynjälä, P., Häkkinen, P. & Hämäläinen, R. (2014). TEL@ work: Toward integration of theory
and practice. British Journal of Educational Technology. doi: 10.1111/bjet.1216