Journal of Cybersecurity, 2021, 1–17
https://doi.org/10.1093/cybsec/tyab025
Research paper
Research paper
Hilda Hadan
*
, Nicolas Serrano and L. Jean Camp
Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, 919 E 10th St,
Bloomington, IN 47408, USA
∗
Correspondence address. Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, 919
E 10th St, Bloomington, IN 47408, USA. Tel: +1-812-856-1865; Email: hhadan@iu.edu; ljcamp@indiana.edu
Received 24 May 2021; revised 21 September 2021; accepted 21 November 2021
Abstract
Public key infrastructure (PKI) is the foundation of secure and trusted transactions across the Internet. This paper presents an evaluation of web-based PKI incidents in two parts. We began with
a qualitative study where we captured security and policy experts’ perceptions of PKI in a set of
interviews. We interviewed 18 experts in two conferences who include security academics and
practitioners. We describe their perceptions of PKI failures. To evaluate whether perceived failures
match real documented failures, we conducted a quantitative analysis of real-world PKI incidents
on the web since 2001. Our data comprise reports from Bugzilla, root program operators, academic
literature, security blogs, and the popular press. We determined the underlying causes of each and
reported the results. We identified a gap between experts’ perceptions and real-world PKI incidents. We conclude that there are significant sources of failures of PKI that neither the usability nor
traditional computer security community is engaging, nor can arguably engage separately. Specifically, we found incidents illustrate systematic weaknesses of organizational practices that create
risks for all who rely upon PKI. More positively, our results also point to organizational and configuration choices that could avoid or mitigate some of these risks. Thus, we also identify immediate
mitigation strategies (where feasible).
Key words: public key infrastructure (PKI), digital certificates, certificate authorities, Bugzilla, CA noncompliance, baseline requirements
Introduction
The public key infrastructure (PKI) underlying HTTPS, X.509, was
designed as the foundation for secure online transactions. PKI allows verified parties to digitally bind their identities to domain
names, code distributions, and other digital artifacts. The resulting attestations can be used to verify messages, sign documents,
authenticate and authorize users, and distribute trustworthy code
[1].
This research aimed to determine to what extent the concerns of
security and policy experts aligned with actual failures in the field.
This required both qualitative and quantitative investigations. To do
this, we first documented PKI risk perceptions from qualitative in-
terviews with 18 security and policy experts. We also implemented a
quantitative analysis of data comprising the results of a comprehensive compilation of real-world incidents.
While there is significant research on developing improved PKI
warnings and specific coding errors, there has been less research on
the larger patterns of failures in the system. Thus, in this project,
we collected real-world PKI incidents from 2001 to January 2020
from reliable sources. In these reports, we categorized major types
of PKI incidents according to the cause and the type of incident, the
party responsible, and the public disclosure practices of the entity at
fault. We document numerous cases of failures, compiling a dataset
of incidents identifying trusted but untrustworthy certificates.
C The Author(s) 2021. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial 1
License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is
properly cited. For commercial re-use, please contact journals.permissions@oup.com
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
A holistic analysis of web-based public key
infrastructure failures: comparing experts’
perceptions and real-world incidents
2
Hadan et al.
Motivation
Public key certificates are issued and validated by CAs, which have
their trust-of-anchor certificates in RPOs’ stores. These CAs provide
certificates that attest to the integrity of the ownership of domain
names on the web and enable secure communications. Failures or
flaws in these are communicated to the individuals in the form of
browser warnings, shifting the final risk decision to the person browsing. With current interaction technologies, these have proven only
moderately efficacious. One can argue if these warnings support informed decision-making in the context of the web. Further, there are
increasing numbers of devices and applications, such as the Internet
of things (IoT), with limited user interaction. In these cases, the decision to trust a CA is not made by those who interact with the remote
services. It is made by RPOs and device manufacturers. Yet the Root
Programs were designed for a very different environment, as reflected
by the name of the governance body: the CA Browser Forum. There
is an increasing need to understand the patterns of PKI operation failures on the web as the web-based PKI’s scope expands into IoT, smart
cities, transportation, and even medical devices. Whatever solutions
are adopted in the pervasive and ubiquitous spaces will depend on
1
For the most recent Baselines Requirement, see: https://cabforum.org/basel
ine-requirements-documents.
the current underlying cryptographic infrastructure (e.g. DNSSEC
[2], DOH [3], RPKI [4], BGPSEC [5]) as with these standards, domain names are used as resource locators and X.509 will likely
be included. Even without the inclusion by direct reference in the
standard, understanding these failures can inform risk mitigation in
future trust infrastructures.
We developed our research questions to identify the differences
between experts’ perceptions and the causes of real-world PKI failures. The specific research questions follow, and together we use these
to answer more significant questions. Are the types and causes of failures embedded in these warnings an accurate reflection of experts’
perceptions and ongoing research on underlying risks? Consider the
uniform warnings for PKI failures on the Internet [6]. Do these warnings reflect actual risks? Are they aligned with the modes of failure
observed in the past?
RQ1. What do experts perceive as the challenges in PKI?
A workshop on risks addressing the gaps in research needed to secure the IoT identified seven failure modes in PKI [7], and provided
a foundation for this research. In our study, we began with these
failure modes as the baseline to design the interview questionnaire.
(These are enumerated in the “PKI failure modes” section.) We interviewed 18 policymakers and cryptographers, selecting those with at
least 10 years of experience working in relevant fields. We captured
participants’ perceptions of the scope and challenges of failures of
the attestations embedded in public key certificates. We detail interview results in the “Findings: overview of experts’ perception of PKI
failures” section.
RQ2. What are the actual causes of PKI failures?
Concurrently with conducting interviews with experts, we compiled a comprehensive dataset of over 1800 real-world PKI incidents.
We began by collecting the failures documented on Bugzilla from
2001, examined security blogs from the RPOs, and included media
reports. The goal was to understand the patterns of PKI failures by
examining two decades of incidents. We implemented a metadata
analysis of the nature and the reported causes of incidents. We describe the detailed results in the “Data Analysis of Real-world PKI
Failures” section.
RQ3. What is the gap between the perceptions and real-world PKI
failures?
While describing the results from RQ2, we compared our findings between the interviews (RQ1) and metadata analysis (RQ2) to
identify the existing gaps between real cases of PKI incidents and
the perceptions of risks and failures in PKI by the experts. We find a
significant disconnection between these two. We detail these comparative results in the “Data Analysis of Real-world PKI Failures” section along with our metadata analysis of real-world PKI incidents.
We provide recommend paths forward in the “Discussion” section
and conclude in the “Conclusion” section.
Related Work
This section summarizes previously identified PKI failures in problems with CA practices, recognizing that these overlap. For example, technical failures are often artifacts that embed the failures of
human developers or organization choices. Failures due to organizational decisions are human decision failures. The decision errors resulting from chronic usability challenges of warnings are also design
failures. We provide a rough classification of previous research and
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
We found that experts identified weak cryptography and software bugs as the major and well-known sources of error. However, in
reality, the primary cause of certificate failures is CAs’ misinterpretation of baseline requirement (BR).1 BRs are the standards set by the
operators of PKI on the web, aptly named the root program operators (RPOs). RPOs are the entities that manage the root programs
that define the policies and processes a provider uses to decide which
certificate authorities (CAs) their software or platforms should trust.
We found that violation of the BRs is the cause of 57% of realworld incidents. Other major sources of failure include systematic
organizational flaws, business practices, individual operator errors,
and simply lack of proper auditing. Our results indicate that PKI has
systematic problems that create risks for all who rely upon it. More
positively, our results also point to organizational and configuration
choices that could avoid or mitigate some of these risks.
This work identifies such a significant gap between perceptions
and reality of PKI failures that we close with more questions than we
answer. The core of this work is an examination of PKI failures to
evaluate its vulnerabilities in holistic terms that include code, cryptography, as well as organizational and human dimensions. Based on
the outcome of this research, we propose specific improvements and
countermeasures that can solve common failures that negatively impact the welfare of PKI users. We also identify large challenges that
neither policymakers nor technologies can overcome independently.
We illustrate our motivations and research questions (RQs) in the
“Motivation” section. In the “Related Work” section, we include PKI
failures identified in previous works. We then explain our qualitative interview design and give an overview of experts’ perceptions of
PKI failures in the “Experts’ Perceptions of PKI Failures” section. In
the “Data Analysis of Real-world PKI Failures” section, we describe
our metadata analysis of 1800 reported PKI incidents and compare
the analysis results with experts’ perceptions. We identify where the
perceptions of PKI failures do and do not align with the causes of
real-world incidents. These incidents are analyzed in terms of their
primary cause, root CA, and incident type. Based on that, we develop
recommendations for a more trustworthy PKI ecosystem in the “Discussion” section and conclude in the “Conclusion” section.
3
A holistic analysis of web-based public key infrastructure failures
tinuous) quality control. These are flaws that occur in applicationlevel code.
A significant contributor to these failures is the insecure “reusable
code” from Stack Overflow and search results [26]. It is particularly
hazardous when flawed evaluation software can also result in accepting certificates known to be noncompliant or even untrustworthy. For
example, the study by Brubaker et al. illustrated that various incorrect certificates would be accepted by one SSL/TLS implementation
but rejected by another [22].
Often, errors in TLS/SSL result from the complexity of deploying HTTPS locally. Deployments and configuration create problems
even for professional network operators. The study by Krombholz et
al. indicates that even professionals prefer easy-to-use solutions. [27]
This result highlights the need for a user-friendly HTTPS configuration tool for people at a wide range of security skills levels [28]. Our
work supports this finding as well.
Technical issues
Human factors in daily use
A well-known example of technical issue is the “GOTO FAIL” logical failure in the OpenSSL library [13–15]. As a consequence, every
valid certificate would be accepted for any claimed domain. The core
requirement of matching the key holder with the domain name did
not occur.
In another example of a software failure, the code to evaluate the
validity of a request for Let’s Encrypt certificate only verified the correctness of the request for the first domain when receiving a batched
request. For example, when a certificate for a host is used for many
subdomains or multiple hosted domains (e.g. coolthings.com, coolsmallthings.com, coolmediumthings.com). Again, an attacker could
have used it to masquerade as any trusted entity. With the Let’s
Encrypt code failure, any number of invalid requests would be accepted if the first one were valid (e.g. coolthings.com, microsoft.com,
google.com) [16].
Similarly, a Windows 10 CryptoAPI failure enabled any website
to masquerade as any other [17]. This vulnerability was announced at
an unusual press conference by the NSA and included an illustration
that rickrolled nsa.gov.
Another cryptographic example of software vulnerability is the
usage of vulnerable keys resulted from a flaw in the key generation algorithm (i.e. shared nontrivial common factors due to entropy problems). As a result, RSA private keys for some TLS hosts and SSH
hosts are subject to mathematically tractable attacks [18].
In addition, and well beyond the domain of human errors, cryptographic algorithms depend on the underlying yet unproven mathematical robustness of various constructions. Cryptographic research
can identify new weaknesses in hash and encryption algorithms; advances in hardware or software may make previously intractable and
secure choices straight-forward to defeat. These changes can result
in vulnerabilities. A well-known example is the Flame malware that
leveraged the weakness of MD5 hash function [19,20]. Chosen-prefix
collisions were also used in the creation of rogue CA certificates in
2009 [21].
Another common failure well documented by previous studies is considered a usability failure. This type of failure refers to when the people are presented with an untrustworthy certificate and ignore the
warnings, as opposed to the case where the untrustworthy certificate does not result in warnings. Here, we also include the fact that
the common perceptions about the meaning of digital certificates are
very different from the assurance they are designed to provide.
The study by Rajivan et al. of people’s perception of certificates
reveals that participants were very optimistic about the scope of security and privacy indicated by a lock icon [29]. Common beliefs were
that certificates convey legal accountability of the website, certify that
a website is secure from hacking and has no vulnerabilities, or that it
has reliable data protection and privacy policies. Clearly, this is very
different from the assurances actually intended by compliance with
the CA/Browser Forum and RPO requirements.
In Park’s study of PKI adoption in South Korean, he concludes
that technical excellence was not what led to the success of creating
widespread trust. Rather, the concentration of root authority and the
use of a single, widely trusted platform enabled widespread adoption
of a national infrastructure [30].
Human failures by developers
Software vulnerabilities, which can also be known as human failures
by developers, have been well documented [22–25]. We differentiate
these from the technical failures above. These are failures in small
projects with individual developers rather than flaws in infrastructure
that could have reasonably been expected to have complete (and con-
Organizational decisions
Previous research has documented that organizations do not necessarily take action when vulnerabilities are identified. For example,
the response to the massive exposure of private keys resulting from
HeartBleed was underwhelming. Few certificates were renewed before they expired. Some of the ones that were renewed were downgraded to less secure algorithms, and some organizations paid the
CAs to simply resign certificates associated with potentially exposed
keys [31].
A particularly mendacious example of failure to address a known
problem is discussed in Mozilla’s research of WoSign and StartCom
[32]. In 2016, the weakness of SHA-1, a core hash algorithm, led
to its removal from those that were acceptable to RPOs. All issuers
identified by the RPOs and CA/Browser Forum as trustworthy were
required to stop issuing SHA-1 certificates. Perversely, WoSign found
a business opportunity for issuing backdated SHA-1 certificates. As
a result, in September 2017, the RPOs removed WoSign and its purchased company, StartCom, from trusted CA lists, yet it still remains
in business and consumers may rely upon its attestations in embedded systems.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
close with a set of predefined PKI failure modes in the “PKI failure
modes” section.
The risks of globally approved and centrally administered CAs
have long been recognized. An early criticism of PKI was by Abadi et
al., who proposed a local governance system that allowed individuals and organizations to define their own trusted roots [8]. Schneier
and Ellison identified the risks of the hierarchical model of trust less
than a year after the IETF2000 released the internet protocol standard that supports X.509-based PKI [9]. Fox and LaMacchia argued
for greater transparency and discourse about the development of revocation policies [10]. Camp and Nissenbaum argued that the design
of the PKI architecture was in opposition with human trust behaviors [11]. Much more recently, Ferreira et al. proposed a model that
tackles the challenges of TLS using sociotechnical solutions [12]. Beyond this early discourse, there has been limited work on large-scale
PKI governance, particularly considering sociotechnical factors.
4
Hadan et al.
In 2017, Symantec, which managed several CAs, including
GeoTrust, RapidSSL, Equifax, and VeriSign, was also found to be untrustworthy after issuing over 30 000 certificates that did not comply
with the BRs [33–36]. As a result, Symantec was required to sell its
PKI business to DigiCert. This was criticized by those who thought
the patterns of organizational behavior by Symantec were a result
of market power and concentration (e.g. too big to fail), and thus
simply shifting to DigiCert would not be a solution. In our evaluation of PKI failure incidents, we found that the purchase of Symantec
corresponded with an increase in the incidents of flawed certificates
released by DigiCert, illustrating that the transfer of ownership did
not resolve the underlying issues.
We used this set of failure modes as the baseline to design our
interview questionnaire. We describe our interviews questionnaires
and findings in the next section.
Experts’ Perceptions of PKI Failures
In this section, we first explain the development and the deployment
of interview instrument. We then give an overview of our findings—
the experts’ perceptions of PKI failures. The goal is to answer our
RQ1: what do experts perceive as the challenges in PKI?
Problems in CA practices
The hazards of global trust are exacerbated by the fact that digital certificates can fail in issuance and authentication. Previous empirical evaluations of the ecosystem have been strictly quantitative,
using large-scale sampling. An early survey that highlighted systematic problems in CA practices, particularly the existence of digital
certificates not compliant with the BRs, was done by researchers of
INRIA and Microsoft [37]. Further, cryptographically attested but incorrect facts and flawed cryptography were found to be common in
banking certificates [24]. Kumar et al. illustrated that some CAs have
been issuing erroneous digital certificates for the entire lifetime of the
CA [38]. Numerous misconfigured digital certificates were found by
Gasser et al. in their research [39]. These certificates were collected
by network scanning and querying of Certificate Transparency logs.
Our analysis of incidents includes all those documented in the Certificate Transparency logs. Like Dong two years earlier [24], Kumar
[38] found certificates with false information and flawed cryptography. Their targeted analysis showed that these were more common
in banking certificates than in nonbanks.
Known CAs incidents such as CA liability problem and technical
weakness were also discussed by S. Roosa et al. [40]. We return later
to how their conclusions are echoed with our reported analysis of
PKI incidents.
PKI failure modes
A Chatham House rule workshop held in August 2017 in Seattle,
WA, focused on identifying a path toward a trustworthy IoT [11].
This workshop included at least one senior security professional from
each organization that operates a Root Program and senior cryptographers from three organizations. In addition, there were privacy
and security experts from academia, industry, and government. The
workshop included discussions about cryptography, secure coding,
isolation, recovery after subversion, market transparency, usability,
and the roles of industry, government, and consumer advocates in
each of these domains. The discussions on recovery and cryptography identified failure modes in PKI. The set of failure modes identified
is as follows:
r no certificate
r expired but semantically correct and secure cryptography
r valid certificates with invalid facts (either issued incorrectly or
r
r
r
r
based on facts that have changed)
corrupted certificate authority (i.e. rogue certificates)
software failures accepting invalid certificates
lack of agile, strong cryptography (e.g. weak cryptography)
usability failures
We began our work with a literature review to investigate previously
documented PKI failures. In addition, we integrated a set of failures
modes developed from the discussion in the Chatham House rule
workshop [7] (see the “PKI failure modes” section).
We interviewed experts at two security conferences: 46th Research Conference on Communications, Information and Internet
Policy (46th TPRC) and the Real World Crypto (RWC). The goal
of the interviews was to capture experts’ perceptions of the current
scope and challenges in PKI. We obtained approval from the University Institutional Review Board (IRB) in May 2018. Then we held
the interviews in the 46th TPRC conference in September 2018 and
in the RWC 2019 symposium in January 2019.
We sought a heterogeneous set of participants, combining participants with high technical expertise and some policy literacy with
participants with high policy expertise and some technical literacy.
The goal was to generate a consensus document informed by the literature noted based partly on previous successes in combining such
populations [7, 41–43]. In addition, we observed disagreements between the policymakers and applied technologists at the previous
Chatham House rule workshop, particularly about the frequency and
modes of failure [7]. For example, the proposal of disposing all IoT
devices upon failure or vulnerability was embraced by policymakers.
However, this solution was rejected by cryptographers and consumer
advocates as the policymakers assumed that failures were infrequent
and isolated. The fact that the policy community identified some concerns as technical problems while the technologists identified these as
policy issues reifies the need for participants of both types. Thus, we
sought to identify the intersection of the concerns of the two communities.
We recruited our participants using a snowball selection method.
We began by talking to conference attendees who we knew had at
least 10-year experience working in information security or information policy. The participants were then asked to introduce other
potential participants with similar work experiences. In the end, we
interviewed one policy journalist, nine policy researchers, two advanced doctoral researchers who attended the 46th TPRC conference
in September 2018, and six expert technologists who attended the
RWC 2019 symposium.
In our interviews, we asked participants to identify failure modes
of PKI, the role of domain names in IoT, and possible solutions as
PKI evolves. We queried their opinions on the current PKI and their
advice on improving PKI in the future. We also asked about their
personal experience with certificate warnings and malicious websites.
We closed with questions about building a trustworthy Internet without assumptions about user interfaces and device capabilities. Each
interview took 30–45 min. All interviews were audio-recorded, transcribed, and coded. We stored the transcripts anonymously with only
a participant ID in our database. We destroyed the audio records as
the voices are identifiable. We include the semi-structured interview
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Interview development and deployment
A holistic analysis of web-based public key infrastructure failures
Findings: overview of experts’ perception of PKI
failures
Here, we present an overview of the findings from the interviews with
policy experts from the 46th TPRC conference and applied cryptographers from the RWC 2019 symposium. Among the participants’
responses, most participants agreed that the worst-case failure mode
is when an untrustworthy certificate is treated as trustworthy. The
major causes of PKI failures mentioned by participants were: weak
cryptography, cryptographic verification of invalid facts with the extreme case being rogue certificates, software that incorrectly classifies
an invalid certificate as valid, and ineffective warnings (i.e. usability
issues). Below we detail the PKI failures that our participant experts
well discussed in the interviews. In the “Findings: comparing realworld PKI failures vs experts’ perceptions” section, we enumerate
real-world PKI incidents, and then compare.
Missing certificate
In our interview, missing certificate was seen as an apparent vulnerability. Without certificates, authenticating credentials would not be
kept secure, privacy is not provided, and data transactions could be
compromised. Individuals submitting credentials into insecure forms
and over insecure channels occurs, and this may not be visible to
the user. Note that the HTTPS Everywhere effort addresses this risk,
among others [45]. This risk is being exacerbated by the lack of inclusion of security or attestations in standards for IoT authentication
and software update information [46].
Certificate expired
The second type of failure discussed by experts was expired certificates with unchanged facts. In this situation, the certificate is expired
but the associated facts, such as certificate ownership and basic business information, are still valid. In this case, the entity is still trustworthy, but warnings are generated nonetheless.
Most participants had experienced this situation along with the
other false positive warnings. Some of the participants reported that
they faced warnings on expired certificates frequently. Policymakers
tended to believe that the warnings were due to maintenance errors
or the laziness of maintainers. In this case, they managed to ignore
the warnings and thought the site they visited was legitimate based
on their familiarity. Technologists also noted server configuration issues, inconsistency between platforms, and inconsistency between
browsers in addition to maintenance errors. As opposed to policymakers, half of the technologists would assume that the site was
compromised, not because the certificate itself is incorrect but because expiration is seen as a sign of bad management. Despite that,
instead of leaving, they would be curious and explore the site to determine the cause of the warning. This may be inadvisable for those
with less technical expertise. It drove the requirement for improving
warnings and negative indicators.
Valid certificates with invalid facts
The third type of failure mentioned in our interviews was valid certificates with changed facts. In this situation, a valid certificate was
issued, but the associated facts are changed. By using a valid certificate, an untrustworthy entity is recognized as trustworthy. The
reason could be that the certificate’s issuer did not correctly verify
the requester or due to a rogue certificate issuance. Half of the participants reported having connected to malicious websites without
receiving any warnings. These responses argued for improvement
in warnings and negative indicators. Many participants, primarily
technologists with a few policymakers, believed they had experienced this situation but were uncertain. One technologist pointed
out that the certificate on a site does not tell people the intent of
the site operator. Having a valid certificate only means the certificate matches the domain the user attempts to visit. This observation
has been supported by quantitative research on the use of HTTPS
in fraud [47–49]. One step toward addressing (at least) malicious
reregistration was to link certificates more closely to the domain registration process although none of the participants explicitly mentioned DNSSEC [2]. If a domain owner changes, the reregistered
certificates should expire before or concurrently with the domain
registration.
Corrupted certificate authorities
The fourth type of failure discussed in the interviews was corrupted
certificate authorities. This type of failure drove recommendations
for people, organizations, and devices to have as a default fewer
trusted CAs. This recommendation was supported by analysis by
Perl, Fahl, and Smith, who identified hundreds of trusted by unneeded
certificates in an analysis of organizational PKI policies [50]. It also
identified the critical need for more efficacious responses to rogue incidents. Rogue CAs were seen as a severe and dreadful but rare threat
by the interviewees in general. There are significant research efforts
to improve speed and accuracy of identification of rogue certificates
that was noted by one participant [24], and Transparency seeks to
make these failures public [51].
Experts’ recommendations
We closed each interview by asking for participants’ recommendations to improve PKI. One policymaker recommended creating a
process of verification of principles in practice with improved auditing. The goal was to create a transitive relationship where a group
of manufacturers and technology providers can trust each other because they have all made verified commitments to the same principles
and are audited by the same parties. That, in turn, could be communicated to customers, including decision-makers in a smart city or
safety-critical domains, in the manner that attestations about other,
physical safety-critical components are validated. This is similar to
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
protocol in Appendix 1. These questions were used to open the discussion in our interviews.
We analyzed interview responses using inductive qualitative coding [44], a ground-up approach where researchers derive codes
(i.e. themes) from the data. This approach enables the systematic
repeatable extraction of conclusions from structured qualitative data.
Three doctoral researchers and two undergraduate researchers analyzed the seventeen responses. We eliminated the response from one
participant because the policy expert answered every question with
some hostility (e.g. every answer was “I do not know”). Each of the
five researchers read the transcripts separately and developed an initial list of themes. We then synchronously and in-person met to create, discuss, revise, and expand codes. In the end, we developed two
codebooks, one for the 46th TPRC conference and one for the RWC
2019 symposium. We calculated the Cohen’s Kappa coefficient of
each codebook to ensure inter-coder reliability [44]. Cohen’s Kappa is
used to measure the agreement between multiple coders. The Kappa
coefficients for the resulting codebooks differed but were both over
75% (TPRC Kappa = 0.751, RWC Kappa = 0.842), indicating
a substantial inter-coder agreement. We include the codebooks in
Appendix 2.
The interviews combined questions from two research projects,
setting standards in the IoT and perceptions of resilience in PKI. In
this paper, we focus on perceptions of resilience of PKI.
5
6
(i) Improvements in warnings, particularly negative indicators
(ii) Linking certificates more closely to the hosting or manufacturing process
(iii) Lifetimes that do not exceed the life of the domain name.
(iv) Fewer roots of trust.
(v) Quicker and more effective identification of and responses to
rogue incidents.
(vi) Audit to support positive indicators and increase compliance
with requirements.
(vii) Increased collaboration between cryptographers and policymakers.
The roots of trust (i.e. CA) failure mode was associated with rogue
certificates. Those that mentioned having fewer roots of trust indicated that rogues were a small but critical problem. Having fewer
roots decreases the exposure of individual users to failures in the
infrastructure. The codebook for each question in the qualitative
coding of the interviews is included as an appendix (Appendix 2,
Table A1).
All participants highlighted the importance of transparency, especially in IoT, where there is no opportunity for traditional user
interaction. Improved transparency is needed to ensure that people
(and devices) connections are made only to servers or services with
compliance attestations. The standards that are built on PKI depend
upon this assurance. Beyond standards such as DNSSEC, [2] DOH
[3], RPKI [4], and BGPSEC [5], the emergence of Manufacturer Usage Description (MUD) standard [52] in the IoT space can meet this
requirement depend on the widespread adoption of a trustworthy
PKI.
Data Analysis of Real-world PKI Failures
Concurrently with the qualitative team interviews, the quantitative
lead compiled a comprehensive dataset of real-world PKI incidents.
We first explain our methodology and data sources. In the “Findings:
comparing real-world PKI failures vs experts’ perceptions” section,
we enumerate the details of the findings and compare these with the
experts’ perceptions. The goal was to answer RQ2: what are the actual causes of PKI failures by analyzing PKI incident reports. And
we aim to answer RQ3: What is the gap between the perceptions
and real-world PKI failures? by comparing the data analysis with
experts’ perceptions.
Metadata collection, compilation, and analysis
This section describes how our data were compiled, including the
specific fields in the dataset, and discusses the coding process. We then
compare these results with the qualitative coding of the interviews of
the experts.
Data collection
To evaluate the failures and challenges of real-world PKI incidents,
we analyzed the metadata of incident reports using quantitative data
analysis.
Our data sources had to be public, reliable, impartial, and trustworthy. These requirements eliminated incidents published in media
without proper sources, e.g. medium blog posts. The backbone of
our incident collection was Mozilla’s Bugzilla.2 This source met the
requirements and offered a database of public incidents related to
PKI. After an analysis of over 1800 reported “bugs” in the platform, we collected 557 unique and unrepeated incidents; 474 came
from this data source. To assure comprehensive coverage and check
the consistency of the information on Bugzilla, we relied on other
sources of complementary information. Specifically, we read the discussions on the ‘mozilla.dev.security.policy’ Google Group3 (a channel for bug discussions), and crt.sh where further information about
a particular digital certificate can be found (information that is directly taken from “Certificate Transparency” logs). Beyond this, we
reviewed the Google Security Blog,4 Mozilla Security Blog,5 and incidents reported on Microsoft’s Security Response Center.6 Other data
sources included the academic literature [21], industry research reports [53], technical reports from SANS [54], and CA self-disclosures
[55]. When there were multiple reports of a same incident, we used
additional sources to confirm the information in the Bugzilla reports.
We then implemented the qualitative analysis on Bugzilla, treating it
as authoritative because of its consistency in reporting format. The
resulting data covers 2001 to January 2020 and includes 557 incidents, many reported in multiple information sources.
Quantitative data compilation
We compiled the metadata into a comprehensive dataset of PKI incidents. For each incident, we include the following data fields:
r The year in which the incident was reported. This value equals
the year the incident happened; however, this is not always true,
especially where incidents lasted over a year.
2 Bugzilla: https://bugzilla.mozilla.org.
3 mozilla.dev.security.policy’ Google Group: https://groups.google.com/foru
m/#!forum/mozilla.dev.security.policy.
4 Google Security Blog: https://security.googleblog.com/.
5 Mozilla Security Blog: https://blog.mozilla.org/security/.
6 Microsoft Security Response Center: https://msrc-blog.microsoft.com/.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
the CA Browser Forum and could be informed by the results of this
work.
In contrast, more than one technologist proposed that government bodies or NGOs should be involved in the PKI process to eliminate the occurrence of rogue certificates and problematic CAs.
At a minimum, these discussions and responses drove the recommendation of audits to support meaningful positive indicators and
reify the need for improvement in warnings and negative indicators.
Neither policymakers nor technologists could provide a clear list
of requirements for CAs. Technologists pointed out that trust is not
a problem that technology can solve. Therefore, they suggested that
creators using PKI in their products should prepare clear public statements about which principles they rely on. Policymakers indicated
that this is a technical problem of usability and risk communication
of attestation. One policymaker said, “be careful not to create more
problems while innovating.” One cryptographer said nearly the same
when asked if he wanted to communicate to policymakers: “Be careful!”
Each group brought a more complete understanding of the limits of their sector to address the underlying problem. Both groups
sought action from the other, indicating a need for greater collaboration and improved communication work between the cryptographic
and policy-making communities.
After developing codebooks, we sought the similarities between
the technologists and policymakers. There was nontrivial disagreement, as noted above, where each had recommendations for the others. However, after having coded the interviews, the results were
seven suggestions that were mentioned multiple times and for which
there was no disagreement. Rather than a statistical analysis, this represents a summary consensus of the experts. The suggestions for future development are as follows:
Hadan et al.
A holistic analysis of web-based public key infrastructure failures
r The responsible entity in the incident, which in almost every case
r
r
r
Data analysis
We used iterative qualitative coding to classify each incident.
The incident reports were first read by a primary doctoral researcher who developed a codebook of possible causes. This codebook and at least one example of an incident with each cause were
reviewed by the other authors (two doctoral researchers and one
faculty member). In the next step, the codebook was evaluated by
a focus group of eight researchers using selected incidents to determine if the codebook was usable and understandable. At this point
in the qualitative process, each incident had been classified at least
once. The codebook was complete and had been further validated in
a focus group that consisted of graduate researchers and faculty in
information and computing science departments. As the final step,
one undergraduate computer science researcher classified every incident report using the codebook, while three additional undergraduate researchers categorized two-thirds of the incident reports (∼370
reports out of 557 reports). Each researcher categorized the incident
reports separately. Any causes that did not fit obviously into the set
of causes identified in the codebook were discussed at weekly meetings. Every incident has been read and classified at least five times,
with every incident having at least one identified cause. Multiple incidents have a set of causes, in which case coders identified a dominant
cause.
The codes from the interpretation of incident reports were directly derived from metadata. These results are used to identify patterns and trends in incidents, which we will discuss next.
We created multiple graphs to help us better understand the data.
Our quantitative exploration of the gathered data allowed the identification of latent unclear phenomena that have been undermining the
trustworthiness in PKI for many years. The result was a macro-study
of responsible Root CAs and other entities, the most common types
of incidents and their primary causes, and the practices of public disclosure of these incidents. It was also a yearly comparison of incidents
(from different perspectives) and a survey related to the issuance of
rogue certificates.
After detailing the analysis of the incident reports, we compare
these with the results of the interviews.
Findings: comparing real-world PKI failures vs experts’
perceptions
Our results illustrate that the failure types from the interview and
a dozen core causes in a few CAs are responsible for the majority
of these. Here, we summarize our findings of metadata analysis as
compared with the interview results.
The findings are presented in terms of the frequency of incidents
across CAs, type of incident, and causes. We initially sought to weigh
the harm done by each incident but found this infeasible. It is because
not all failures are equally harmful. For example, backdating a certificate is somewhat trivial for operational purposes but is dangerous
when used to keep weak cryptography in the field. Similarly, failure
to confirm compliance with agreements with trademark owners may
be harmless or may underpin a global fraud operation [56].
In Fig. 1, we show the frequency of each type of incidents, their
causes, and the relationship between them. Again, we include the explanation of each incident type and cause with examples of bug reports in Appendices 3 and 4.
Common type of incidents
First, we classified incident types. We found that the most common
type of incident is the existence of Fields in certificates noncompliant
to the BR (see Table 1). This type of incident leads to any other by a
wide margin. It is also easy to trivialize; however, most of these noncompliant fields were issues of semantics and syntax. The other three
leading incident types are Audit report failures, Non-BR-compliant
revocation info, and Serial number failures. These incidents reflect
serious issues: an inability to identify and revoke a flawed certificate,
a failure in the fundamentals, and a lack of entropy. The first subversion of TLS was based on the lack of entropy. It remains an issue
in 2020, suggesting that the advances in the PKI ecosystem have not
yet resolved all the underlying problems [57]. Note that the lack of
an audit report reflects a recommendation that was identified in the
qualitative component of the research, specifically the need for highly
trustworthy audits and corresponding indicators.
Overall, we only found one incident related to IoT devices. In this
incident, a major IoT vendor used SHA-1 for its long-lived IoT hub.
On the contrary, in our interviews, experts believed that the current
PKI mode is marginally acceptable for the web but is not adequate for
the IoT in terms of providing good enough security. Experts generally
believed that IoT devices offered lower usability for communicating
warnings and PKI errors than other devices.
Common causes of incidents
Here, we enumerate the different causes of PKI incidents to answer
our RQ2: What are the real cases of PKI failures? The common
causes of incidents are summarized in Table 2. Again, for incidents
with multiple causes, we list the most deterministic and hazardous
cause.
The interviews indicated an intense focus on the endpoint at the
critical failure mode. Our experts argued that the likely cause of incidents originated in the local management of websites they visited. Recall that the failure types mentioned in interviews primarily focused
on the missing or validity of the certificates. Both policymakers and
technologists perceive that failures are most likely to originate in the
internet browser or server associated with a specific visit or service
request. However, most parties assumed that the corresponding CAs
correctly issued certificates to begin with. This assumption appears
unduly optimistic.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
r
was a CA; however, there were cases where the incident was generated in Intermediate CAs, Resellers, Registration Authorities,
or Auditors.
For each incident the related Root CA, whose root digital certificates are included in the Root Programs.
Which party disclosed the incident is a Boolean variable identifying if incident was disclosed by the responsible entity.
The type of incident, meaning the problem that was identified in
the disclosure. The detailed explanation of codes for incident type
with example bug reports can be found in appendix (Appendix
3, Table A2).
The cause of the incident. When multiple causes are identified,
we list the deterministic and hazardous cause based on the interviews, research, and potential harm. For example, when software bug and BR misinterpretation are identified, we select software bug given the existence of it is more certain. When weak key
and lack of revocation information are both identified, we select
weak key as the cause given it is the primary cause to the certificate failure. The detailed explanation of codes for incident causes
with example bug reports can be found in appendix (Appendix 4,
Table A3).
7
8
Hadan et al.
Table 1: Most repeated types of incidents, number of reported incidents by an entity other than the CA, number of reported incidents by
the CA, and percentage of CA self-reported incidents compared with the total.
Incident
Third-party reported
Self reported
Total
Self reporting rate
Fields in Certificates not compliant to BR
Audit report failures
Non-BR-compliant revocation info
Serial number failures
513/1024 bits key
Possible rogue certificates
Undisclosed SubCA
Failure to revoke certificates in a timely manner
CAAa misissuance
Verified rogue certificate
Insecure hashing (SHA-1/MD5)
CA/RA/SubCA/reseller hacked
Otherb
123
47
32
11
24
11
13
5
11
11
8
7
49
91
17
8
29
1
6
4
9
2
0
3
3
32
214
64
40
40
25
17
17
14
13
11
11
10
81
42.52%
39.41%
20.00%
72.50%
4.00%
35.29%
23.52%
64.29%
15.38%
0%
27.27%
30.00%
39.51%
Total
352
205
557
36.80%
a
b
DNS Certification Authority Authorization (CAA) Resource Record, RFC 6844.
Other includes incident types that occurred ≤5 times.
In fact, our metadata analysis indicates that the most common
cause of real-world failure is CAs’ misinterpretation/unawareness of
BR. It was identified as the primary cause of 111 out of 557 incidents.
The high percentage illustrates either a lack of policies or a lack of
process to ensure these are followed. The same trend also showed up
in the interviews. The policymakers could not list any requirements
that should be implemented to regulate PKI and CAs in the future.
The second most common cause was Software Bugs, which was
identified as the primary cause of 87 incidents. Software failures are
a focus of vulnerability disclosures (e.g. [31, 13, 17, 14, 15]), tra-
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Figure 1: Types of PKI incidents and their causes. Note that Y-axis includes the incident types and X-axis includes the causes. We listed the top 12 types of PKI
incidents in this graph. Notes: 1, ’Other’ includes incident types that occurred ≤5 times; 2, ’Other’ includes causes that occurred ≤5 times; 3, No data includes
reports without detailed information about the cause at the time of coding.
9
A holistic analysis of web-based public key infrastructure failures
Table 2: Most repeated cause of incident, number of reported incidents by an entity other than the CA, number of reported incidents by
the CA, and percentage of CA self-reported incident compared with the total. Recall that when multiple causes are identified, we list the
most deterministic and hazardous cause.
Third-party reported
Self reported
Total
Self reporting rate
Believed to be compliant/misinterpretation/unaware
Software bugs
Operational error
Human error
Business model/CA decision/testing
Nonoptimal request check
Change in baseline requirements
Improper security controls
Inadequate audit compliance
Organizational constraints
Othera
No datab
65
41
41
41
58
17
13
7
4
5
14
46
46
46
35
32
11
8
8
4
4
2
4
5
111
87
76
73
69
25
21
11
8
7
18
51
41.44%
52.87%
46.05%
43.84%
15.94%
32.00%
38.10%
36.36%
50.00%
28.57%
22.22%
9.80%
Total
352
205
557
36.80%
a
Other includes causes that occurred ≤5 times. No data includes reports without detailed information about the cause at the time of coding.
b
ditional computer security research [22, 18], and research into usability focused on developers (e.g. [26, 58]). We acknowledge that
errors, bugs, and vulnerabilities may be present in every system or
application. It is a major challenge that is both a component of ongoing research and indicated in the interviews (see Appendix 2 for
interview codes).
Another severe cause worth mentioning is Business model/CA decision, which was identified as the primary cause of 69 incidents. In
this situation, the CAs decide not to comply with the requirements
set by the RPOs and the CA/Browser Forum. Often there is an obvious profit motive aligned with the decision to violate these guidelines.
However, none of the experts mentioned this cause in our interviews,
reflecting the importance of auditing and verification. More interview
participants identified end-users and network engineers than CAs as
bad judgment sources (e.g. ‘lazy’ site maintainers).
As a major point of agreement, our metadata analysis shows that
the issuance of rogue certificates is an unsolved problem. We found
28 incidents related to the possible or actual issuance of rogue certificates, linked to 12 Root CAs, and distributed over the range of
time studied (2001–January 2020). On the contrary, experts in our
interviews expected PKI to play a more significant role in preventing
phishing attacks or fraudulent and malicious sites. The same perception was also shown in a previous phishing resilience test. Participants assumed that they have a “secure connection” with a malicious
domain if it allows HTTPS connections [59].
Top problematic root CAs
Third, we classified the responsible entities of problematic certificates. We found that the top 15 Root CAs with the most public incidents registered were present in 51.71% of the total incidents studied.
(see Fig. 2) The other 48.29% of the incidents occurred due to the
other 83 CAs. In addition, the number of problematic Root CAs are
increasing over the year (see Fig. 3)
Incidents caused by the Problematic Root CAs were more numerous and more likely to be associated with subCAs and intermediate
CAs than other incidents. Note that the most problematic CAs with
problematic certificates are not merely the most prolific nor the ones
with the largest customer base. CAs with a history of being systematically unreliable should not receive the same indicator and default
inclusion as CAs with histories of excellence. Familiar and unfamiliar CAs should be identified differently. Jurisdiction of CA may be
a valuable indicator in addition to the history of the CA. More nuanced indicators can provide levels of warnings, and such customized
warnings should be a focus of research. Now indicators are binary:
the certificate checks out, or it does not.
Experts argued that CAs providing digital certificates for free is a
significant source of rogue certificates. Their opinions were that these
CAs had been associated with the issuance of certificates with malicious motivations behind them. However, in our metadata analysis,
we identified 0 rogue certificate incidents associated with those CAs,
which means they are not more unreliable than other CAs. For example, only 10 out of 557 incidents were associated with Let’s Encrypt,
which has a very high self-reporting rate (70%). Within these 10 incidents, only 1 of them possibly resulted in a rogue certificate. All the
other 27 rogue certificate incidents were generated by misbehaving
for-profit companies, hacked for-profit companies, and governmental
agencies. Thus, free CAs appear more reliable than other CAs. Examining the certificates that are legitimate but associated with malicious
domains (such as phishing sites) is a subject of ongoing data compilation and future analysis.
Evolution of self-reporting rate
Our analysis showed a trend of increasing transparency and selfreporting. Such disclosures are arguably a positive externality of the
mandated use of “Certificate Transparency” in the issuance of digital
certificates. Issued certificates are now publicly logged and monitored
by third parties. The discovery of faulty certificates has been made
both more likely and easier, particularly with the inclusion of different tools tailored to the analysis of the logs platforms (i.e. lints). Annual disclosures are shown in Fig. 4. The figure ends in January 2020
not only because it is necessarily the case that disclosure lags issuance
and detection but also because disclosure may be delayed for remediation. For example, the Let’s Encrypt error that allows misissuance
of certificates in bulk requests was not reported until remediation to
prevent abuse [16].
Even with increased transparency, the lack of disclosure practices
by CAs remains an important contributor to reasons not to trust
the current. Only 52 out of 98 Problematic Root CAs (53.06%) had
made at least one incident disclosure despite dominating in a number
of incidents. Further, incident disclosures do not always correspond
to immediate and effective revocation. See Fig. 4 for the self-reporting
rate by year.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Causes
10
Hadan et al.
Figure 3: The chart shows the distribution of incidents across CAs by year. This illustrates both the overall increase and the increasing number of CAs
associated with an incident. This illustrates that there are few CAs that have chronic issues with incidents.
Figure 4: Incidents per year, indicating which of these were self-reported by the CA (orange) and those identified by a third party (blue). Note that third-party
disclosures dominate every year, with an improvement in that ratio in 2019.
Discussion
Our results indicate a clear mismatch between experts’ perceptions
and the causes of real-world PKI failures. Experts primarily focused
on the endpoint (e.g. website warnings, missing certificates), assuming that corresponding CAs behaved correctly. On the contrary, the
metadata analysis of real-world incidents indicated that the major cause of PKI failures is the systematic problems with CAs. For
example, the misinterpretation of BR and CAs’ decision to fail to be
compliant. Arguably, the CAs are not as trustworthy as expected. The
particular areas of disconnect reify the need for more collaboration
and communication in PKI governance.
Table 3 shows a more precise comparison between the experts’
perceptions and the metadata analysis of real-world PKI incidents.
Notice that we neither include end-user comprehension of certificates
nor include the lack of usability of current warnings. We only indicate that these warnings do not align with the underlying failures.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Figure 2: Count of incidents by Problematic Root CAs. The top 15 Root CAs accounted for 51.71% of the total incidents.
11
A holistic analysis of web-based public key infrastructure failures
Table 3: Differences between perceptions and real cases of PKI failures. Note that this study did not include user misconceptions and
usability issues, thus the asterisk. Usability concerns and understanding of certificates is an area of active research, e.g. (Krombholz et al.
2019 [60]; [6])
Element
Incident types
Causes
Real-world incidents
r
r
r
r
No certificates
Expired certificates with valid facts
Valid certificates with invalid facts
Corrupted CAs result in rogue certificates
r
r
r
r
Incorrect certificates
CA security incidents (hacked)
Noncompliant CAs
Only a few corrupted CAs
r
r
r
r
r
r
Maintenance error
Untrustworthy CAs (generate certificates for free)
Key management
Inconsistency between browsers/devices
Server configuration issues
malicious website operators
r
r
r
r
r
r
CAs’ misinterpretation/unawareness of BR
Software Bugs
Human or operational error
CA decision to not compliant with BR
CA with improper security control
Free CAs are trustworthy than other CAs
r
r
r
Problematic PKI model
False-positive warnings
Loss of data confidentiality
r
r
r
Rogue certificates
Untrustworthy CAs
Unreliable audits, low self-report rate
The lack of comprehension and usability of standard TLS/SSL warnings is well known in the usable security community, including a wide
range of technologies, and social scientists from the public and private sectors.
In our interview, experts expected PKI to play a significant role
in preventing phishing attacks. However, our metadata analysis indicated that the issuance of misleading certificates is one of the least
solvable problems. Since the interviews, phishing sites have adopted
to HTTPS Everywhere, and the majority of phishing sites are now
SSL enabled according to the Anti Phishing Working Group (APWG)
[61]. The purpose of certificates on the web was to authenticate
domain names when they were considered critical signposts in cyberspace (to quote the title of the National Academies report [62]).
Now, with HTTPS Everywhere, web certificates play less of a role
in distinguishing legitimate sites due to the misconception of HTTPS
Everywhere. The green or gray “lock” icon does not imply the domain is not malicious. It is not an effective strategy for distinguishing
miscreants.
All experts reported having visited malicious sites where no warnings appeared, and all participants visited legitimate websites that
generated warnings. This result reflects that effective communication
to users requires the differentiation of types of risk posed by the certificate. The collapse of all possible risks of legitimate or miscreant
certificates into a single interaction is a major weakness in the current system. Addressing this requires significant reconsideration and
improvement in warnings, including personalization of warnings and
risk communication [63].
Given the high number of noncompliant certificates (214 out of
557 incidents), we recommend that warnings distinguish between an
expired certificate, an otherwise noncompliant certificate, certificates
from an unknown root, unfamiliar root, or a new CA for a familiar site under the recommendation for more effective, more nuanced
warnings. Note that other fields where there is noncompliance do not
necessarily generate warnings even when these are more dangerous,
specifically lack of revocation information.
Our metadata analysis identified CA’s Misinterpretation/Unawareness of BR as the most common cause of real-world
failures, which experts did not realize in the interviews. The dramatic changes in the security of Microsoft code from the nineties
to recent times show the potential of organizational commitment,
such as an internal cryptography review board. Thus it is reasonable
to assert that both policies and practices need to be improved
in the CAs that repeatedly report these incidents. In addition, it
is also concerning that a major cause of incidents is the CAs’
self-admitted unawareness of their commitments to comply with
standards.
Although Software bugs may be present in every system or application, having it as the second most repeated cause of incidents is a
signal that some practices could be improved in the CAs and libraries
commonly used to validate certificates (see Table 2).
While Extended Validation (EV) certificates were intended to improve verification in the requester verification process, there is no
consensus that their benefits are widespread, significant, or even extant. In our interviews, more than one participant suggested that EV
certificates correspond to the original expectation of basic certificates
in that they provide some indication of the legitimacy of the holder.
This confirms previous research by Gutmann on the lack of verification associated with EV certificates [64].
In addition, it seems that the current structure for digital certificates issuance and CAs’ operations is not transparent, clear, nor accessible. All certificates appear equal to any but the expert. Some experts in our study said that there is not enough reputation information in the CA market. Different participants were concerned about
different jurisdictions or the provision of free certificates. Our study
did not include the revocation of certificates used for phishing domains, so the concern in the second case cannot be addressed in these
conclusions.
Our perceptions study also found that the actual requirements for
CAs are not very well understood. When asked about what requirements should be implemented to regulate PKI and CAs in the future,
none of our participants could provide a path forward. We found that
CA governance is mainly defined by RPOs and the CA/Browser Forum. The latter regulates and publishes the BRs for digital certificates
and CA practices, which can be considered as the de facto standard
for PKI. Entities in the former group define a corpus of requirements
that CAs must comply with to have their root certificates in the different root programs.
In addition, we consider two additional sources of CA governance. First, CAs’ operations may need to meet certain restrictions based on specific countries’ laws and regulations. For example,
WoSign issued a statement that it was forced by the Chinese government to issue digital certificates using the elliptic curve SM2 al-
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Consequences
Experts’ perception
12
Hadan et al.
Conclusion
We offer this work as a contribution to the understanding of the
scope and limits of PKI governance model. One core and unexpected challenge was while the failure modes were the subject of
much agreement, resolving these failures was identified as a technical challenge by policymakers and a regulatory challenge by technical
experts.
The use of PKI to support secure communications and implement
key exchange is an immense success. It is well understood that the
use of PKI for authentication and communicating information about
the degree to which a site is trustworthy has not been so successful.
Based on the comparison of interviews and metadata analysis, we
found that the frequency of Certificate Authority noncompliance is
a more significant source of vulnerability than generally documented
and discussed. Such noncompliance is not limited to rogue certificates
and hacked CAs but rather standard operating procedures dominate
the creation of risks. However, as PKI is being used in IoT and CyberPhysical Systems (CPS), problems that were arguably reasonable as
standard operating procedure are now less so. A decision to accept
risk in web commerce should be treated as distinct from a decision
to accept risk in CPS.
Our interviews and metadata analysis result in the following recommendations, which align with current research. Note that all of
these are sociotechnical requiring interdisciplinary and cross-domain
efforts.
7
Baseline Requirement Section 6.1.5: https://wiki.mozilla.org/CA:WoSign_I
ssues#Issue_P:_Use_of_SM2_Algorithm_.28Nov_2015.29.
r HTTPS Everywhere should differentiate invalid certificates and
lack of a certificate.
r There should be faster blocking of rogue certificates.
r There should be investments in improved software development.
r There should be a clear strategy to address the requirement for
more agile cryptography.
Other recommendations result from the metadata analysis. These
are arguably a component of usability failures, as warnings are now
frequent and frequently meaningless. We did not include usability in
our inquiry. Yet that both sets of participants consistently raised it
indicates both its importance and the awareness of its importance.
r fewer or easily rejected warnings for cases with expired certificates previously accepted by the use, with likely valid facts
r stronger warnings for valid certificates with invalid facts (such as
a potential rogue or after a website changes ownership)
r lack of revocation information for an otherwise compliant and
r
r
correct certificate should result in a specific indicator expressing
uncertainty
fewer trusted roots by default
disincentives for software that accepts invalid certificates, which
complements the requirement for investments in improvements
in open code
In addition, our results found that there are a few systematically
responsible CAs. These are in direct opposition to current practice:
r provide interactions that distinguish between consistently rer
sponsible CAs and others, and between chronically irresponsible
CAs
provide interactions that identify CAs with legal requirements
that violate standard CA requirements (i.e. mandatory weak
cryptography, law enforcement copies of private keys)
Future research includes evaluating simple but more precise
warnings that differentiate: certificates that might have been revoked,
certifications from a remote and unfamiliar CA, cryptographically
weak certificates, certificates from chronically responsible CAs, and
other less common causes. While it would be unreasonable to expect an average person to understand cryptographic strength and entropy concepts, communication of relative toxicity and volatility of
chemicals has proven relatively feasible and is arguably no more complex. The identification of repeated identical warnings, many incorrect and all lacking nuance, combined with the distribution of causes
and types of incidents, argues for more customization in warnings.
The prevalence of software errors may be a result of our primary data source. Alternatively, we searched widely for incident reports and found that the incidents reported elsewhere were contained
in Bugzilla. Thus, the argument for completeness must be balanced
against an argument for bias.
Based on our combined findings, there is a need for improvement
in PKI at the moment, and this need will only increase as the interaction space for warnings and indicators decreases with IoT and embedded systems. No single party can improve the PKI’s resilience and
security independently. Interdisciplinary research and cross-industry
collaboration are needed.
Acknowledgments
We would like to acknowledge graduate researchers Jacob Abbott
and Shakthidhar Gopavaram and undergraduate researchers Andrew
Kim and HyeonJung “Julie” Lee for assistance with qualitative coding. We would also like to thank undergraduate researchers Jack
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
gorithm, which is not accepted by the BR (see BR Section 6.1.5).7
However, the BR recognizes that conflicts like these inherently occur; so, in BR Section 9.16.3, they address the discrepancies between
the governance requirements of the BR and a country/jurisdiction
regulation. Yet, there is no indicator or guidance for the individual
operator either seeking a certificate or relying upon one. Second, intermediate and subordinate CAs may be under the direct governance
of a parent root CA. In these cases, CAs in subordinate positions of
the CA chain-of-trust must operate under root CAs’ guidelines and
follow any stipulated requirement. Again, there are no clear indicators of this.
Our study of incidents of PKI identified both international business intrigue and blithe malpractice. The tale of backdated certificates by WoSign in 2016 after the rejection of the SHA-1 algorithm
reads like fiction, particularly as Mozilla discovered that WoSign (a
Chinese company) had purchased StartCom (an Israeli company) in
April 2016 through two Hong Kong subsidiaries, and in 2016 this
was not known by StartCom customers [32]. In terms of mundane
quotidian malpractice, Symantec was forced to sell the PKI business
to DigiCert after the steady misissuance of SSL certificates [33, 34].
Yet even after embracing an ethos of noncompliance, it was still thriving in the market because it was too large a component of the infrastructure to remove completely.
Finally, it is interesting that no experts felt web pages without
certificates (not using HTTPS) were a major problem. For example,
Google Chrome marked “Not Secure” http-accessed web pages and
the growth in free certificates usage as the ones provided by “Let’s
Encrypt.” Recent research has shown that almost 60% of the most
visited sites automatically redirect to https (if not accessed using the
secure protocol in the first place) [65]. That figure has increased from
∼7% near the end of 2015.
A holistic analysis of web-based public key infrastructure failures
Ruocco, Joshua Johnson, Zora Streiff, and Trevor Cutshall for assistance with the metadata collection of PKI incident reports.
Supplementary data
Supplementary data available at Cybersecurity Journal online.
Funding
Conflict of interest statement. None declared.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Braun J, Volk F, Classen J. et al. CA trust management for the Web PKI.
J Comput Secur 2014;22:913–59.
Arends R, Austein R, Larson M. et al. Resource Records for the DNS
Security Extensions. RFC Editor. 2005. https://doi.org/10.17487/RFC40
34.
Hoffman P, McManus P. DNS Queries over HTTPS (DoH). RFC Editor.
2018. https://doi.org/10.17487/RFC8484.
Bush R, Austein R. The Resource Public Key Infrastructure (RPKI) to
Router Protocol. RFC Editor. 2013. https://doi.org/10.17487/RFC6810.
Lepinski M, Sriram K (eds). BGPsec Protocol Specification. RFC Editor.
2017. https://doi.org/10.17487/RFC8205.
Reeder RW, Felt AP, Consolvo S. et al. An experience sampling study of
user reactions to browser warnings in the field. In: Conference on Human
Factors in Computing Systems – Proceedings, 2018-April. 2018. https:
//doi.org/10.1145/3173574.3174086.
Camp J, Henry R, Kohno T. et al. Toward a secure internet of things:
directions for research. IEEE Secur Priv 2020;18:28–37.
Abadi M, Birrell A, Mironov I. et al. Global authentication in an untrustworthy world. In: Proceedings of the 14th USENIX Conference on Hot
Topics in Operating Systems. 2013.
Ellison C, Bruce S. Ten risks of PKI: What you’re not being told about
public key infrastructure. Comput Secur J (2000);16(1):1–7.
Fox B, LaMacchia B. Certificate revocation: mechanics and meaning. Lect
Notes Comput Sci 1998;1465:158–164.
Camp LJ, Nissenbaum H, McGrath C. Trust: a collision of paradigms.
In: International Conference on Financial Cryptography. 2001, 91–105.
https://doi.org/10.1007/3-540-46088-8_10.
Ferreira A, Giustolisi R, Huynen JL. et al. Studies in socio-technical security analysis: authentication of identities with TLS certificates. Proceedings – 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications. TrustCom 2013, 2013, 1553–8.
https://doi.org/10.1109/TRUSTCOM.2013.190.
Langley A. ImperialViolet: Apple’s SSL/TLS bug. 2014. https://www.im
perialviolet.org/2014/02/22/applebug.html.
Synopsys Editorial Team. Understanding the Apple ‘goto fail;’ Vulnerability | Synopsys. 2014. https://www.synopsys.com/blogs/software-securi
ty/understanding-apple-goto-fail-vulnerability-2/.
Wheeler DA. The Apple goto fail Vulnerability: Lessons Learned. 2014.
https://dwheeler.com/essays/apple-goto-fail.html.
Bugzilla. 1619047: Let’s Encrypt: CAA Rechecking Bug. 2020. https://bu
gzilla.mozilla.org/show_bug.cgi?id=1619047.
17. Microsoft. CVE-2020-0601 – Security Update Guide – Microsoft – Windows CryptoAPI Spoofing Vulnerability. 2020. https://msrc.microsoft.c
om/update-guide/en-US/vulnerability/CVE-2020-0601.
18. Heninger N, Durumeric Z, Wustrow E. et al. Mining Your Ps
and Qs: Detection of widespread weak keys in network devices. In: 21st Usenix Security Symposium. Bellevue, WA, 2012.
https://www.cs.bham.ac.uk/∼garciaf/publications/Gone_in_360_seconds
_Hijacking_with_Hitag2_poster_2012.pdf.
19. Fillinger M, Stevens M. Reverse-engineering of the cryptanalytic attack
used in the flame super-malware. Lect Notes Comput Sci 2015;9453:
586–611.
20. sKyWIper Analysis Team. sKyWIper (a.k.a. Flame a.k.a. Flamer): a complex malware for targeted attacks. Laboratory of Cryptography and System Security (CrySyS Lab). 2021. https://www.crysys.hu/publications/fil
es/skywiper.pdf.
21. Stevens M, Sotirov A, Appelbaum J. et al. Short chosen-prefix collisions
for MD5 and the creation of a rogue CA certificate. Lect Notes Comput
Sci 2009;5677:55–69.
22. Brubaker C, Jana S, Ray B. et al. Using frankencerts for automated adversarial testing of certificate validation in SSL/TLS implementations. In:
2014 IEEE Symposium on Security and Privacy. 2014.
23. Chen Y, Su Z. Guided differential testing of certificate validation in
SSL/TLS implementations. In: 2015 10th Joint Meeting of the European
Software Engineering Conference and the ACM SIGSOFT Symposium on
the Foundations of Software Engineering, ESEC/FSE 2015 – Proceedings.
2015, 793–804. https://doi.org/10.1145/2786805.2786835.
24. Dong Z, Kane K, Camp LJ. Detection of rogue certificates from trusted
certificate authorities using deep neural networks. ACM Trans Priv Secur
(TOPS) 2016;19:31.
25. Kaloper-Meršinjak D, Mehnert H, Madhavapeddy A. et al. Not-quiteso-broken TLS: lessons in re-engineering a security protocol specification
and implementation. In: 24th USENIX Security Symposium (USENIX
Security 15). 2015, 223–38. https://www.usenix.org/conference/usenixse
curity15/technical-sessions/presentation/nan.
26. Acar Y, Backes M, Fahl S. et al. You get where you’re looking for:
the impact of information sources on code security. In: Proceedings
– 2016 IEEE Symposium on Security and Privacy, SP 2016. 2016,
289–305.
27. Krombholz K, Mayer W, Schmiedecker M. et al. “I Have No
Idea What I’m Doing” – on the usability of deploying HTTPS. In:
26th USENIX Security Symposium (USENIX Security 17). 2017,
1339–56. https://www.usenix.org/conference/usenixsecurity17/technical
-sessions/presentation/krombholz.
28. Tiefenau C, von Zezschwitz E, Häring M. et al. A usability evaluation of
Let’s Encrypt and Certbot: usable security done right. In: Proceedings of
the 2019 ACM SIGSAC Conference on Computer and Communications
Security. 2019. https://doi.org/10.1145/3319535.
29. Rajivan P, Moriano P, Kelley T. et al. Factors in an end user security
expertise instrument. Inf Comput Secur 2017;25:190–205.
30. Park D. Social life of PKI: sociotechnical development of Korean publickey infrastructure. IEEE Ann Hist Comput 2015;37:59–71.
31. Chen S, Kelley T, Dong Z. et al. The Effects of HeartBleed on Certificate
Change. Poster Presented at ACSAC(Los Angeles, CA) 7 –11 December
2015. 2015. http://www.ljean.com/files/meh.pdf.
32. Gervase M. WoSign and StartCom – Google Docs. Mozilla. 2016.
https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6
szuSB4sMYUcVrR8vQ/edit.
33. Google Groups. Misissued/Suspicious Symantec Certificates. 2017. https:
//groups.google.com/g/mozilla.dev.security.policy/c/fyJ3EK2YOP8.
34. O’Brien D, Sleevi R, Stark E, Chrome security team. Google Online Security Blog: Distrust of the Symantec PKI: Immediate action needed by
site operators. 2018. https://security.googleblog.com/2018/03/distrust-o
f-symantec-pki-immediate.html.
35. O’Brien D, Sleevi R, Whalley A, Chrome Security team. Google Online
Security Blog: Chrome’s Plan to Distrust Symantec Certificates. Google
Security Blog. 2017. https://security.googleblog.com/2017/09/chromes-pl
an-to-distrust-symantec.html.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
This research was supported in part by the National Science Foundation
awards CNS 1565375 and CNS 1814518, the National Security Agency
#H8230-19-1-0310, Cisco Research Support, Google Research, the Comcast
Innovation Fund, and the Indiana University Ostrom Workshop Fellowship.
Any opinions, findings, and conclusions, or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views of
the US Government, the National Science Foundation, the National Security
Agency, Cisco, Comcast, Google and Indiana University Ostrom Workshop.
13
14
58. Naiakshina A, Danilova A, Tiefenau C. et al. Why do developers get password storage wrong? A qualitative usability study. In: Proceedings of the
2017 ACM SIGSAC Conference on Computer and Communications Security. 2017. https://doi.org/10.1145/3133956.
59. Kelley T, Bertenthal BI. Attention and past behavior, not security knowledge, modulate users’ decisions to login to insecure websites. Inf Comput
Secur 2016;24:164–76.
60. Krombholz K, Busse K, Pfeffer K. et al. “If HTTPS Were Secure, I Wouldn’t
Need 2FA”- End User and Administrator Mental Models of HTTPS. In
2019 IEEE Symposium on Security and Privacy (SP). IEEE 2019;246–63.
61. Anti Phishing Working Group (APWG). Phishing activity trends report
1st quarter 2021. APWG, 2021. https://docs.apwg.org/reports/apwg_tre
nds_report_q1_2021.pdf.
62. Front Matter. Signposts in Cyberspace: tHe Domain Name System and Internet Navigation. Washington, DC: The National Academies Press, 2005,
1–391. https://doi.org/10.17226/11258.
63. Blythe J, Camp J, Garg V. Targeted risk communication for computer
security. In: International Conference on Intelligent User Interfaces, Proceedings IUI. 2011, 295–8. https://doi.org/10.1145/1943403.1943449.
64. Scheffler TD. Engineering security. In: Military Engineer, Vol. 108. Book
Draft. 2016. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.4
15.5998.
65. Helme S. Alexa Top 1 Million Analysis – February 2019. 2019. https:
//scotthelme.co.uk/alexa-top-1-million-analysis-february-2019/.
Appendix 1: Interview protocol
The interviews combined questions from two research projects, setting standards in the IoT and perceptions of resilience in PKI. In this paper, we focus on
perceptions of resilience of PKI.
Q1. How would you define or differentiate the IoT?
Q2. When an IoT device connects from inside to outside of the home what
kind of trust information is needed?
Q3. How do you trust a website now. Do you think the current web trust
model of public key certificates will work in IoT?
Q4. Have you ever had a false negative, where you knew a site was valid
but received a warning when trying to connect? In what context can
you describe that experience? Do you know what failed?
Q5. Do you think you have ever had a false positive, where a site was malicious but there was no warning? Can you describe the context? Do
you know what failed?
Q6. If the model for CAs and trust as built on the web (X.509) models what
should we learn and do differently in the IoT?
Q7. Can nontechnical people know their devices are trustworthy? If so,
how?
Q8. Based on what we have learned from malicious activities on the Web,
what are the requirements that you need as a stakeholders and a (technologist or a user) to certify trustworthiness of devices and connections?
Q9. How would this protect you and others by preventing malicious activities?
Q10. How can devices be updated when the manufacturer has gone out of
business or no longer supports the device?
Q11. Is this a case for exceptional access?
Q12. What do you see as the role of domain names in the IoT? What have
we learned and what should be change?
Q13. What are the roles of the marketplace, technologists, and policy makers
in resolving the failure models we discussed?
Q14. If you could tell a policy maker one thing, what would it be?
Q15. If you could communicate one thing to users about trust in IoT, what
would it be?
Q16. Do you have any other comments for usability scholars or technologists about the need for action?
Q17. What questions have I failed to ask about trust in IoT?
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
36. SSLShopper. Symantec sells its CA business to DigiCert. SSL Certificate
Comparison and Reviews. 2017. https://www.sslshopper.com/symantecsells-ca-business-to-digicert.html.
37. Delignat-Lavaud A, Abadi M ´, Birrell A. et al. Web PKI: closing the gap
between guidelines and practices – NDSS Symposium. Proceedings of the
ISOC, Network and Distributed System Security Symposium (NDSS ’14).
2014. https://www.ndss-symposium.org/ndss2014/programme/web-pkiclosing-gap-between-guidelines-and-practices/.
38. Kumar D, Wang Z, Hyder M. et al. Tracking certificate misissuance in the
wild. In: Proceedings – IEEE Symposium on Security and Privacy, 2018May, 2018, 785–98. https://doi.org/10.1109/SP.2018.00015.
39. Gasser O, Hof B, Helm M. et al. In log we trust: revealing poor security
practices with certificate transparency logs and internet measurements.
Lect Notes Comput Sci 2018;10771:173–85.
40. Roosa SB, Schultze S. Trust darknet: control and compromise in the internet’s certificate authority model. IEEE Internet Comput 2013;17:18–25.
41. Camp LJ. Identity the civic scenario. In: Proceedings of the 2004 Annual
National Conference on Digital Government Research. 2004. https://dl.a
cm.org/doi/10.5555/1124191.1124331.
42. Camp LJ, Bowman W, Friedman A. Voting, vote capture & vote counting
symposium. In: Proceedings of the 2005 National Conference on Digital
Government Research. 2005, 198–9. https://dl.acm.org/doi/10.5555/106
5226.1065282.
43. Camp LJ, Cranor L, Feamster N. et al. Data for Cybersecurity Research:
Process and “Wish List” – Reports & Papers – CERIAS: Purdue University. Purdue University. 2009. http://www.cc.gatech.edu/fac/feamster/pap
ers/data-wishlist.pdf.
44. Bernard HR, Harvey R, Ryan GW. et al. Analyzing Qualitative
Data: Systematic Approaches. Thousand Oaks, CA: SAGE Publications,
2016.
45. Electronic Frontier Foundation. HTTPS Everywhere FAQ | Electronic Frontier Foundation. https://www.eff.org/https-everywhere/faq
(21 November 2021, date last accessed).
46. Lear E, Rose S. SBOM Extension for MUD. IETF. 2020. https://datatrac
ker.ietf.org/doc/html/draft-lear-opsawg-mud-sbom-00.
47. Almishari M, de Cristofaro E, el Defrawy K. et al. Harvesting SSL certificate data to identify web-fraud. Int J Netw Secur 2009;14:324–338.
48. Dacosta I, Ahamad M, Traynor P. Trust no one else: detecting MITM
attacks against SSL/TLS without third-parties. Lect Notes Comput Sci
2012;7459:199–216. https://doi.org/10.1007/978-3-642-33167-1_12.
49. Huang LS, Rice A, Ellingsen E. et al. Analyzing forged SSL certificates
in the wild. In: Proceedings – IEEE Symposium on Security and Privacy.
2014, 83–97. https://doi.org/10.1109/SP.2014.13.
50. Perl H, Fahl S, Smith M. You won’t be needing these any more: on
removing unused certificates from trust stores. Lect Notes Comput Sci
2014;8437:307–15.
51. Gustafsson J, Overier G, Arlitt M. et al. A first look at the CT landscape: Certificate transparency logs in practice. Lect Notes Comput Sci
2017;10176:87–99.
52. Lear E, Droms R, Romascanu D. Manufacturer Usage Description Specification. RFC Editor. 2019. https://doi.org/10.17487/RFC8520.
53. European Union Agency. Certificate Authorities: The Weak Link of Internet Security. 2016. https://www.enisa.europa.eu/publications/info-notes/
certificate-authorities-the-weak-link-of-%0A internet-security.
54. Gomes F. Security Alert: Fraudulent Digital Certificates | SANS Institute.
SANS. 2001. https://www.sans.org/white-papers/679/.
55. Comodo. Comodo Report of Incident: Comodo detected and thwarted
an intrusion on 26-MAR-2011. Comodo. 2011. https://www.comodo.c
om/Comodo-Fraud-Incident-2011-03-23.html.
56. Cimpanu C. Microsoft takes control of 17 domains used by West
African BEC gang – The Record by Recorded Future. 2021.
https://therecord.media/microsoft-takes-control-of-17-domains-us
ed-by-west-african-bec-gang/.
57. Bard Gv. The vulnerability of SSL to chosen plaintext attack. IACR Cryptol. ePrint Arch 2004;2004:111.
Hadan et al.
15
A holistic analysis of web-based public key infrastructure failures
Appendix 2: Codebooks of interview about
experts’ perceptions of PKI failures
Table A1: Interview codebooks.
TPRC conference interview codebook
RWC conference interview codebook
Q1
electronic devices, connection/communication, software/application,
hardware, example/anecdote, public, privacy, data collection, smart
home, infrastructure, ambiguity, nonsense/dismissal, ubiquity/broad
scope, Internet of things
level of trust, unsure/don’t know, level of consequences, seek advice,
identity, decentralized, centralized, limited access, application
component, data handling, context-dependent, data sharing, data
collection
trust, privacy, untrusted CA, scope down, stakeholders, untrusted, failure,
too heavyweight, key management, transparency, secure data
transmission, hacks, uncertain, not just IoT
need more time to think, ambiguity, agreed with solutions, audit CAs,
corrupted CAs, self-signed certificates, invalid facts, comprehensive, no
certificates, expired certificate
idiot/lazy, expired certificates, ignore warning, risk-averse, frequent,
maintenance error, valid certificate, risk calculation, familiarity, hard to
determine
not detectable, CAs, untrustworthy network, risk-averse, unknowable,
frequency, context-dependent, attack, yes, no
identity validation, responsible/liability, data control, awareness,
user-centered, regulations, undisclosed collection, strengthen trust
model, confusion (trust model), transparency, privacy, scale, hidden
risks, sensitive info, best practice, yes, no, don’t know
standard/regulation, data collection/privacy, user can’t know, user
shouldn’t know, third party brand trust, user education/training,
security
don’t know, trust environment, DNS, no requirements, some
requirements, regulations/standards, audit
discard/throw away, recovery/repair/isolate, regulations, incentives,
responsibility/liability, white-hat hacking
electronic devices, connection/communication
over Internet, data handling, functions, broad
scope
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Q11
Q12
no backdoor, add backdoor, security risks, context-based, user rights,
repair escrow
DNS for people, IP address for machines, no role, new system, yes
Q13
market failures, lost trust, communication risk, cooperation,
regulation/policy, self-regulation, restrict enforcement, willingness to
pay, abuse of information/data
Q14
identity validation, privacy, consumer education, nontechnical solution,
developer education, design modification
privacy, externalities, vulnerability, usability, standards/regulations, risk
awareness, risk acceptance
key management, marketplace, consumer protection, human centered
design, tool management, third party control
definitions, precision, open ended, examples, clarification, too technical
Q15
Q16
Q17
authorization manufacturer, data handling,
certificate, policy, trustworthy, network,
isolation
credential, CAs, relationship, verify, different
solution, trustworthiness
insecure, expired certificates, algorithm issue,
curiosity, server configuration
informal CA, insecure page, accident
(unexpected), typo, no single solution, certified
website, PKI is not a solution
secure out of the box, value return, usability,
trust model, user control
manufacturer responsibility, indicator, trust
relationship, policies, secure out of the box,
self-certificate, government body
indication, protocols, no answer, trustworthy CA,
social problem, equivalent, some requirements,
unknown requirements
authority, validation, device dependent,
trustworthy, no answer, didn’t answer
update system (building prior), “brick” device,
open-source, escrow code, legal regulations,
warranty, government or organizations, device
dependent, paradox, attack resistance
no, law enforcement, context-dependent, no
answer
no trusted, same as in internet, disclosure,
identity of device creator, risk shift
Market failure, combination of technologies,
generation gap, uncertain roles, regulation,
early adopters, lack of attention, uneven
involving speed
regulation, indicator, escrow, consumer right,
privacy protection, security
trade-offs, user accommodation, buyer beware,
generation gap, isolation
usability workflow, user accommodation,
resilient against failure, no comment
goal, environment-dependent, no answer
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Questions
16
Hadan et al.
Appendix 3: Incident type codes classification
Table A2: Code classification: incident types.
Description
Example
Noncompliant info in
other fields
Fields in certificates (not OCSP or CRL) are not compliant to
Baseline Requirements (BRs). This option is selected when the
incident is not associated with OCSP responder or CRL, or when
the bug report only has a general description of the noncompliant
issue with no details.
The OCSP responder or CRL is not compliant to BR. BR page 5
specify the requirements for OCSP responder
SAN or CN field was not included8 ;
invalid date format9
Non-BR-compliant or
problematic
responder or CRL
Audit report failures
Serial number failures
Undisclosed SubCA
513\1024 bits key
Possible rogue
certificates
Insecure hashing
(SHA-1/MD5)
CAA misissuance
Verified rogue certificate
CA/RA/SubCA/reseller
hacked
Other
8
Having errors, delay, or completely missing audit reports. BR page
52 details the requirements for “compliance audit.”
Certificates have repetitive or inappropriate serial numbers. BR
page 5 details the requirements for serial number. Any certificates
issued after 9/30/2016 are required to have a serial number
containing at least 64 bits generated from a Cryptographically
Secure Pseudorandom Number Generator (CSPRNG).
Mozilla requires all its participating root CAs fully disclose their
hierarchy. This category includes the undisclosed SubCAs
BR requires certificates issued after 12/31/2010 use 2048 bits key.
This category includes certificates issued/renewed after 2010 with
513/1024 bits key.
Incidents possibly involve the issuance of rogue certificates
Certificate with SHA-1/MD5 hashing algorithm
OCSP responder returns “good” status
for unissued certificates10 ; OCSP
responders do not support GET
method11
lack of appropriate audit12 ; insufficient
audit statements13
serial number less than 64-bit entropy14
undisclosed intermediates15
1024 bits key16 ; 512 bits key17
DV certificate for possible fraudulent
use18
For example19
CAA resource record allows a DNS domain name holder to specify
CAs to issue certificates to that domain. The DNS CAA record is
specified by20 . This category includes incidents where CAs failed
to follow the CAA rules.
This describes the incidents where rogue certificates are certainly
involved.
CA, RA, SubCA, or resellers are hacked
CA issued a certificate for a domain,
which it should not have21 ; CAA tags
that were not lowercase being
ignored during CAA validation22
Certificates issued for MITM attempts23
all other less frequent incidents such as backdating SHA-1
certificates, delayed certificate revocations, certificates issued for
nonexistent domains
–
For example24
Bug 1017138: https://bugzilla.mozilla.org/show_bug.cgi?id=1017138. 9 Bug 1266501: https://bugzilla.mozilla.org/show_bug.cgi?id=1266501.10 Bug 1398233:
https://bugzilla.mozilla.org/show_bug.cgi?id=1398233.11 Bug 1426233: https://bugzilla.mozilla.org/show_bug.cgi?id=1426233.12 Bug 1145270: https://bugzilla.m
ozilla.org/show_bug.cgi?id=1145270.13 Bug 1425805: https://bugzilla.mozilla.org/show_bug.cgi?id=1425805.14 Bug 1390998: https://bugzilla.mozilla.org/show_b
ug.cgi?id=1390998, and Bug 1397951: https://bugzilla.mozilla.org/show_bug.cgi?id=1397951.15 Bug 1417771: https://bugzilla.mozilla.org/show_bug.cgi?id=14
17771, and Bug 1455132: https://bugzilla.mozilla.org/show_bug.cgi?id=1455132.16 Bug 1015772: https://bugzilla.mozilla.org/show_bug.cgi?id=1015772.17 Bug
1034949: https://bugzilla.mozilla.org/show_bug.cgi?id=1034949.18 Bug report: Symantec DV Certificate Issuance System Improperly Handled Domain Email Address Special Characters. See https://support.symantec.com/en_US/article.SYMSA1345.html.19 Bug 1313872: https://bugzilla.mozilla.org/show_bug.cgi?id=1313
872, and Bug 1315019: https://bugzilla.mozilla.org/show_bug.cgi?id=1315019.20 RFC 6844: DNS Certification Authority Authorization (CAA) Resource Record.
https://datatracker.ietf.org/doc/html/rfc6844.21 Bug 1409766: https://bugzilla.mozilla.org/show_bug.cgi?id=1409766.22 Bug 1462735: https://bugzilla.mozilla.org
/show_bug.cgi?id=1462735.23 Bug report: Maintaining digital certificate security. See https://security.googleblog.com/2015/03/maintaining-digital-certificate-secu
rity.html.24 Bug report: Maintaining digital certificate security. See https://security.googleblog.com/2014/07/maintaining-digital-certificate-security.html.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Incident types (Codes)
17
A holistic analysis of web-based public key infrastructure failures
Appendix 4: Incident cause codes classification
Table A3: Code classification: causes.
Description
Example
Software bugs
The cause is a software error or flaw that generated an incident in
the PKI ecosystem. The bugs may produce faulty certificates,
problems in OCSP responders, or erroneous check in a request
There is a lack of awareness or failed to comply with updates of the
BR or the Root Program Requirements, or misunderstood the
concepts or regulations behind these requirements.
CA is aware of BR or Root Program Requirements but places its
own business strategies over compliance
OCSP signed by wrong certificate
because the MXR query has error
codes25
misunderstood the BR, did not include
SAN field, RFC 612526
Requirements unknown
by CA
CA business decision
Single Certificate human
error
Operational error
Nonoptimal request
check
Improper security
controls
Change in Baseline
Requirements
Infrastructure problem
Organizational
constrains
Other
No data
25
One specific unique employee makes an error in a process. The
error is caused by human mistakes in manual entry of data for
certificate requests, or forgetting steps in the setup of a new
intermediate CA.
The incidents are generated because of an internal faulty procedure
in the CA or a related entity. This does not include errors resulted
from a specific identifiable employee’s manual error, but includes
errors in the operational processes of the organization.
The checks for an applicant for a certificate were not performed
correctly, and can potentially enable the issuance of rogue
certificates, non-verified EV certificates, or simply improper
digital certificates. This does not include includes caused by
“software bugs” or “human error” but only certificate requests
that were not performed properly.
CAs or other entities were hacked, with the possible or actual
outcome of rogue certificates being generated and used in the
wild as a result
a sudden change in the Baseline Requirements made the CAs change
their certificate issuance. Here, the CAs were not compliant until
they updated.
problems in the hardware or technical components that support the
business of the CA
incidents caused by constraints in the environments where the CAs
operated, such as national legal requirements. Some countries
require government agencies domains to have certificates
provided by a national public CA. This CA is obligated to follow
governmental requirements (e.g. ,using encryption algorithms
that are not compliant to BR)
all other less frequent causes and incidents that were not closed at
the time of writing this work
here, we include incidents that do not have detailed information
about the causes. In this case, the detailed information was
written in an attachment which became unavailable after incident
closed.
backdating SHA-1 certificates in order
to evade its prohibition, charging for
the revocation of certificates, selling
certificates for MITM attempts.
RA administrator manually selected an
incorrect EV policy27
mistakes in audit reports, not disclosing
subCAs, and delayed revocation of
compromised certificates
no technical control about the
DNSName input field in the request
form28
ISP KPN hacked29
no enough time to make certificate
replacement30 ; certificates approved
before BR change but issued after BR
change due to delayed payment31
unavailable servers, defective networks
Spain law requirements resulted in
non-BR-compliant certificates32
misconfigured software, misissuance not
in the sample provided to auditors
–
Bug 1007892: https://bugzilla.mozilla.org/show_bug.cgi?id=1007892.26 Bug 1261919: https://bugzilla.mozilla.org/show_bug.cgi?id=1261919.27 Bug 1266230:
https://bugzilla.mozilla.org/show_bug.cgi?id=1266230.28 Bug 1390977: https://bugzilla.mozilla.org/show_bug.cgi?id=1390977.29 Dutch ISP KPN hacked, credentials and personal information leaked. See https://nakedsecurity.sophos.com/2012/02/11/dutch-isp-kpn-hacked-credentials-and-personal-information-leaked
/.30 Bug 1516545: https://bugzilla.mozilla.org/show_bug.cgi?id=1516545.31 Bug 1449371: https://bugzilla.mozilla.org/show_bug.cgi?id=1449371.32 Bug 1267049:
https://bugzilla.mozilla.org/show_bug.cgi?id=1267049.
Downloaded from https://academic.oup.com/cybersecurity/article/7/1/tyab025/6470936 by guest on 17 October 2023
Incident causes (Codes)