1
1.1
一级标题
二级标题
1.1.1 三级标题
1.1.1.1
四级标题
五级标题
1.1.1.1.1
1.1.1.1.1.1
六级标题
注意: 这个必须放到正在编写的文档的头部。一个文档只能根据自己文档中的标题进行
格式化才能得到正确的格式。
2
背景知识
2.1
缩略语
2.2
基本概念
2.2.1 Security baseline
The security baseline of an evaluated network product is a set of security requirements and
environmental assumptions defining its capacity to resist a given attack potential.
This resistance to a given attack potential relies on:
➢ Attacker model and attacker potential agreed to be relevant for a given network product
class.
➢ The completeness and correct implementation of security requirements and operational
environment assumptions which limit the capacity of this attacker to threaten given
assets:
 Security requirements can be more demanding in some network elements, e.g.
exposed nodes will have to implement hardening requirements which will not
necessarily be needed in elements less exposed.
 Vulnerability assessment will be performed with more depth whenever the element
is expected to resist a stronger attacker.
【3GPP TR 33.916 V15.1.0】
2.2.2 V&V
2.2.2.1
定义
确认(validation)
通过检查并提供客观证据，证实明确提出的预期使用的特定需求得以满足。
【GBT25069-2010 信息安全术语, Ch2.3.80, P50】
验证(verification)
将某一活动或处理过程的输出与其相对应的安全需求或规范相比较，并证实该输出满足
需求或规范的过程。 例如：在安全操作系统的开发中，将某一安全策略的实现与相应的规
范相比较。
【GBT25069-2010 信息安全术语, Ch2.3.103, P53】
validation
confirmation, through the provision of objective evidence, that the requirements for a specific
intended use or application have been fulfilled。
Note 1 to entry:
A system is able to accomplish its intended use, goals
and objectives (i.e., meet
stakeholder requirements) in the intended operational
environment. The right system was built.
【[SOURCE: ISO 9000:2005, modified – Note 1 to entry has been added；ISO12207-2017_软
件生命周期流程, Ch3.1.71】
verification
confirmation, through the provision of objective evidence, that specified requirements have
been fulfilled
Note 1 to entry: Verification is a set of activities that compares a system or system element
against the required characteristics. This includes, but is not limited to specified requirements,
design, descriptions, and the system itself. The system was built right.
[SOURCE: ISO 9000:2015, modified, Note 1 to entry added;ISO12207-2017_软件生命周期流
程, Ch3.1.71]
independent verification and validation
[IEEE 610.12]
Verification and validation (V&V) performed by an organization that is technically,
managerially, and financially independent of the development organization.
2.2.2.2
区别
V&V
Verification ensures you built the system right
Validation ensures you built the right system
( INCOSE SE Handbook )
V&V in Requirements, Design, System
Requirement Verification: ensuring the requirement meets the rules and
characteristics defined for writing a good requirement. The focus is on the wording and structure
of the requirement.
Requirement Validation: confirmation that the requirements and requirement set is an
agreed to transformation that clearly communicates the stakeholder needs and expectations
in a language understood by the developers.
V & V System
System Verification: a process done after design and build or coding,
ensuring the designed and built or coded system meets its requirements. The focus is on the built
or coded system and how well it meets the agreed to requirement set that drove the design and
fabrication.
Methods used for system verification include: test, demonstration, inspection, or analysis.
“Did we build the thing right?”
System Validation: a process that occurs after system verification that confirms the
designed,
built,
and
verified system
meets
its intended
purpose
in
its
operational environment.
The focus is on the completed system and how well it meets stakeholder expectations
(needs) that were defined during the scope definition phase that should have occurred at the
beginning of the project.
“Did we build the right thing?”
【https://www.sesge.org/images/docs/Eventos/20190529/SESGE_2019-v7.pdf 】
2.2.2.3
所使用方法
验证可采用方法:
需求梳理，需求评审，设计评审，代码评审，单元测试，集成测试，系统测试，DoD 等。
确认可采用方法:
用户评审，用户划分优先级，原型，模拟，验收测试，用户反馈使用意见，试运行，sprint
review，验收标准定义等。
【白话 CMMI DEV V2.0, P43】
3
测试的定义与分类
更多详情见《安全术语&基本概念》
。
3.1
3GPP 测试分类&定义
注意: 【3GPP 33.805, 3.1
表 3-1
Definitions】也有类似的定义
3GPP 33.805 将评估分成以下类型(33.805, Ch5.2.8)
SCAS instantiation
Vendors Development process compliance
Evaluation
Security compliance testing
Basic Vulnerability Testing
Enhanced Vulnerability Analysis
3.1.1 安全测试(SCT)
Security Compliance Testing (SCT): Evaluation process step used to describe activities for
checking the compliance of a network product with applicable Security Assurance Specifications
(SCAS).
【 3GPP TR 33.916, Security Assurance Methodology (SCAS) for 3GPP network products
Release 16, V15.1.0】
3.1.2 基本漏洞测试(BVT)
Basic Vulnerability Testing (BVT): The process of running security tools against a network
product.
BVT is defined by the use of Free and Open Source Software (FOSS) and Commercial off-theshelf (COTS) security testing tools on the external interfaces of the network product.
【3GPP TR 33.916 V15.1.0】
3.1.3 高级漏洞测试(EVA)
Enhanced Vulnerability Testing (EVA): Evaluation process step described in Clause 7.2.5. This
activity takes the output of the earlier Security Compliance Testing (SCT) and Basic Vulnerability
Testing (BVT) into account.
【3GPP TR 33.916 V15.1.0】
3.1.4 渗透测试
A penetration test, colloquially known as a pen test, pentest or ethical hacking, is an
authorized simulated cyberattack on a computer system, performed to evaluate the security of the
system.【wiki】
3.1.5 渗透测试与漏洞扫描的区别
渗透测试通常是手动执行以发现新的漏洞。
3.1.6 测试不止是找 bug
很多人认为测试就是运行软件、检查功能点、找 bug、提交 bug，这严重低估了测试的
重要性。
随着软件系统越来越复杂，软件迭代越来越快，软件开发潜在 bug 的可能性也会越来
越高，
软件测试作为软件研发流程中保障质量的最后一关，也显得越来越不可或缺。找 bug、
提交 bug 只是软件测试工作中的一小段过程，测试脚本化、自动化都只是测试手段。
软件测试人员拿到需求书、需求分析书后，开始撰写测试案例、评审测试案例、执行测
试、找 bug、提 bug，再做回归测试，最后项目 / 变更上线，这一套测试做得很全面，但
项目 / 变更在生产上还是时不时出点问题。软件测试人员很苦恼，测试工作繁杂而又紧张，
一套测试流程走下来，身体累，如果再出生产问题，心更累。
软件质量贯穿于整个软件研发周期，单凭软件测试人员，不能百分之百保证软件质量。
而好的质量意味着交付更多的价值，不仅仅是没有缺陷那么简单。
软件测试人员从事测试活动是对软件产品的质量做出系统、全面、有效的反馈，开发人
员依据这些反馈信息，进一步完善软件产品，进而提升软件的整体质量。有价值质量反馈体
现为深入的、系统的和犀利的质量相关的见解。软件测试人员在日常开展功能测试、性能测
试、安全测试等测试活动，都是为了提供有价值的反馈，如产品缺陷、设计优化和用户体验
优化。
现在，很多测试人员和开发人员之间的协作在逐步增强，团队之间的界限随着时间的推
移变得越来越模糊。测试人员的职责范围也在扩展，逐渐参与到流程的其他方面，团队在不
同领域需要完成的任务和需要解决的挑战也在扩展。
一些测试人员需要指导开发人员进行自测，还有一些人需要在生产环境中进行质量监控，
并定义流程，确保质量保证成为整个团队工作的一部分，而不仅仅是在版本发布之前发现漏
洞和给漏洞打补丁。
软件测试最重要价值的是保障软件质量，只有以软件质量保障为顶级原则，测试工程师
才能真正做好、做深软件测试工作。
3.1.7 功能测试与安全测试之间的关系
In an effort to reduce redundant functional and security testing activities, it is recommended
that functional test plans include general security features testing (to the greatest extent
possible).
【NIST.SP.800-64r2_Security Considerations in the System Development Life Cycle, Ch3.2.3.6
Conduct Testing, P34】
4
4.1
测试相关标准&方法论
概述
主要标准&指导
➢ OWASP ASVS 应用安全验证标准，
➢ ISO 27002 14.2.8 System security testing
➢ 【3GPP 33.805 security compliance Testing，】这个已经是 2012/13 年的标准。
➢ NIST SP 800-115, Technical Guide to Information Security Testing and Assessment
➢ NIST.SP.800-53r5,
 CA(3.4 ASSESSMENT, AUTHORIZATION, AND MONITORING), CA-8 PENETRATION
TESTING, P112
 SA(3.17 SYSTEM AND SERVICES ACQUISITION), SA-11 DEVELOPER TESTING AND
EVALUATION, SA-11(5) PENETRATION TESTING, P297
➢ FS.16-NESAS-Development-and-Lifecycle-Security-Requirements-v1.0 [REQ-05] Security
➢
➢
➢
➢
4.2
Testing。
ISF_Standard_of Good_Practice_for_Information_Security_2020,
SD2.5 System
Testing。
ISF_Standard_of Good_Practice_for_Information_Security_2020, AS1.2 Security
Testing。
SAFECode_Fundamental_Practices_for_Secure_Software_Development_March_2018,
Testing and Validation, Perform Penetration Testing, P25
OpenSAMM 验证
普通测试
4.2.1 ISO 29119 软件测试
ISO/IEC/IEEE 29119-2:2013(en)，Software and systems engineering — Software testing
IEEE/ISO/IEC 29119-1-2013 - ISO/IEC/IEEE International Standard - Software and systems
engineering Software testing--Part 1:Concepts and definitions
IEEE/ISO/IEC 29119-2-2013 - ISO/IEC/IEEE International Standard - Software and systems
engineering Software testing--Part 2:Test processes
IEEE/ISO/IEC 29119-3-2013 - ISO/IEC/IEEE International Standard - Software and systems
engineering Software testing--Part 3: Test documentation
IEEE/ISO/IEC 29119-4-2015 - ISO/IEC/IEEE International Standard - Software and systems
engineering--Software testing--Part 4: Test techniques
ISO 29119
ISO/IEC/IEEE 29119-1:2013
Software and systems engineering — Software testing — Part 1: Concepts and definitions
90.92
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE DIS 29119-1
Software and systems engineering — Software testing — Part 1: Concepts and definitions
40.00
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE 29119-2:2013
Software and systems engineering — Software testing — Part 2: Test processes
90.92
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE FDIS 29119-2
Software and systems engineering — Software testing — Part 2: Test processes
50.00
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE 29119-3:2013
Software and systems engineering — Software testing — Part 3: Test documentation
90.92
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE FDIS 29119-3
Software and systems engineering — Software testing — Part 3: Test documentation
50.00
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE 29119-4:2015
Software and systems engineering — Software testing — Part 4: Test techniques
90.92
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE FDIS 29119-4
Software and systems engineering — Software testing — Part 4: Test techniques
50.00
ISO/IEC JTC 1/SC 7
ISO/IEC/IEEE 29119-5:2016
Software and systems engineering — Software testing — Part 5: Keyword-Driven Testing
60.60
ISO/IEC JTC 1/SC 7
ISO/IEC CD TR 29119-6
Software and systems engineering — Software testing — Part 6: Guidelines for the use of
ISO/IEC/IEEE 29119 in Agile projects
30.60
ISO/IEC JTC 1/SC 7
ISO/IEC TR 29119-11:2020
Software and systems engineering — Software testing — Part 11: Guidelines on the testing of
AI-based systems
4.2.2 ISO/IEC 9216(普通测试)
The International Standard ISO/IEC 9216 provides guidance for establishing quality in
software products. With respect to testing, this standard focuses on a quality model built around
functionality, reliability, and usability. Additional issues of efficiency, maintainability, and portability
are included in the quality model of the standard. With respect to security and testing, it is
important to remember the differences between quality and security. Quality is defined as fitness
for use, or conformance to requirements. Security is less cleanly defined, but can be defined by
requirements. One issue addressed by the standard is the human side of quality, where
requirements can shift over time, or be less clear than needed for proper addressing by the
development team. These are common issues in all projects, and the standard works to ensure a
common understanding of the goals and objectives of the projects as described by requirements.
This information is equally applicable to security concerns and requirements.
4.2.3 ISF SD2.5 系统测试
4.2.4 IEEE 测试标准
ANSI—IEEE Std 1008-1987(R2009)_Standard for Software Unit Testing
IEEE Std 1012-2016_Standard for System, Software, and Hardware Verification and Validation
IEEE Std 829-2008_Software and System Test Documentation
4.3
安全测试
安全测试
➢
➢
CC 安全功能组件&安全保障组件的测试
➢
ISECOM OSSTMM(开源安全测试方法论)，Open Source Security Testing Methodology
Manual; ISECOM; http://www.isecom.org/，https://www.isecom.org/OSSTMM.3.pdf。
OWASP 测试指南, ASVS 应用安全验证标准，Web Security Testing Guide。
 Web Security Testing Guide (WSTG)
 Mobile Security Testing Guide (MSTG)
 Firmware Security Testing Methodology
【3GPP 33.805 security compliance Testing，】这个已经是 2012/13 年的标准。
Common Attack Pattern Enumeration and Classification; MITRE; http://capec.mitre.org/
➢
➢
➢
NIST SP 800-115, Technical Guide to Information Security Testing and Assessment.
4.3.1 安全功能组件&安全保障组件的测试(应该就是 CC)
安全功能组件&安全保障组件功能测试(等同于 ISO/IEC 15408):
➢ GBT 18336.2-2015 信息技术 安全技术 信息技术安全评估准则 第 2 部分安全功能
组件。
➢ GBT 18336.3-2015 信息技术 安全技术 信息技术安全评估准则 第 3 部分安全保障
组件。
4.3.2 NIST 53 SA-11 开发者测试与评估(待录入)
【NIST.SP.800-53r5, SA-11 DEVELOPER TESTING AND EVALUATION, P295
4.3.3 NIST SP 800-115
NIST SP 800-115, Technical Guide to Information Security Testing and Assessment.
4.3.4 3GPP 33.805 安全合规测试
【3GPP 33.805 security compliance Testing，】这个已经是 2012/13 年的标准。
4.3.5 OWASP
OWAS 提供的测试标准:
➢ https://owasp.org/www-project-web-security-testing-guide/,
➢
https://github.com/OWASP/wstg。
https://owasp.org/www-project-mobile-security/(它只是一个框架，内部众多多子项
目，包括 mstg 项目。)
https://github.com/OWASP/owasp-mstg (The Mobile Security Testing Guide)
➢
➢
OWASP Top10、CWE/SANS Top25
OWASP ASVS (Application Security Verification Standard Project)
4.3.6
SAFECode Testing and Validation
4.3.7
ISF SOGP AS1.2 安全测试
4.3.8 OpenSAMM 验证
Verification focuses on the processes and activities related to how an organization checks and
tests artifacts produced throughout software development. This typically includes quality
assurance work such as testing, but it can also include other review and evaluation activities.
➢ Architecture Assessment：This practice focuses on validating the security and compliance
of the software and supporting infrastructure architecture.
➢ Requirements-driven Testing: This practice focuses on using both positive (control
verification) and negative (misuse/abuse testing) security tests based on requirements
(user stories).
➢ Security Testing： This practice focuses on the detection and resolution of basic security
issues through automation, allowing manual testing to focus on more complex attack
vectors.
4.3.9 OSSTMM
The Open Source Security Testing Methodology Manual (OSSTMM) is a peer reviewed system
describing security testing. OSSTMM provides a scientific methodology for assessing operational
security built upon analytical metrics. It is broken into five sections: data networks,
telecommunications, wireless, human, and physical security, as shown in Table 15-1. The purpose
of the OSSTMM is to create a system that can accurately characterize the security of an operational
system in a consistent and reliable fashion.
4.3.10 OISSG ISSAF(最新版为 2005 年发布的 V1.0)
Open Information Systems Security Group
The ISSAF standard (Information System Security Assessment Framework) contains an even
more structured and specialized approach to penetration testing than the previous standard. If
your organization’s unique situation requires an advanced methodology entirely personalized to
its context, then this manual should prove useful for the specialists in charge of your penetration
test.
These sets of standards enable a tester to meticulously plan and document every step of the
penetration testing procedure, from planning, assessment, to reporting and destroying artefacts.
This standard caters for all steps of the process. Pentesters who use a combination of different
tools find ISSAF especially crucial as they can tie each step to a particular tool.
The assessment section, which is more detailed, governs a considerable part of the procedure.
For each vulnerable area of your system, ISSAF offers some complementary information, various
vectors of attack, as well as possible results when a vulnerability is exploited. In some instances,
testers may also find information on tools that real attackers commonly use to target these areas.
All this information proves worthwhile to plan and carry out particularly advanced attack scenarios,
which guarantees a great return on investment for a company looking to secure their systems from
cyberattacks.
图 4-1
ISSAF 渗透测试过程(https://www.researchgate.net/figure/The-Phases-of-PenetrationTesting-ISSAF-Standard_fig2_325030351 )
4.3.10.1 参考文献
https://untrustednetwork.net/files/issaf0.2.1.pdf
https://ht.transparency.tools/FileServer/FileServer/whitepapers/issaf/issaf0.2.1A.pdf
https://www.securearc.com/wiki/index.php/ISSAF
最新版为 2005 年发布的 V1.0: https://sourceforge.net/projects/isstf/
4.4
渗透测试
行业标准渗透测试方法:
➢ Penetration
Testing
➢
Execution
Standard(PTES)
,
http://www.pentest-
standard.org/index.php/PTES_Technical_Guidelines，
PCI
DSS
关
于
渗
透
测
试
的
信
息
补
充
(https://www.pcisecuritystandards.org/documents/Penetration_Testing_Guidance_Ma
rch_2015.pdf )。
 PCI DSS Penetration Testing Guidance
 PCI DSS Penetration Testing Requirements
➢
FedRAMP 渗透测试指南。The Federal Risk and Authorization Management Program
(FedRAMP) is a US government-wide program that provides a standardized approach to
security assessment, authorization, and continuous monitoring for cloud products and
services.The General Services Administration (GSA) established the FedRAMP Program
Management Office (PMO) in June 2012.
https://www.fedramp.gov/assets/resources/documents/CSP_Penetration_Test_Guidan
ce.pdf
https://www.fedramp.gov/assets/resources/documents/CSP_Timeliness_and_Accuracy
_of_Testing_Requirements.pdf
此外，还有一个联邦信息安全评估框架，但是时 2000 年版，已经很老了。
Federal
Information
Technology
Security
Assessment
Framework
https://www.nist.gov/publications/federal-information-technology-security-
➢
，
assessment-framework。
NIST 53 CA-8(渗透测试)
4.4.1 渗透测试简介
A penetration test, colloquially known as a pen test, pentest or ethical hacking, is an
authorized simulated cyberattack on a computer system, performed to evaluate the security of the
system.[1][2] Not to be confused with a vulnerability assessment.[3] The test is performed to
identify both weaknesses (also referred to as vulnerabilities), including the potential for
unauthorized parties to gain access to the system's features and data,[4][5] as well as strengths,[6]
enabling a full risk assessment to be completed.
The process typically identifies the target systems and a particular goal, then reviews available
information and undertakes various means to attain that goal. A penetration test target may be a
white box (which provides background and system information) or black box (which provides only
basic or no information except the company name). A gray box penetration test is a combination
of the two (where limited knowledge of the target is shared with the auditor).[7] A penetration
test can help determine whether a system is vulnerable to attack if the defenses were sufficient,
and which defenses (if any) the test defeated.[8][6]
Security issues that the penetration test uncovers should be reported to the system owner.[9]
Penetration test reports may also assess potential impacts to the organization and suggest
countermeasures to reduce risk.[9]
The National Cyber Security Center describes penetration testing as the following: "A method
for gaining assurance in the security of an IT system by attempting to breach some or all of that
system's security, using the same tools and techniques as an adversary might." [10]
The goals of a penetration test vary depending on the type of approved activity for any given
engagement with the primary goal focused on finding vulnerabilities that could be exploited by a
nefarious actor and informing the client of those vulnerabilities along with recommended
mitigation strategies.[11]
Penetration tests are a component of a full security audit. For example, the Payment Card
Industry Data Security Standard requires penetration testing on a regular schedule, and after
system changes.[12]
Several standard frameworks and methodologies exist for conducting penetration tests. These
include the Open Source Security Testing Methodology Manual (OSSTMM), the Penetration Testing
Execution Standard (PTES), the NIST Special Publication 800-115, the Information System Security
Assessment Framework (ISSAF) and the OWASP Testing Guide.
Flaw hypothesis methodology is a systems analysis and penetration prediction technique
where a list of hypothesized flaws in a software system are compiled through analysis of the
specifications and documentation for the system. The list of hypothesized flaws is then prioritized
on the basis of the estimated probability that a flaw actually exists, and on the ease of exploiting
it to the extent of control or compromise. The prioritized list is used to direct the actual testing of
the system.
【https://en.wikipedia.org/wiki/Penetration_test】
4.4.2 PTES 解读
Penetration Testing Execution Standard (PTES) defines penetration testing as 7 phases.
Particularly, PTES Technical Guidelines give hands-on suggestions on testing procedures, and
recommendation for security testing tools.
➢ Pre-engagement Interactions
➢ Intelligence Gathering
➢ Threat Modeling
➢ Vulnerability Analysis
➢ Exploitation
➢
➢
Post Exploitation
Reporting
【http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines 】
4.4.3 参考文献
https://en.wikipedia.org/wiki/Penetration_test
对各个测试标准与方法论进行了解读
https://github.com/OWASP/wstg/blob/master/document/3The_OWASP_Testing_Framework/1-Penetration_Testing_Methodologies.md
https://owasp.org/www-project-web-security-testing-guide/v41/3The_OWASP_Testing_Framework/1-Penetration_Testing_Methodologies
https://www.vumetric.com/blog/top-penetration-testing-methodologies/
Selection of penetration testing methodologies: A comparison and evaluation and evaluation
https://ro.ecu.edu.au/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1
&article=1181&context=ism
4.5
BSIMM 安全测试(ST)与渗透测试(PT)的对比
BSIMM 10 中安全测试与渗透测试的对比:
安全测试:
➢ ST1.1 确保 QA 支持边缘/边界值条件测试。
➢ ST1.3 利用安全性要求和安全性功能开展测试。
➢ ST2.1 把黑盒安全性工具整合到 QA 流程中。
➢ ST2.4 与 QA 共享安全性检查结果。
➢ ST2.5 把安全性测试纳入到 QA 自动化中。
➢ ST2.6 开展专为应用 API 量身定制的模糊测试。
➢ ST3.3 利用风险分析结果开展测试。
➢ ST3.4 利用（代码）覆盖分析。
➢ ST3.5 开始构建并应用对抗性安全测试（滥用案例）
。
渗透测试(PT):
➢ PT1.1 聘请外部渗透测试者来寻找问题。
➢ PT1.2 把结果反馈至缺陷管理和缓解系统中。
➢ PT1.3 在内部使用渗透测试工具。
➢ PT2.2 为渗透测试者提供所有的可用信息。
➢ PT2.3 定期开展渗透测试，以了解应用覆盖情况
➢ PT3.1 聘请外部渗透测试者开展深度分析。
➢ PT3.2 由 SSG 定制渗透测试工具和脚本。
4.5.1 BSIMM 攻击模型(AM)
第1级
• AM1.2 制定数据分类方案并制作数据清单。
• AM1.3 发现潜在攻击者。
• AM1.5 收集并使用攻击情报。
第2级
• AM2.1 构建与潜在攻击者有关的攻击模式和滥用案例。
• AM2.2 创建与特定技术相关的攻击模式。
• AM2.5 创建并维护最危险的 N 种攻击列表。
• AM2.6 收集并发布攻击案例。
• AM2.7 建立内部论坛，以讨论各种攻击。
第3级
• AM3.1 拥有一支开发新攻击方法的科学团队。
• AM3.2 创建并使用自动化方法来模拟攻击者。
• AM3.3 监控并自动化资产创建工作。
4.6
法律法规要求
4.6.1 欧盟网络安全法
欧盟网络安全法，Article 52.7, URL: https://eur-lex.europa.eu/eli/reg/2019/881/oj
A European cybersecurity certificate that refers to assurance level ‘high’ shall provide
assurance that the ICT products, ICT services and ICT processes for which that certificate is issued
meet the corresponding security requirements, including security functionalities, and that they
have been evaluated at a level intended to minimise the risk of state-of-the-art cyberattacks carried
out by actors with significant skills and resources. The evaluation activities to be undertaken shall
include at least the following: a review to demonstrate the absence of publicly known
vulnerabilities; testing to demonstrate that the ICT products, ICT services or ICT processes correctly
implement the necessary security functionalities at the state of the art; and an assessment of their
resistance to skilled attackers, using penetration testing. Where any such evaluation activities are
not appropriate, substitute activities with equivalent effect shall be undertaken.
保证等级为“高”的欧洲网络安全证书应保证发出该证书的 ICT 产品，ICT 服务和 ICT 流
程满足相应的安全要求，包括安全功能，并且已对其进行了评估。 旨在最大程度地减少具
有重要技能和资源的参与者进行最新网络攻击的风险。 进行的评估活动至少应包括以下内
容：进行审查以证明没有公开的漏洞； 测试以证明 ICT 产品，ICT 服务或 ICT 流程在最新技
术水平上正确实现了必要的安全功能； 并使用渗透测试评估他们对熟练攻击者的抵抗力。
如果此类评估活动不合适，则应进行具有同等效力的替代活动。
4.6.2 EECC 要求
《ENISA Guidelines on Security Measures under the EECC》
4.6.2.1
安全测试
4.4.7.3 SO25: Network and information systems testing, P39
Establish and maintain policies for testing network and information systems35, particularly
when connecting to new networks or systems.
1
Security measures
Evidence
a) Test networks and information systems
i. Test reports of the network and information
before using them or connecting them to
systems, including tests after big changes or the
existing systems.
introduction of new systems.
a)在使用网络或信息系统或将其连接到现
有系统之前，对其进行测试。
2
b) Implement policy/procedures for testing
ii. Policy/procedures for testing networks and
network and information systems,
information systems, including when tests must
c) Implement tools for automated testing
be carried out, test plans, test cases, test report
b)实施测试网络和信息系统的政策/程序，
templates.
c)实施自动化测试工具
3
d) Review and update the
iii. List of test reports.
policy/procedures for testing, taking into
iv. Updated policy/procedures for testing
account changes and past incidents.
networks and information systems, review
d)考虑更改和过去的事件，检查并更新测
comments, and/or change log.
试的策略/过程。
4.6.2.2
安全评估
4.4.7.4 SO26: Security assessments, P39
Establish and maintain an appropriate policy for performing security assessments of network
and information systems.
1
Security measures
Evidence
a) Ensure critical systems undergo
security scans and security testing
regularly, particularly when new
systems are introduced and following
changes.
i. Reports from past security scans and security
tests.
i. 来自过去的安全扫描和安全测试的报告。
a)确保对关键系统进行定期的安全扫
描和安全测试，尤其是在引入新系统
并进行更改时。
2
b) Implement policy/procedures for
security assessments and security
testing.
b)实施安全评估和安全测试的政策/
程序。
3
c)Evaluate the effectiveness of
policy/procedures
for
security
assessments and security testing.
ii. Documented policy/procedures for security
assessments and security testing, including,
which assets, in what circumstances, the type
of security assessments and tests, frequency,
approved parties (internal or external),
confidentiality levels for assessment and test
results and the objectives security assessments
and tests.
ii.安全评估和安全测试的书面政策/程序，包
括哪些资产，在什么情况下，安全评估和测
试的类型，频率，批准方（内部或外部），评
估和测试结果的机密性以及目标安全评估
和测试。
iii. List of reports about security assessment
and security tests.
iv. Reports of follow up actions on assessment
d)Review
and
update
policy/procedures
for
security
assessments and security testing,
taking into account changes and past
incidents.
c)评估(Evaluate)用于安全评估和安全
测试的策略/过程的有效性。
d)在考虑更改和过去事件的情况下，
审 查 (Review) 和 更 新(update) 用 于安
全评估和安全测试的策略/过程。
5
and test results.
v. Up to date policy/procedures for security
assessments and security testing, review
comments, and/or change log.
iii.有关安全评估和安全测试的报告列表。
iv. 报告评估和测试结果的后续行动。
v. 用于安全评估和安全测试，查看评论和/
或更改日志的最新策略/过程。
常见的安全验证手段
常见安全验证手段有:
➢ 静态&动态缺陷分析&Fuzz 测试(特殊的动态缺陷分析)
➢ 人工渗透测试
 Testing for OWASP Top 10
 OWASP ASVS testing
➢ Code 审查
5.1
MS SDL 验证手段
图 5-1
微软的安全验证要求(https://download.microsoft.com/download/9/3/5/935520EC-
D9E2-413E-BEA7-0B865A79B18C/Introduction to the Microsoft Security Development Lifecycle
(SDL).ppsx )
5.1.1 实践 11:动态程序分析
SDL Practice 11: Dynamic Program Analysis
Run-time verification of software programs is necessary to ensure that a program’s
functionality works as designed. This verification task should specify tools that monitor application
behavior for memory corruption, user privilege issues, and other critical security problems. The
SDL process uses run-time tools like AppVerifier, along with other techniques such as fuzz testing,
to achieve desired levels of security test coverage.
SDL 练习 11：动态程序分析
必须对软件程序进行运行时验证，以确保程序的功能按设计工作。该验证任务应指定用
于监视应用程序行为的工具，以检查内存损坏，用户特权问题和其他严重的安全问题。 SDL
流程使用诸如 AppVerifier 之类的运行时工具以及诸如模糊测试之类的其他技术来实现所需
级别的安全测试覆盖率。
5.1.2 实践 12:模糊测试(动态分析的特殊形式)
SDL Practice 12: Fuzz Testing
Fuzz testing is a specialized form of dynamic analysis used to induce program failure by
deliberately introducing malformed or random data to an application. The fuzz testing strategy is
derived from the intended use of the application and the functional and design specifications for
the application. The security advisor may require additional fuzz tests or increases in the scope and
duration of fuzz testing.
模糊测试是动态分析的一种特殊形式，用于通过有意将畸形或随机数据引入应用程序来
导致程序失败。 模糊测试策略是从应用程序的预期用途以及应用程序的功能和设计规范得
出的。 安全顾问可能需要其他绒毛测试或增加绒毛测试的范围和持续时间。
5.1.3 威胁模型和攻击面审查
SDL Practice 13: Threat Model and Attack Surface Review
It is common for an application to deviate significantly from the functional and design
specifications created during the requirements and design phases of a software development
project. Therefore, it is critical to re-review threat models and attack surface measurement of a
given application when it is code complete. This review ensures that any design or implementation
changes to the system have been accounted for, and that any new attack vectors created as a result
of the changes have been reviewed and mitigated.
应用程序通常会偏离软件开发项目的需求和设计阶段中创建的功能和设计规范，这是很
常见的。 因此，在代码完成后，重新审查给定应用程序的威胁模型和攻击面度量至关重要。
该检查确保已考虑到系统的任何设计或实现更改，并且已检查和缓解了由于更改而创建的任
何新攻击媒介。
5.2
渗透测试
Goal is to simulate actual attacks against an application and measure how well the application
is able to withstand such attacks。
Attack the application as a malicious user by using:
➢ Manual techniques
➢ Attack tools
Penetration testing provides a security posture snapshot only at the specific time of testing。
5.3
代码审查(Code Review)
Security code review as a complementary method to other security assessment methods.
Manual review of the source code of an application to identify common code weaknesses
Automation tools to assist code reviewers are available
Pros:
➢ Very effective,
➢ Deeper analysis possible with fewer false-positives and false-negatives
➢ More context-aware analysis
Cons:
➢ Manual process
➢ but also very labor-intensive
➢ Dependent on skill and expertise of the reviewer
注意事项
➢ Should be performed for all high-priority code
➢ Multi-pass approach
 Specific vulnerabilities
 Different reviewers
➢ Focus efforts on higher priority code
➢ Establish clear objectives
 Check return values
 Check all pointers
 Validate all un-trusted data
5.4
渗透测试标准&规范
ISF_Standard_of Good_Practice_for_Information_Security_2020 ， AS1.2 Security Testing,
P338。
Penetration testing should follow a rigorous approach based on more recognised industry
methodologies, such as the:
– Open Source Security Testing Methodology Manual (OSSTMM)
– Open Web Application Security Project (OWASP) Testing Guide
– Penetration Testing Execution Standard (PTES).
Some security vendors offer a crowdsourced solution for penetration testing, harnessing the
skills of ethical hackers to provide organisations with results that usually take much longer to
produce (e.g. by using techniques such as brute force password attacks and social engineering).
渗透测试应采用基于更广为人知的行业方法的严格方法，例如：
–开源安全测试方法手册（OSSTMM）
–开放式 Web 应用程序安全项目（OWASP）测试指南
–渗透测试执行标准（PTES）
。
一些安全供应商提供了用于渗透测试的众包解决方案，利用道德黑客的技能为组织提供
通常需要更长的时间才能产生的结果（例如，通过使用暴力密码攻击和社会工程学等技术）。
6
安全测试工具
见相关专题。
6.1
验证工具分类
验证工具可以分成以下几种类型:
➢ 源代码合规检查工具。此类工具通常利用自动化静态分析工具发现代码中不符合安
全编程规范的内容。目前国际上比较主流的安全编程标准包括 CERT C/C++/Java、
MISRA C/C++等。源码合规检查，一般是轻量级的工具，很短时间内完成检测，可
以方便的集成在开发者的 IDE 当中或者在代码提交时自动被调用。例如针对 C/C++
语言的 pclint，针对 Java 语言的 checkstyle/findbugs/PMD。
➢ 缺陷分析工具。源码缺陷分析指利用自动化静态/动态分析工具发现源代码中的缓
冲区溢出、SQL 注入、跨站脚本等安全缺陷的过程。
 静态应用安全测试（SAST）
✓ 静态源码缺陷分析工具。
✓ 静态二进制缺陷分析工具。
 动态缺陷分析工具，即动态应用安全测试 (DAST)工具。
 模糊(fuzz)测试(特殊的动态测试)。
 Interactive Application Security Testing (IAST)。Gartner 在 2017 年的研究报告中
明确提倡用 Interactive Application Security Testing (IAST) 替换 SAST 和 DAST。
主要原因: SAST 因效率差、误报率高。DAST 无法爬取应用完整 URL；因重放
数据导致重复提交数据，使得自动化功能测试失败（产生了脏数据、功能异常
等问题）
。IAST 漏洞详情中会包括漏洞形成的应用内部数据流的详细传播过程。
➢ 源码溯源检测工具。此类工具是面向复杂的供应链现状提出的，主要是基于开源代
码库，自动化地检测软件中是否引用了开源代码模块，引用的开源代码模块是否存
在已知的安全漏洞，以及软件使用授权（License）问题。例如:
 使用黑鸭子软件成分分析，实现组件及其版本与配置的全流程追踪。
 免费的 OWASP Dependency Check 也不错。OWASP Dependency Check 两次入选
ThoughtWorks 技术雷达，它能自动完成第三方组件识别、漏洞数据库维护，以
及漏洞匹配、生成检查报告等一些列活动，并且可以集成到持续构建过程中。
6.1.1 推荐使用 IAST 安全工具
《Gartner 2017 研究报告：DevSecOps 应当做好的十件事》和《Gartner 2019 研究报告：
DevSecOps 应当做好的十二件事》两篇研究报告中都对成功实施 DevSecOps 进行了研究。两
篇报告一致认为实施 DevSecOps 的关键挑战和第一要素是：“安全测试工具和安全控制过程
能够很好的适应开发人员，而不是背道而驰”
。安全团队想要将安全测试或安全控制成功的
融入在 DevOps 中的前提是：不要试图改变程序员和测试人员的工作方法，也不要去增加他
们额外的工作负担。
随着 DevSecOps 被广泛接纳，Gartner 在 2017 年的研究报告中明确提倡用 Interactive
Application Security Testing (IAST) 替换 SAST 和 DAST，原文如下：考虑交互式应用安全测试
(IAST) 替代传统的静态应用安全测试（SAST）和动态应用安全测试 (DAST) 是可行的，我们
建议这么做。IAST 是在 DAST 基础上的一种改进形式，测试过程更加可视化，包括内部和外
部的。IAST 综合了 DAST 和 SAST 的优势，在确保测试代码覆盖率的基础上尽可能的做到
了减少误报。
处理安全漏洞
7
处理验证时发现的漏洞。
➢ Define a security bug bar
➢ Includes threats, such as (in order from most severe to least severe):
 Elevation of privilege
 Denial of service
 Targeted information disclosure
 Spoofing
 Tampering
7.1
Bug 栏(感觉与 CVSS 定义标准类似)
表 7-1
bug bar 范例(Microsoft SDL - Developer Starter Kit - COMPLETE\Secure Verification
Principles)
Severity Rating
Threat Conditions
SEV1
Elevation of Privilege: Remote anonymous user
SEV 2
Denial of Service: Remote anonymous user, permanent
Elevation of Privilege: Remote user
SEV 3
Denial of Service: Temporary
SEV 4
Information Disclosure: Untargeted
Defense in Depth
Vulnerabilities with security ramifications without any known
methods for exploit
In the first column the bug bar defines the severity of a security bug and the second column
describes the threat type and criteria required to be considered in that severity category. A severity
rating of 1 represents those bugs that have the highest potential for inflicting damage, whereas a
severity rating of 4 represents those bugs that have the lowest potential for inflicting damage.
The Defense in Depth row represents bugs with security ramifications without any known methods
for exploit.
When using a security bug bar, application development teams agree upon how a bug in each
severity category is to be addressed. For instance, a response to a severity 1 category bug might
be that is must be fixed immediately, whereas a severity 3 category bug might not need to be
addressed until the next major release of an application is scheduled.
8
代码测试
安全测试主要分为白盒代码安全审计，黑盒漏洞扫描，人工渗透测试。
8.1
测试方案
安全测试实践方案:
➢ 白盒静态应用安全测试技术(Static Application Security Testing，简称 SAST)技术。又可
分为:
 简单的代码编程规范检查。
 根据编译时信息生成语法树进行代码执行全路径的扫描(每一个 if/switch)是一
个分支。
➢ 黑盒动态应用安全测试技术(Dynamic Application Security Testing，简称 DAST)技术，
可以尝试使用漏洞扫描和 fuzz 扫描工具。
 安全自动化扫描平台
➢ 交互式(IAST)。
➢ 软件成分分析(Software Component Analysis，简称 SCA)技术等。
➢ 人工渗透测试。
静态代码分析的白盒测试，可以进行代码的语义和缺陷分析，主要针对研发人员。支持
主流 SDLC 方案的集成，需要分析的代码可以在夜间提交到构建服务器。
动态应用安全测试通常是黑盒测试，不需要软件源代码，针对测试和安全人员，在产品
完成后，进行协议级别的模糊测试，和版本迭代测试。
交互式介于白盒和黑盒测试之间，也可称之为灰盒测试，不需要源码，但是可以提供代
码级问题定位的精准测试结果。
静态代码扫描工具在代码提交时即可前进行扫描，并在静态代码层面上发现各种问题，
其中包括安全问题。
黑盒测试则可以在每个迭代完成时对版本进行扫描。
各项目可以根据人手以及风险状况决定是否进行人工渗透测试。
8.2
测试实践
在测试阶段，安全活动主要围绕安全报告验收、人工安全测试、漏洞验证开展。
要求业务方将安全设计 checklist 自检报告、静态代码扫描报告、web 漏洞扫描报告纳入
安全提测工单中，待安全测试人员进行检查并通过后进行人工安全测试，各项的安全标准可
参照：
表 8-1
安全测试标准(https://mp.weixin.qq.com/s/WO089RBiLuaHMzQ4yTMnUg )
开发阶段
安全活动
输出物
安全标准
设计阶段
安全设计自检
安全设计 checklist 自检报告
无不符合项
编码阶段
静态代码扫描
静态代码扫描报告
无设定的高危漏洞
测试阶段
Web 漏洞扫描
Web 漏洞扫描报告
无高中危漏洞
人工安全测试
TODO
无高中危漏洞
值得注意的是需要设置抽查和复测机制，防止业务方有意或无意的提供不符合实际的报
告，包括安全活动实施范围、安全活动结果。
8.2.1 功能测试与安全测试区别
Functional testing and security testing are not the same
Functional Testing: Verifying that an application can be used by legitimate users
Security Testing: Verifying that an application cannot be misused by malicious users
安全测试注意事项:
➢ Single Rule of Security Testing: there are no rules!
➢
➢
➢
“Malicious users wouldn’t try to do that”! Yes they will!
Malicious users will try to circumvent application client components
Malicious users will try to compromise application dependencies
8.2.2 人工测试
待各项安全报告均验收通过后，进行人工安全测试。在自动化验收和自动化安全测试之
前，该活动往往会成为整个流程中压力最大和最易造成阻塞的点。据了解，部分公司会选择
性进行人工安全测试；有的会针对特有的常见问题进行专项安全测试，比如仅测试越权漏
洞…；有的会通过堆人进行较为全面的人工安全测试，比如内部没有 hc 就让很多外包驻场。
8.2.3 漏洞验证
人工安全测试往往是进一步提高软件的安全质量，发现之前各环节遗留的安全漏洞，包
括但不限于：越权类漏洞、安全配置错误类漏洞、API 返回过多敏感信息类漏洞、重要场景
重放攻击以及常见的较为隐蔽 web 漏洞。然而同样重要的，还有推动安全漏洞修复，再次
验证直至修复闭环。
图 8-1
安全测试与普通测试的区别
(https://mp.weixin.qq.com/s/WO089RBiLuaHMzQ4yTMnUg)
图 8-2
常见漏洞(https://mp.weixin.qq.com/s/WO089RBiLuaHMzQ4yTMnUg )
8.2.4 参考文献
【SDL 最初实践】安全测试
https://mp.weixin.qq.com/s/WO089RBiLuaHMzQ4yTMnUg
9
缺陷&漏洞扫描
源码编程规范检查速度快、成本低，可以放在开发者的 IDE 或者提交代码的时候自动在
后台中执行；
静态弱点分析成本较高，时间较长，可能需要项目根据需要定期进行；
动态分析成本高，不仅时间长，可能还需要准备相关的环境，并且会对环境带来脏数据，
因此项目根据需要设定(也许设定成质量门限)。因此，动态分析和模糊测试(一种特殊的动态
分析)比较适合放在验证阶段。
动静态分析的交互式分析应该比较适合在 CI 构建时或者在验证阶段执行。
9.1
主要措施
9.1.1 源码编程规范检查
源代码合规检查指利用自动化静态分析工具发现代码中不符合安全编程规范的内容，目
前国际上比较主流的安全编程标准包括 CERT C/C++/Java、MISRA C/C++等。依据 SEI CERT(计
算机安全应急响应小组)编码标准与 OWASP 安全编码实践、CWE(通用缺陷列表)、STIG( 安全
技术实现指南)以及其他互联网大厂编码规范，企业自身最佳实践。
源码合规检查，一般是轻量级的工具(例如 pclint)，能很短时间内完成检测，可以方便的
集成在开发者的 IDE 当中或者在代码提交时自动被调用。
软件开发企业应该根据主管部门监管要求和自身的开发现状，总结归纳符合自身特点的
企业安全编程规范，并采用合适的工具，通过规则定制，实现自动化检查。
9.1.2 源码缺陷分析(静态)
源码缺陷分析指利用自动化静态分析工具发现源代码中的缓冲区溢出、SQL 注入、跨站
脚本等安全缺陷的过程。目前静态分析工具的缺陷检测规则大多是基于 CWE 、OWASP Top10、
CWE/SANS Top25 等 标 准 提 取 的 ， 不 同 工 具 支 持 的 语 言 多 少 不 等 ， 但 对 于 主 流 的
C/C++/JAVA/PHP 等语言支持的较多。
当前静态分析工具主要的问题仍然是误报较多，因此需要进行人工审计，该项工作需要
一定的背景知识，一般由安全人员辅助开发人员来完成。
9.1.3 源码溯源&合规治理
源码溯源检测是面向复杂的供应链现状提出的，主要是基于开源代码库，自动化地检测
软件中是否引用了开源代码模块，引用的开源代码模块是否存在已知的安全漏洞，以及软件
使用授权(License)问题。
DevSecOps 等开发方法中对开源软件和供应链安全的关注度较高，而源码溯源检测可以
在很大程度上规避和降低开源代码引入的法律和安全风险。完善的开源代码库是溯源检测的
基础。
第三方源码尽量保持到最新的发布版本。
9.1.4 开发流程对接
与开发流程对接主要是指，源码分析工具应当以最小代价透明地融入开发和测试流程中。
如今开发流程越来越规范，为了方便版本控制，许多企业在开发时都使用代码管理系统，如
SVN、Git 等，而开发人员均使用自己习惯的 IDE。因此，良好的对接应该是源码分析工具可
以以插件的方式嵌入主流的 IDE 中，实现一键式启动；并且分析工具支持从代码管理系统中
自动获取代码进行检测，检测结果可与 Bugzilla 等 Bug 管理系统进行整合。
9.1.5 结果可视展现
可视化的结果呈现、方便的验证操作和统计数据对比等都是提高源码安全保障效率的有
效手段。具体而言，可视化的分析结果应当包括：缺陷密度、缺陷分布、不同版本的检测结
果对比、缺陷触发路径的图形化展示等内容。这些结果对于快速地定位和修复源码安全问题
提供方便。
9.1.6 静态分析
SDL Practice 10: Static Analysis【来自微软简化版 SDL】
Project teams should perform static analysis of source code. Static analysis of source code
provides a scalable capability for security code review and can help ensure that secure coding
policies are being followed. Static code analysis by itself is generally insufficient to replace a manual
code review. The security team and security advisors should be aware of the strengths and
weaknesses of static analysis tools and be prepared to augment static analysis tools with other
tools or human review as appropriate.
Generally speaking, development teams should decide the optimal frequency for performing
static analysis – to balance productivity with adequate security coverage.
项目团队应对源代码执行静态分析。
源代码的静态分析为安全代码审查提供了可伸缩的功能，并且可以帮助确保遵循了安全
编码策略。 静态代码分析本身通常不足以代替人工代码审查。 安全团队和安全顾问应了解
静态分析工具的优缺点，并准备适当地使用其他工具或人工审查来扩充静态分析工具。
一般而言，开发团队应确定执行静态分析的最佳频率-以平衡生产力和足够的安全覆盖
率。
9.1.7 参考文献
做源头上的安全：内建安全源码保障
https://www.freebuf.com/articles/es/150242.html
9.2
自定义 checker
KWCOV 工具不是银弹
非 libc 标准接口操作引发资源泄漏监控:自封装 API， KW、Coverity 不识别
非串行化资源申请与释放监控:异步消息流程资源申请与释放
十年前的问题今天仍在出现: 资源泄漏、指针越界、数组下标越界、栈覆盖
开发自定义 Checker && 固化到 CI 流程
➢ 编程规范 Checkers(部分)
 禁止以八进制形式初始化变量
 禁止 SWITCH-CASE 使用魔法数字
 禁止条件语句内间接调用函数
 禁止循环控制变量为全局变量
 禁止 DOUBLE 与 LONG 变量以小写后缀`l`数字初始化
 禁止函数返回值为魔法数字
 禁止 SIEZOF 作用于指针变量
 禁止浮点数数字直接==比较
 本地变量命名规范性检查
 全局变量命名规则检查
➢
 使用一致的前缀来区分变量的作用域
 PRINTF 系列函数缺少 FORMAT 控制,存在注入攻击风险
 一条语句只完成一个功能，禁止监控嵌套赋值
 避免布尔变量与 TRUE 或者数字比较
 条件判断句数值变量不可采用 BOOL 方式比较
 应当将指针变量与 NULL 比较
 避免直接 STRUCT 定义变量
 函数参数个数须要小于 5
 避免参数变量用作当做工作变量
经典故障经验实例化
10 业界实践
10.1 Mitre 隐私测试
11 测试类型
11.1 测试方法
Testing approach
3.1 Static, dynamic and passive testing
3.2 Exploratory approach
3.3 The "box" approach
3.3.1 White-box testing
3.3.2 Black-box testing
3.3.2.1 Visual testing
3.3.3 Grey-box testing
11.1.1 Exploratory approach
Exploratory testing is an approach to software testing that is concisely described as
simultaneous learning, test design and test execution.
11.1.2 The "box" approach
Software testing methods are traditionally divided into white- and black-box testing. These
two approaches are used to describe the point of view that the tester takes when designing test
cases. A hybrid approach called grey-box testing may also be applied to software testing
methodology.
11.1.2.1 白盒测试
Techniques used in white-box testing include:
➢ API testing – testing of the application using public and private APIs (application
➢
➢
➢
➢
programming interfaces)
Code coverage – creating tests to satisfy some criteria of code coverage (e.g., the test
designer can create tests to cause all statements in the program to be executed at least
once)
Fault injection methods – intentionally introducing faults to gauge the efficacy of testing
strategies
Mutation testing methods
Static testing methods
Code coverage tools can evaluate the completeness of a test suite that was created with any
method, including black-box testing. This allows the software team to examine parts of a system
that are rarely tested and ensures that the most important function points have been tested.[23]
Code coverage as a software metric can be reported as a percentage for:
➢ Function coverage, which reports on functions executed
➢ Statement coverage, which reports on the number of lines executed to complete the test
➢ Decision coverage, which reports on whether both the True and the False branch of a
given test has been executed
100% statement coverage ensures that all code paths or branches (in terms of control flow)
are executed at least once. This is helpful in ensuring correct functionality, but not sufficient since
the same code may process different inputs correctly or incorrectly.[25] Pseudo-tested functions
and methods are those that are covered but not specified (it is possible to remove their body
without breaking any test case).[26]
11.1.2.2 Black-box testing
Black-box testing (also known as functional testing) treats the software as a "black box,"
examining functionality without any knowledge of internal implementation, without seeing the
source code. The testers are only aware of what the software is supposed to do, not how it does
it.[27] Black-box testing methods include: equivalence partitioning, boundary value analysis, allpairs testing, state transition tables, decision table testing, fuzz testing, model-based testing, use
case testing, exploratory testing, and specification-based testing.
11.2 测试级别
4 Testing levels
4.1 Unit testing
4.2 Integration testing
4.3 System testing
4.4 Operational acceptance testing
Testing levels
Broadly speaking, there are at least three levels of testing: unit testing, integration testing,
and system testing。However, a fourth level, acceptance testing, may be included by developers.
➢ Unit testing refers to tests that verify the functionality of a specific section of code,
usually at the function level. In an object-oriented environment, this is usually at the class
level, and the minimal unit tests include the constructors and destructors.
These types of tests are usually written by developers as they work on code (white-box
style), to ensure that the specific function is working as expected. One function might
have multiple tests, to catch corner cases or other branches in the code.
➢ Integration testing is any type of software testing that seeks to verify the interfaces
between components against a software design. Software components may be
integrated in an iterative way or all together ("big bang").
➢
System testing。System testing tests a completely integrated system to verify that the
system meets its requirements.[
Operational acceptance is used to conduct operational readiness (pre-release) of a product,
service or system as part of a quality management system. OAT is a common type of non-functional
software testing, used mainly in software development and software maintenance projects. This
type of testing focuses on the operational readiness of the system to be supported, or to become
part of the production environment. Hence, it is also known as operational readiness testing (ORT)
or Operations readiness and assurance (OR&A) testing.
11.3 测试类型
5 Testing types, techniques and tactics
5.1 Installation testing
5.2 Compatibility testing
5.3 Smoke and sanity testing
5.4 Regression testing
5.5 Acceptance testing
5.6 Alpha testing
5.7 Beta testing
5.8 Functional vs non-functional testing
5.9 Continuous testing
5.10 Destructive testing
5.11 Software performance testing
5.12 Usability testing
5.13 Accessibility testing
5.14 Security testing
5.15 Internationalization and localization
5.16 Development testing
5.17 A/B testing
5.18 Concurrent testing
5.19 Conformance testing or type testing
5.20 Output comparison testing
11.3.1 回归测试 Regression
Regression testing focuses on finding defects after a major code change has occurred.
Specifically, it seeks to uncover software regressions, as degraded or lost features, including old
bugs that have come back. Such regressions occur whenever software functionality that was
previously working correctly, stops working as intended. Typically, regressions occur as an
unintended consequence of program changes, when the newly developed part of the software
collides with the previously existing code. Regression testing is typically the largest test effort in
commercial software development,[52] due to checking numerous details in prior software
features, and even new software can be developed while using some old test cases to test parts of
the new design to ensure prior functionality is still supported.
Common methods of regression testing include re-running previous sets of test cases and
checking whether previously fixed faults have re-emerged. The depth of testing depends on the
phase in the release process and the risk of the added features. T
11.3.2 Continuous testing
Continuous testing is the process of executing automated tests as part of the software delivery
pipeline to obtain immediate feedback on the business risks associated with a software release
candidate.[57][58] Continuous testing includes the validation of both functional requirements and
non-functional requirements; the scope of testing extends from validating bottom-up
requirements or user stories to assessing the system requirements associated with overarching
business goals.
11.3.3 Software performance testing
Performance testing is generally executed to determine how a system or sub-system performs
in terms of responsiveness and stability under a particular workload. It can also serve to investigate,
measure, validate or verify other quality attributes of the system, such as scalability, reliability and
resource usage.
Load testing is primarily concerned with testing that the system can continue to operate under
a specific load, whether that be large quantities of data or a large number of users. This is generally
referred to as software scalability. The related load testing activity of when performed as a nonfunctional activity is often referred to as endurance testing. Volume testing is a way to test software
functions even when certain components (for example a file or database) increase radically in size.
Stress testing is a way to test reliability under unexpected or rare workloads. Stability testing (often
referred to as load or endurance testing) checks to see if the software can continuously function
well in or above an acceptable period.
There is little agreement on what the specific goals of performance testing are. The terms load
testing, performance testing, scalability testing, and volume testing, are often used
interchangeably.
11.4 测试流程(process)
测试流程比如传统瀑布流、敏捷或 XP 开发模式、
11.4.1 Agile or XP development model
In contrast, some emerging software disciplines such as extreme programming and the agile
software development movement, adhere to a "test-driven software development" model. In this
process, unit tests are written first, by the software engineers (often with pair programming in the
extreme programming methodology). The tests are expected to fail initially. Each failing test is
followed by writing just enough code to make it pass.[68] This means the test suites are
continuously updated as new failure conditions and corner cases are discovered, and they are
integrated with any regression tests that are developed. Unit tests are maintained along with the
rest of the software source code and generally integrated into the build process (with inherently
interactive tests being relegated to a partially manual build acceptance process).
The ultimate goals of this test process are to support continuous integration and to reduce
defect rates.
11.5 参考文献
https://en.wikipedia.org/wiki/Software_testing
12 测试简介
图 12-1 测试使用场景(https://www.guru99.com/smoke-sanity-testing.html )
12.1 冒烟测试简介
12.1.1 定义
In computer programming and software testing, smoke testing (also confidence testing, sanity
testing,[1] build verification test (BVT)[2][3][4] and build acceptance test) is preliminary testing to
reveal simple failures severe enough to, for example, reject a prospective software release. Smoke
tests are a subset of test cases that cover the most important functionality of a component or
system, used to aid assessment of whether main functions of the software appear to work
correctly.[1][2] When used to determine if a computer program should be subjected to further,
more fine-grained testing, a smoke test may be called an intake test.[1] Alternately, it is a set of
tests run on each new build of a product to verify that the build is testable before the build is
released into the hands of the test team.[5] In the DevOps paradigm, use of a BVT step is one
hallmark of the continuous integration maturity stage.[6]
A daily build and smoke test is among industry best practices.[8][need quotation to verify]
Smoke testing is also done by testers before accepting a build for further testing.
Smoke tests can be functional tests or unit tests. Functional tests exercise the complete
program with various inputs. Unit tests exercise individual functions, subroutines, or object
methods.
【https://en.wikipedia.org/wiki/Smoke_testing_(software)】
“Smoke testing is a type of software testing in which the most important functions are
tested to ensure that they work properly. Smoke testing, also known as ‘Build Verification
Testing’, comprises a set of non-exhaustive tests that verify that the build is stable enough for
further testing.”
Smoke testing is a set of basic cheap to run tests that precede actual testing. It aims to
verify that the build is deployed successfully and that all test env. aspects are running and
ready for the actual test process. It saves you bringing the full extent of your testing wrath
down a faulty build and just realizing that you have been testing on a bad env. or erroneously
deployed build possibly too late.
Smoke testing, also called build verification testing or build acceptance testing, is
nonexhaustive software analysis that ascertains that the most crucial functions of a program
work but does not delve into finer details.
The term smoke testing originates from a similarly basic type of hardware testing in
which a device passes the test if it doesn't catch fire the first time it turns on. Daily build and
smoke tests are among industry best practices advocated by the Institute of Electrical and
Electronics Engineers in Code Complete by former IEEE editor in chief and software engineer
expert Steve McConnell.
【https://searchsoftwarequality.techtarget.com/definition/smoke-testing 】
Smoke Testing is a software testing method that determines whether the employed build
is stable or not. It acts as a confirmation whether the quality assurance team can proceed with
further testing. Smoke tests are a minimum set of tests run on each build.
Smoke Testing is also known as Confidence Testing or Build Verification Testing.
【https://www.geeksforgeeks.org/smoke-testing-software-testing/ 】
12.1.2 测试用例组成
A smoke test consists of functional or unit tests of critical software functionality. Smoke testing
comes before further, in-depth testing.
Smoke testing answers questions like
➢ "Does the program start up correctly?"
➢ "Do the main control buttons function?"
➢ "Can you save a simple blank new test user account?"
If this basic functionality fails, there is no point investing time in more detailed QA work at
this stage.
表 12-1
T.ID
冒烟测试的一个用例(https://www.guru99.com/smoke-testing.html )
TEST
SCENARIOS
EXPECTED ACTUAL
STATUS
RESULT RESULT
DESCRIPTION
TEST STEP
Test the
login
functionality
of the web
application
to ensure
that a
registered
user is
allowed to
login with
username and
password
1.Launch the
application
2.Navigate
the login
page
3.Enter
valid
username
4.Enter
valid
password
5.Click on
login button
Login
as
Pass
should expected
be
success
1.Select
categories
list
2.Add the
item to cart
Item
should
get
added to
the cart
Item is Fail
not
getting
added to
the cart
The user
should
be able
to sign
out.
User is Fail
not able
to sign
out
1
Valid login
credentials
2
Adding item Able to add
functionality item to the
cart
3
Sign out
Check sign
1. select
functionality out
sign out
functionality button
12.1.3 使用场景
Use Smoke Tests during the early stages of a project or product
You can conduct Smoke Tests every day, every sprint, every release
图 12-2
检验构建是否有效(https://www.testbytes.net/blog/smoke-testing-explanationexample/)
12.1.4 优点
主要优点:
➢ It helps to find faults earlier in the product lifecycle.
➢ It saves the testers time by avoiding testing an unstable or wrong build
➢ It provides confidence to the tester to proceed with testing
➢ It helps to find integration issues faster
➢ Major severity defects can be found out
➢ Detection and rectification will be an easy process
➢ The unstable build is a ticking time bomb. Smoke Testing diffuses it
➢ Can be executed within a few minutes
➢ Since execution happens quickly, there will be a faster feedback
➢ Security, privacy policy, performance, etc. can also be tested
12.1.5 与 Sanity 测试区别
12.1.6 参考文献
https://en.wikipedia.org/wiki/Smoke_testing_(software)
https://searchsoftwarequality.techtarget.com/definition/smoke-testing
https://reqtest.com/testing-blog/smoke-testing-2/
https://developer.mozilla.org/en-US/docs/Glossary/Smoke_Test
https://www.testbytes.net/blog/smoke-testing-explanation-example/
https://www.softwaretestinghelp.com/smoke-testing-and-sanity-testing-difference/
https://www.edureka.co/blog/what-is-smoke-testing/
12.2 Sanity 测试
Sanity testing is different than smoke testing and is done during the release phase to check
the main functionalities of the application without going deeper. It is also called a subset of
Regression testing.
12.3 Sanity 与冒烟测试对比
#5 Difference between Sanity Testing and Smoke Testing
They’re not the same – no they’re not.
Sanity Testing is Smoke Testing’s cousin, once removed. That is, Sanity Tests happen
slightly later in the product lifecycle, and mainly as a safety net to check whether intended
enhancements and bug fixes have been completed as expected.
So Sanity Tests have a much more specific scope compared to Smoke Tests, and happen
later in the project or product lifecycle. If you are interested to learn more, the link above
describes the differences in greater detail.
表 12-2
两者对比(https://www.testbytes.net/blog/smoke-testing-explanation-example/ )
Smoke Testing
Sanity Testing
To check critical functionalities
To check new functionalities are
working or bugs are fixed
Used to the check stability of the Used to check rationality in
system
order to move into deeper tests
Performed by both developers and
testers
Restricted to testers
A form of acceptance testing
A form of regression testing
Build can be stable and unstable
when smoke testing is performed
Relatively stable when sanity
testing is performed
The entire application is tested
Critical components is tested
表 12-3
Features
两者对比(https://www.edureka.co/blog/what-is-smoke-testing/ )
Smoke Testing
Sanity Testing
System
Builds
Tests are executed on
initial builds of software
product
Tests are done over builds
that have passed smoke
tests & rounds of
regression tests
Motive of
Testing
To measure the stability of
the newly created build to
face off more rigorous
testing
To evaluate rationality &
originality of the
functionalities of
software builds
Subset of?
Is a subset of acceptance
testing
Is a subset of regression
testing
Documentat
ion
Involves documentation and
scripting work
Doesn’t emphasize any
sort of documentation
Test
Coverage
Shallow & wide approach to
include all the major
Narrow & deep approach
involving detailed testing
Performed
By?
functionalities without
going too deep
of functionalities and
features
Executed by developers or
testers
Executed by testers
12.4 参考文献
https://www.guru99.com/smoke-sanity-testing.html
13 参考文献
[1] SAFECode_Fundamental_Practices_for_Secure_Software_Development_March_2018,
Testing and Validation, Perform Penetration Testing, P25
[2] CSSLP Certification All-in-One - Wm. Arthur Conklin(英文第二版)，Secure Software Testing，
Chapter 15 Security Quality Assurance Testing，Chapter 16 Security Testing