The 8th International Common Criteria Conference,
25~27, Sep. 2007, Rome, Italy
Gang Soo Lee
Professor, Dept. of Computer Engineering
Han Nam University, Dae-jeon, 306-791, KOREA gslee@eve.hannam.ac.kr
ICCC-2007, Rome, Gang Soo Lee 1/51




Quality Assurance


Dependability : a critical quality attribute of IT products and systems
: a function of security , safety, reliability and availability
Time & cost consuming activity for developers (or venders) and evaluators
Security Engineering




A set of methodologies for cost-effective development and evaluation of security systems
An instance of software engineering
Project management : a main activity of software engineering
Project : limited project resources (time, cost, man and tools)
CC evaluation project


Minimize time (duration) & cost of evaluation
Consider both the minimum EAL and security functionality
2/51 ICCC-2007, Rome, Gang Soo Lee



Developers (or vendors) and evaluators

Interests in evaluation cost and duration
This presentation


Results of research on estimation of evaluation cost/duration (2003 ~)
Survey on information about evaluation cost/duration at CC schemes and research results




Estimation of effort-ratio among Product-types
Estimation of effort-ratio among EALs of CC 2.3 and CC 3.1
Combination of evaluation effort-ratio among EALs and Product-types
Analysis and evaluation
ICCC-2007, Rome, Gang Soo Lee 3/51





How much does evaluation cost ?
How long does evaluation take?
Reasons




Creative and mental activity : confirm, determine, execute/conduct/perform, test, analysis (= software development, architecture building, lawyer)
Depends on : evaluator’s environment (controllable), sponsor’s environment
(hard to control, un-predictable)
Evaluator’s environment: evaluation tools in evaluation test laboratory, ability of evaluator, support form sponsors
Sponsor’s (developer) environment: complexity of TOE (PP, EAL, ST), sponsor’s support to evaluator, quality of deliverables, communication, evaluator's environment.
4/51 ICCC-2007, Rome, Gang Soo Lee



Treat as confidential information
[“corsec” company's FAQ and “CC user Guide” (1999)]

Composition of evaluation cost (view 1)

Testing lab fee: charge on an hourly basis

Modification of product to meet CC

Preparation of deliverables: Costs of document preparation vary depending on the quality and content in the product documentation, as well as the author's familiarity with CC, and the evaluation of the vendors’ class of products.

Validation fee: pay to the Certification Body
5/51 ICCC-2007, Rome, Gang Soo Lee


Composition of evaluation cost (view 2)

Pre-evaluation cost: Pre-evaluation consultancy may be taken either from an evaluation facility or from an independent consultant.

Evaluation cost: National schemes will differ in the approach that is taken to charges for formal evaluation. A typical approach would be to offer a fixed price for the evaluation, with a further sum requested for any re-work that has to be carried out following the identification of problems.

Certification cost

Internal cost: additional testing, additional analysis, addressing issues raised by the evaluators
6/51 ICCC-2007, Rome, Gang Soo Lee


Cost influence factors

EAL: The more assurance that is required, the more work that the evaluators are required to do.

Scope of security evaluated functionality

Design of TOE: Application of software engineering methodologies (e.g., structured-oriented SE, object-oriented SE, component-based SE, aspectoriented SE)

Problems encountered: Where the evaluators encounter problems, either with the evaluation deliverables or with the product itself, there will be a need for remedial action, and some of the evaluation activities will need to be revisited.
7/51 ICCC-2007, Rome, Gang Soo Lee


Duration influence factors

EAL: assurance package claimed

The extent of security functionality

Product development timescales: timing of product releases

Quality of evaluation deliverables: If the documentation is clear, and conforms to requirements, then the evaluation will progress smoothly.

Availability of developer and evaluator resources

Quality of communication between developer and evaluator
ICCC-2007, Rome, Gang Soo Lee 8/51



KISA’s evaluation cost

Researched on estimation of evaluation cost/duration for CC since 2003.

Evaluation effort-ratio among EALs and product-types of CC 2.1

Calculate real evaluation cost for EAL1, EAL2, EAL3, EAL4 at KISA by using real cost data (in fiscal year 2000 ~ 2002) in 2005.

Even though the Republic of Korea has joined CCRA member as
"Certificate Authorizing" in May 9. 2006, tens of firewall products, introduction detection products, VPN and so on, have been evaluated and certified by using "K-criteria“ and CC since 1998. The K-criteria is previous
IT security product evaluation criteria which is a modification of ITSEC.
Evaluation level of K-criteria are K1 ~ K6. K1, K2, K3, K4 corresponds
EAL1, EAL2, EAL3, EAL4, respectively .)
9/51 ICCC-2007, Rome, Gang Soo Lee


Evaluation cost data (informal)

Obtain evaluation cost/duration information (informal and rough) from various media such as documents, communication and homepage of evaluation schemes and facilities.

The information is informal and rough.

Decide base-line value from the obtained information.

Fig. 1 presents F. Forge's data [7'th ICCC].

Fig. 2 presents GAO's data [Information Assurance - National Partnership
Offers Benefits, but faces considerable challenges, GAO-06-392, GAO,
March 2006].
10/51 ICCC-2007, Rome, Gang Soo Lee

 Fig.1
 Fig.2
ICCC-2007, Rome, Gang Soo Lee 11/51



“Korea’s SW development cost estimation criteria”
 based on B. Boehm’s COCOMO model and Function Point model

 includes
 recommended estimation method of SW development

SW maintenance and re-engineering

 construction of system operational environment construction of Database
 information strategy planning (ISP)
COCOMO’s Project cost driver
(1) Product types

 required system reliability complexity of system module
ICCC-2007, Rome, Gang Soo Lee 12/51









 extent of documentation required size of Database used required % of reusable component
(2) computer execution time constraint volatility of development platform memory constraint
(3) personal capability of project analysis personal continuity programmer capability
• programmer experience in project domain
• analyst experience in project domain
• language and tool experience
(4) project
• use of software tool
• development schedule compression
• extent of multisite working and quality of inter-site communication
ICCC-2007, Rome, Gang Soo Lee 13/51


Activity of CC evaluation
 confirm, determine, test, vulnerability analysis, etc.

SW development cost estimation model is not suitable for evaluation cost/duration of CC and CMVP evaluation.

Hardware and material testing domain

Microsoft announced test cost

‘platform’ test ($400), ‘designed for’ test ($600 ~ $11,000), ‘certified for’ test ($1,000$ ~ $30,000$), ‘retired’ test ($25,000 ~ $20,000)
[https://partner.microsoft.com/global/]

Cisco security test cost

$5,400 ~ $7,750 [http://www.keylabs.com/cisco/csec/fees.html]
14/51 ICCC-2007, Rome, Gang Soo Lee




Results of KISA evaluation cost: specific evaluation environment such as Korea’s K-criteria, and Firewall and IDS products
F. Forge’s data and GAO’s data : too rough to estimate evaluation cost for each EALs

Germany BSI scheme fee : not evaluation cost, but certification fee

There is no research on evaluation cost for each EALs of CC 2.3 and CC 3.1.

We need empirical study on effort-ratio among EALs and product types for estimation of evaluation duration and cost.
15/51 ICCC-2007, Rome, Gang Soo Lee



Product-types

DB, Firewall, VPN, Network,
OS, Smart Cards, Access
Control, Key Recovery, IDS,
MISC

Surveyed 33 PPs and 67s ST in August 2003
Product-types
DB
Firewall
VPN
Networking
OS
Smart card
Access Control
Key recovery
IDS etc.
Total
# of PPs # of STs
2
4
1
5
3
5
3
4
3
3
33
4
8
1
16
1
14
9
0
2
12
67
ICCC-2007, Rome, Gang Soo Lee 16/51




When ST (or TOE) is developed,

28% of ST has “PP conformance claim”.

83.6% of ST use functional requirements in CC
 effort-weight for all functional component by


‘Function Point method’
‘Hierarchical relation among components’
 effort-amount for all functional components
 effort-ratio among product-types from effort-amounts for all functional components
ICCC-2007, Rome, Gang Soo Lee 17/51

[evaluation effort-amount estimation algorithm]





FClass = {FAU, FCO, FCS, FDP, FIA, FMT, FPR, FPT, FRU, FTA, FTP}: a set of functional classes fs i
: a functional class ( ∈ FClass). wfs i
WFS i
: an effort-amount of fs i
: an effort-weight of fs i
. ff i
: a functional family.
wff i of ff i
: an effort-weight of ff i
. WFF i
: an effort-amount fc i
: a functional component i .
wfc i amount of fc i
: an effort-weight of fc i
. FFC i
: an effort-
PP : a set of real PP

TYPE = {DB, Firewall, VPN, Network, OS, Smart card, Access Control,
Key recovery, IDS, etc.}: a set of product-types
18/51 ICCC-2007, Rome, Gang Soo Lee



PPT i
: set of PP of Product-type type i
.
ppt i
: real PP i in PPT i
FR i
: functional requirements set used in ppt i functional requirement in CC)
(i.e., FR i
⊆ FReq, FReq is total
 fr i
: a functional component in FR i
For all fs i in FClass, do
Get wfs i by means of Function Point method; end_do;
For all ff i
in fs j in FClass, do
Get wff i by means of Function Point method; end_do;
For all fc i in ff j in fs k in FClass, do
Get wfc i by means of Hierarchical relation among components; // wfc
1 or 2 or 3 //; end_do ; i
=
19/51 ICCC-2007, Rome, Gang Soo Lee

For PPT i in TYPE, do // analysis of 33 PPs
For
Get all ppt
FR i
; i
in PPT j
, do
Compute U i,j
(usage-percentage of fr i in FReq)
TTP i
// U i,j
= (numbers of PPs using fr
) ; end_do; i
)
÷
(numbers of PPs in
// U i,j is end_do usage ratio (0 ~ 1) of functional component j of type i
//
For all i, j, k, do
WFC i,j,k
// WFC i,j,k
= wfc i
, × wff j
× wfs k
; end_do is functional component in family ff j
For all type p
For all k, do in TYPE, do in class fs i
EFF p
= EFF end_do // EFF p p
+ WFC
*,*,k
× U k,p
; end_do; is evaluation effort-amount of Product-type type p
ICCC-2007, Rome, Gang Soo Lee 20/51



Product-type of VPN has more security functions than other product-type.
DB and OS, those seem to have larger amount of evaluation effort, has small effort-ratio, because only of security functions are evaluated.
Effort-ratio among product-types
Producttype
DB
Firew all
VPN
Netw orkin g
OS
Smart
Card
Access
Contr ol
Key recov ery
IDS etc.
effortratio
1.00
0.92
1.88
1.50
1.71
1.68
1.65
1.24
0.93
1.25
ICCC-2007, Rome, Gang Soo Lee 21/51




Usage-percentage

“usage-percentage” of specific function for each product-type
Function point





Relative effort-weight (i.e., complexity of evaluation of specific security function) among functional class and family in a specific product-type (e.g.,
OS) is estimated
An useful complexity metrics for SW development cost estimation
Empirically estimated functional point values of each functional classes and families from 4 graduate students who has 1~3 years of SW development
If effort-weight of FAU is 69.71, then that of class FCO is 58.49.
If effort-weight of FAU_ARP is 51.58, then that of class FAU_SAA is 72.28.
22/51 ICCC-2007, Rome, Gang Soo Lee


Hierarchica l relation among components


Effort-weight of a component in a family is defined by level of tree which present ‘hierarchical relations among functional components’
For example, FAU_SAA’s tree (i.e., hierarchical relations) includes 4 components.
 effort-weight of FAU_SAA.1 = 1
 effort-weight of FAU_SAA.2 and FAU_SAA.3 = 2
 effort-weight of FAU_SAA.4 = 3
23/51 ICCC-2007, Rome, Gang Soo Lee



Overview

Assurance component

‘developer action elements’, ‘content and presentation of evidence
(CPE) elements’, ‘evaluator action (EA) elements’

Common evaluator action (CEA)

“The evaluator shall confirm that the information provided meets all requirements for content and presentation of evidence.”

Evaluation effort : proportion to numbers, amount (i.e., whole or part) and rigorousness (i.e., informal, semi-formal or formal) of the ‘CPE elements’
24/51 ICCC-2007, Rome, Gang Soo Lee

1. Identify and adjustment of numbers of ‘CPE elements’ for each assurance component



In some case, numbers of elements in ‘CPE elements’ of upper level of component is the same as that of lower component, even semantic (i.e., amount and rigorousness) of statement is different from each other.
Thus, we adjustment the numbers of ‘CPE elements’ for each assurance component.
Let AF.i.Ck
( k = 1 , ... , #AF.i.C
) be element statement of ‘CPE elements’ of assurance component i (denotes AF.i
) in an Assurance Family AF, #AF.i.C
be number of elements, #AF.i.C* be adjusted number of elements.
Then following assertions should be satisfied:
#AF.i.C ≤ #AF.i+1.C
( i=1, ..., n )
#AF.i.C* ≤ #AF.i+1.C* ( i=1, ..., n )
#AF.i.C ≤ #AF.i.C* ( i=1, ..., n )
25/51 ICCC-2007, Rome, Gang Soo Lee


To Satisfy the assertion, following rules should be applied:


Rule 1: #AF.1.C* = #AF.1.C
Rule 2: #AF.i.C* = #AF.i.C + added # of statements
×
0.5 ( i = 2, ...., n)
(example, ‘part’ is changed to ‘full’, ‘semi-formal’ changed to ‘formal’.)

Rule 3: if #AF.i.C > #AF.i+1.C then #AF.1+1.C* = #AF.i.C* +
1 (for ADV_FSP of CC 3.1)

For example, results of adjustment of the six assurance components in
ADV_FSP of CC 3.1 are present in next Table.

[Note: # of added statements of ADV_FSP.4 is “ - 1”. This is an error of
CC 3.1. #ADV_FSP.4.C* = 9+1 = 10 according to Rule 3.
26/51 ICCC-2007, Rome, Gang Soo Lee

components metrics
# of statements
(#AF.i.C)
ADV_FSP.1 ADV_FSP.2 ADV_FSP.3 ADV_FSP.4
4 6 7 6
# of added statements added word adjusted # of statements
(#AF.i.C*)
4
2 1
‘ complete’,
‘all’,
‘describ e’
‘ complete’,
‘all’,
‘describe’
,
‘summari ze’
7.7 = 6 +
3 × 0.5
9 = 7 + 4 × 0.5
‘
-1 all'
10 = 9+1
ADV_FSP.5
9
3 1
‘ complete’, ‘all’,
‘describe’,
‘summarize’
, ‘all’,
‘semiformal’
‘ complete’, ‘all’,
‘describe’,
‘summarize’
, ‘all’,
‘semiformal’
, ‘formal’
11 = 9 + 6 × 0.5
ADV_FSP.6
10
13.5 = 10 + 7 × 0.5
ICCC-2007, Rome, Gang Soo Lee 27/51

2. Evaluation effort-weight of EA elements

EA elements for each assurance components in CC are consisted of
 one common EA.
 zero or more component specific EA elements.

Next Table presents result of effort-weight for EA element.

Empirically obtained by means of
 a question from real evaluators working in the Korea Information
Security Agency (KISA), an evaluation facility of the Korea evaluation and certification scheme (KECS) in 2003.
28/51 ICCC-2007, Rome, Gang Soo Lee

Evaluator action elements (EA) check confirm
(satisfactio n of requiremen t)
1. sample check
2. all check
3. confirm of ‘content and presentation of evidence element’ (common action)
4. application of security measures (e.g., ALC_DVS.1.2E)
5. conformance (meets) (e.g., AVA_CCA.1.2E)
6. partial result
7. selected verification
8. analysis result
9. correctness (e.g., AVA_SOF.1.2E)
10.consistency (complete, coherent, and internally consistent)
11. application of standards (e.g., ALC_TAT.2.2E)
12. scope
13. execution effort-weight
0.54
0.78
1.40
1.41
1.68
1.63
1.00
1.41
1.46
1.34
1.47
1.33
1.38
ICCC-2007, Rome, Gang Soo Lee 29/51

determine
(independent analysis) test repeat vulnerability analysis
14.accuracy and completeness of design
15. resistant to penetration attack (high)
16. resistant to penetration attack (moderate)
17. resistant to penetration attack (low)
18.dependencies of requirements
19. partial test (sample of tests )
20. sample test of test result
21. test of all test result (all tests )
22. independent test (e.g., AVA_MSU.3.5E)
23. penetration test (e.g., AVA_VLA.1.2E)
24. additional penetration test (e.g., AVA_VLA.2.2E)
25. repeat of configuration and installation
26. etc.
27. vulnerability analysis
ICCC-2007, Rome, Gang Soo Lee
2.53
2.35
2.40
5.24
4.97
2.15
2.61
5.92
9.71
5.45
5.26
1.48
1.49
6.11
30/51


Rigorousness among EALs
 confirm: used to indicate that something needs to be reviewed in detail, and that an independent determination of sufficiency needs to be made.
The level of rigorous required depends on the nature of the subject matter.
 demonstrate: refers to an analysis leading to a conclusion, which is less rigorous than a “proof”.

 determine: requires an independent analysis to be made, with the objective of reaching a particular conclusion. It implies a truly independent analysis, usually in the absence of any previous analysis having been performed. verify: similar in context to “confirm”, but has more rigorous connotations.
31/51 ICCC-2007, Rome, Gang Soo Lee

 effort-weight
 relative amount or difficulty of evaluation of EA element in comparing to the common EA.
 may be changed according to evaluation environment such as, evaluator's ability, evaluation tools, and so on.
3. Evaluation effort-weight of assurance components and families


Next table presents estimation of effort-weight of AVA_VAN of CC 3.1.
Assumed effort-weight
: ‘independent vulnerability analysis’ = 6.11,
‘implementation presentation’ = 7.11, no level penetration test = 5.45, basic level penetration test = 4.45, enhanced-basic level penetration test = 5.45, moderate level penetration test = 6.45, high level penetration test = 7.45.
32/51 ICCC-2007, Rome, Gang Soo Lee

components items
ADA_VAN.1 ADA_VAN.2 ADA_VAN.3
EA elements
Penetration test (for identified vulnerability,
Basic attack ) [4.45]
ADA_VAN.4 confirm [1] (This is a common EA.) perform (open source search) [0.78]
Penetration test
(for identified vulnerability,
Enhanced-Basic attack [4.45]
Penetration test
(for identified vulnerability,
Moderate attack )
[6.45]
[effortweight]
perform
(independent vulnerability analysis)
[6.11] perform
(independent vulnerability analysis, implement presentation) [7.11] perform
ADA_VAN.5
Penetration test
(for identified vulnerability,
High attack )
[7.45]
(independent and methodological vulnerability analysis, implement presentation) [7.11] total effortweight
6.23
12.34
14.34
15.34
16.34
ICCC-2007, Rome, Gang Soo Lee 33/51

 evaluation effort-weight of assurance component
= adjusted number of elements + total effort-weight of assurance component - 1
(where, “- 1” because common EA is counted twice.)
4. Evaluation effort-weight of EALs

From the evaluation effort-weight of assurance component and assurance component definition table of EALs (EAL1 ~ EAL7), we evaluate effortweights for each EALs of CC 2.3 and CC 3.1.

Next table presents evaluation effort-weight of EALs.
34/51 ICCC-2007, Rome, Gang Soo Lee

EALs metrics
# of families
# of developer requirement elements
CC 2.3
CC 3.1
CC 2.3
CC 3.1
# of 'CP elements'
CC 2.3
CC 3.1
# of adjusted 'CP elements'
CC 2.3
CC 3.1
effort-weight among EALs effort-ratio among EALs
EAL1
15
13
20
17
68
49
68
49
EAL2
CC 2.3
91.9
128.59
CC 3.1
64.81
113.47
CC 2.3
CC 3.1
1
1
1.4
1.75
93
84
93
86
21
19
31
31
EAL3
107
98
109
103
25
22
35
35
149.63
131.88
1.63
2.03
ICCC-2007, Rome, Gang Soo Lee
EAL4
140
108
143
116.5
31
24
46
39
203.51
147.38
2.21
2.27
EAL5
231.25
158.5
2.52
2.45
156
113
162
123
34
25
52
42
EAL6
178
125
187
138.5
34
27
54
45
267
177.65
2.9
2.74
EAL7
181
130
193
146
34
27
58
50
279.41
190.61
3.04
2.93
35/51

5. Evaluation effort-weight of composed assurance packages in CC 3.1

Evaluation effort-weights for each CAP-A, CAP-B, CAP-C of composed TOE are estimated
CAPs
CAP-A CAP-B metrics
# of families
# of ‘developer action elements’
# of ‘CP elements’
# of adjusted ‘CP elements’ effort-weight among CAPs effort-ratio among CAPs
15
21
55
55.5
75.86 95.47
1
ICCC-2007, Rome, Gang Soo Lee
1.26
16
23
68
69
CAP-C
16
23
71
72
99.47
1.31
36/51




Effort-ratio of EAL i
= (effort-weight of EAL i
) / (effort-weight of EAL1)
We distribute evaluation activity of AST class to all EAL of CC v
2.3, because the activity of ST evaluation is included in all EALs of CC 3.1.

Effort-ratio of CAP i
= (effort-weight of CAP i
) / (effort-weight of CAP-A)
37/51 ICCC-2007, Rome, Gang Soo Lee



Next table presents combination of effort-ratios of EAL and effort-ratios of product-types.
EALs and their effort-ratio
Product-types and their effort-ratio
CC 2.3
EAL1
1
CC 3.1 1
EAL2
1.40
1.75
EAL3
1.63
2.03
EAL4
2.21
2.27
DB
Firewall
VPN
1.00
0.92
1.88
CC 2.3
1.00
CC 3.1
1.00
CC 2.3
0.92
CC 3.1
0.92
CC 2.3
1.88
CC 3.1
1.88
1.40
1.75
1.29
1.61
2.63
3.29
1.63
2.03
1.50
1.87
3.06
3.82
2.21
2.27
2.03
2.09
4.15
4.27
EAL5
2.52
2.45
2.52
2.45
2.32
2.25
4.70
4.61
EAL6
2.9
2.74
2.90
2.74
2.67
2.52
5.45
5.15
EAL7
3.04
2.93
3.04
2.93
2.80
2.70
5.72
5.51
avg.
1.71
2.03
1.59
2.03
1.49
1.92
2.65
3.27
38/51 ICCC-2007, Rome, Gang Soo Lee

Network
OS
Smart card
Key recovery
Intrusion etc.
avg.
detection
1.50
1.71
1.68
Access control 1.65
1.24
0.93
1.25
1.38
CC 2.3
1.50
CC 3.1
1.50
CC 2.3
1.71
CC 3.1
1.71
CC 2.3
1.68
CC 3.1
1.68
CC 2.3
1.65
CC 3.1
1.65
CC 2.3
1.24
CC 3.1
1.24
CC 2.3
0.93
CC 3.1
0.93
CC 2.3
1.25
CC 3.1
1.25
CC 2.3
1.24
CC 3.1
1.24
2.89
1.74
2.17
1.30
1.63
1.75
2.19
1.99
1.99
2.10
2.63
2.39
2.99
2.35
2.94
2.31
3.35
2.02
2.52
1.52
1.89
2.04
2.54
2.37
2.37
2.45
3.05
2.79
3.47
2.74
3.41
2.69
ICCC-2007, Rome, Gang Soo Lee
3.75
2.74
2.81
2.06
2.11
2.76
2.84
2.93
2.93
3.32
3.41
3.78
3.83
3.71
3.81
3.65
4.04
3.12
3.04
2.34
2.28
3.15
3.06
3.30
3.30
3.78
3.68
4.31
4.19
4.23
4.12
4.16
4.83
3.77
3.63
2.83
2.72
3.80
3.66
4.06
4.06
4.56
4.40
5.20
5.01
5.11
4.92
5.02
4.52
3.60
3.40
2.70
2.55
3.63
3.43
3.78
3.78
4.35
4.11
4.96
4.69
4.87
4.60
4.79
2.95
1.88
2.37
1.51
1.93
1.62
2.38
2.19
2.74
2.45
3.02
2.41
2.99
2.38
39/51



Base-line ratio and value
 base-line : a specific product-type and EALs of which evaluation cost and duration information have available (e.g., EAL4 of Firewall product)
 base-line ratio : a effort-ratio of specific product-type and EAL (e.g., EAL4 of Firewall product is 2.03.)
 base-line value : substitute cost (or duration) value for base-line (e.g.,
EAL4 of Firewall product is $25,000). It is estimated from historical evaluation cost data of real CC evaluation cases.

If we assume base-line cost of EAL4 of Product-type X as $25,000, then estimated evaluation costs for each EALs are presented in the next table .
40/51 ICCC-2007, Rome, Gang Soo Lee


Estimation of Evaluation Duration

Evaluation cost (or evaluation Man-Month or Man-Day) is proportioned to evaluation duration * number of evaluators.

If we fix number of evaluators (e.g., 3 persons), then evaluation cost is proportioned to evaluation duration.

Thus, evaluation duration is estimated by means of the cost estimation method.

If we assume base-line duration of EAL4 of Product-Type X as 12 months
(i.e., 260 working days), then estimated evaluation durations for each EALs are presented in the table of next page.
41/51 ICCC-2007, Rome, Gang Soo Lee

EAL
EAL1 EAL2 EAL3 EAL4 EAL5 EAL6 EAL7
CC
Cost
CC
3.1
Duration
(Months)
$11,013 $19,796 $22,963 $25,192 $27,715 $30,995 $33,145
5.4
9.5
11.0
12.3
13.3
14.9
15.9
Cost
CC
2.3
Duration
(Months)
$11,331 $15,837 $18,439 $25,000 $28,507 $32,805 $34,389
5.4
7.6
8.9
12 13.7
15.8
16.5
ICCC-2007, Rome, Gang Soo Lee 42/51


In lower EAL (EAL2 and EAL3), CC 3.1 has more evaluation effort-amount than CC 2.3.

If a TOE is evaluated by using CC 3.1, then it will takes more cost or duration then those of CC 2.3 in evaluation of lower EALs.

In case of composed TOE, difference of evaluation effort-ratios among CAPs are not so high: CAP-A : CAP-B : CAP-C = 1 : 1.26 :
1.31
43/51 ICCC-2007, Rome, Gang Soo Lee

Trend of evaluation effort-ratio among EALs of CC 2.3 and CC 3.1
ICCC-2007, Rome, Gang Soo Lee 44/51




In KISA’s evaluation fee and Germany BSI (The Bundesamt fur
Sicherheit in der Informationstechnik)’s certification fee, evidences of effort-ratio among EALs does not available.
There is no open evidence to answer “why does certification fee of
EAL 7 charged 9.1 times larger than EAL1 in Class I type products. in Germany BSI’s certification fee.”
BSI’s fee policy, example product of classes are:

Class I: smart card readers

Class II: PC security desktops, smart card applications software, general applications software

Class III: standard OS, smart card hardware and OS, firewalls
45/51 ICCC-2007, Rome, Gang Soo Lee

EALs criteria and results
CC3.1
This result
CC2.3 (including ST evaluation)
Result of 2003's work (CC 2.1)
Result of
2005’s work
KISA's real evaluation cost
(not CC)
KISA 's predicted evaluation duration evaluation effort-ratio
KISA's evaluation fee (01.5
~ now) re-evaluation effort-ratio
Germany
BSI fee
Standard ratio of evaluation day
(2007)
Class I (simple)
Class II (medium)
Class III (extensive and complex)
EAL1 EAL2 EAL3
1
1
1
1
1
1
1
1
1
1
1
1.75
1.4
1.39
1.36
1.65
1.26
1.72
1.71
1.92
1.81
1.49
2.03
1.63
1.62
1.78
1.93
1.63
2.58
2.11
3.06
3.09
2.34
EAL4
2.27
2.21
2.15
2.66
3.11
2.16
3.30
3.03
3.89
3.89
3.25
ICCC-2007, Rome, Gang Soo Lee
EAL5
2.45
2.52
2.39
-
-
3.06
4.16
-
5.18
5.18
5.2
EAL6
2.74
2.9
2.69
-
-
3.85
4.88
-
6.91
6.86
5.77
EAL7
2.93
3.04
2.80
-
-
4.74
5.74
-
9.1
8.85
7.35
46/51




Estimate relative evaluation effort-ratio among EALs of CC 2.3 and CC 3.1




 effort-weigh t of evaluator action elements adjustment of number of statements in ‘content and presentation of evidence elements’ effort-weight of assurance components effort-ratio among EALs amount of evaluation efforts of EALs of CC 2.3 and CC 3.1
Estimate relative evaluation effort-ratio among product-types from real PP/ST

 usage percentage of specific function for each product-type function point (effort-weight) of security functions (class, family, components)
ICCC-2007, Rome, Gang Soo Lee 47/51

 hierarchical relation of components in a family

Estimate relative evaluation effort-ratio among combination of
EALs and product-types

Survey and estimate a base-line value (e.g., EAL4 of DB producttype is 150,000$)

Estimate evaluation duration

The results, even they are estimated and empirical value, will useful for

TOE sponsors as well as evaluation project managers in a new CC 3.1 based evaluation scheme

CC based evaluation project management
48/51 ICCC-2007, Rome, Gang Soo Lee



This research should be continued as CC is to be evolved.

Empirical study on evaluation of test and evaluation cost

Get more real cost and duration data from CC evaluation facility, even they are dealt with confidential business data.

Develop a new estimation model by using conventional Software
Engineering and Project Management methods such as COCOMO, Function
Point, and so on.

Relative complexity of security functional classes, families, components, as well as real products, packages, PPs, product-types

Relative complexity of security assurance class, family, component, as well as EALs
ICCC-2007, Rome, Gang Soo Lee 49/51


Note


CC evaluation cost and duration is depended on evaluation environment (e.g., tool, evaluator, policy, developer, ….)

This presentation gives overview of my approach on estimation of
CC evaluation cost and duration.

Research on estimation of CC evaluation cost and duration is open problem.


This work has been supported by Wan-suck Lee, KISA, and Ji-hoon Jung, NSRI.
This presentation is supported by Kab-seung Kou, a Ph.D student of Hannam
Univ .
50/51 ICCC-2007, Rome, Gang Soo Lee

















Common Criteria for Information Technology Security Evaluation Part 1, 2, 3, Version 3.1, Revision 1, September 2006.
Common Criteria for Information Technology Security Evaluation - Evaluation methodology, Version 3.1, Revision 1,
September 2006.
Common Criteria for Information Technology Security Evaluation Part 1, 2, 3, Version 2.3, August 2005.
“CC based evaluation duration estimation method and evaluation fee policy”, KISA research report, Nov. 2003.
Gang-soo Lee et al., “Evaluation effort amount model for EAL and type of product under CC” (Korean), Journal of
Korea Information Security , Vol. 14, No. 1, pp.25-34, Feb. 2004.
“Research on evaluation fee model”, KISA TM, 2005.12.
“Criteria of man-day cost of SW engineers” (Korean), Korea Software Industry Association, 2006.
“Software development cost estimation criteria”, Korea MOE, 2005-22 (2006. 4. 27) http://www.sw.or.kr
.
“Criteria of evaluation of engineering cost” (Korean), Korea MOS, 2004.
F. Forge, “Ways to CC evaluation cost reduction - beyond CC V3”, 7'th ICCC, Sep. 2006. http://www.7iccc.es/t3/t3191430.pdf.
Information Assurance - National Partnership Offers Benefits, but faces considerable challenges, GAO-06-392, GAO,
March 2006. http://www.gao.gov/new.items/d06392.pdf
.
Regulations on Ex-parte Costs on Official Acts of the Federal Office for Information Security (BSI Regulations on Exparte Costs – BSI-KostV), March 2005 http://www.bsi.bund.de/english/exparte_costs.pdf
.
IT security system evaluation authentication guide (korean), KISA, 2006.12.
B. Boehm, et al., “Software Cost Estimation with COCOMO II”, Prentice-Hall, 2000.
T. Jones, “Estimating Software Costs”, McGraw-Hill, 1998.
Albert B. Jeng and Yu-Min Yu, “Analysis of the composition problems in CC v3.1 rev.1 with some suggested solutions”, ICCC 2006, Spain, 2006.9.
51/51 ICCC-2007, Rome, Gang Soo Lee