CRASH Report 2014
CAST Research on Application Software Health
The Global State of Application Structural Quality in 2014
Dr. Bill Curtis
Stanislas Duthoit
CAST Research Labs
CAST
Confidential
Alexandra
Szynkarski
How the CRASH Data Was Collected
Appmarq
Repository
CAST Application
Intelligence Platform
Industry
Benchmarking
Compliance
checks for
architecture
rules
Appmarq
Quality
characteristics
Quality
& Sizing
Size
Code,
Architecture,
and Data
Structure
Analysis
Application
Calibration
Violations
Demographics
Anonymized /
Normalized
Application Source
Code Analysis
CAST Confidential
Reports
CRASH
Reports
1
1
Developing Structural Quality Data
Language
Parsers
Oracle PL/SQL
Sybase T-SQL
SQL Server T-SQL
IBM SQL/PSM
C, C++, C#
Application
Analysis
CAST Application
Intelligence
Platform
Detected
Violations
Quality
Measurements
Expensive operation in loop
Static vs. pooled connections
Complex query on big table
Large indices on big table
Performance
Pro C
Cobol
CICS
Visual Basic
VB.Net
Empty CATCH block
Uncontrolled data access
Poor memory management
Opened resource not closed
Evaluation of
1200+ coding &
architectural rules
Robustness
ASP.Net
Java, J2EE
JSP
SQL injection
Cross-site scripting
Buffer overflow
Uncontrolled format string
Application
meta-data
XML
HTML
Javascript
Security
VBScript
PHP
PowerBuilder
Oracle Forms
PeopleSoft
SAP ABAP,
Netweaver
Tibco
Business Objects
Universal Analyzer
for other languages
Unstructured code
Misuse of inheritance
Lack of comments
Violated naming convention
Transferability
Highly coupled component
Duplicated code
Index modified in loop
High cyclomatic complexity
Changeability
CAST Confidential
2
The CRASH Data
Quality Characteristics

Robustness  stability, resilience, and recovery from operational interruptions

Performance  responsiveness and efficient use of resources

Security  ability to prevent unauthorized intrusions

Transferability  understandability of an application by other teams

Changeability  ease of modifying source code

Total Quality Index  aggregate score of the 5 quality characteristics above

Size  lines of code
Demographic Characteristics

Language/Technology

Industry Sector

Source and Shore
212 orgs

CMMI Level
706 Mloc

Development Method
CAST Confidential
1316 apps
3
2
Application Sizes in the CRASH Sample
300
Frequency
250
200
150
100
50
0
10K 20K
>20K 50K
>50K 100K
>100K - >200K - >500K - >1M - 5M
200K
500K
1M
>5M
Lines of Code
CAST Confidential
4
Distribution of Languages by Industry Sector
Industry
Orgs
Apps
J-EE
Cobol
.NET
Mixed
Financial Serv.
51
421
179
146
20
40
Insurance
34
314
61
97
26
4
Telecom
19
187
106
1
22
Manufacturing
25
169
65
14
17
Utilities
15
56
34
Government
17
56
36
Retail
12
48
18
IT Consulting
17
41
C
C++
ASP
KLOC
17
5
6
8
299,249
5
9
4
1
6
113,930
16
7
9
2
14
6
4
62,786
5
49
4
8
6
1
65,787
1
3
1
3
2
8
4
18,057
1
1
3
8
1
1
1
3
9
4
6
5
1
2
15
4
9
2
1
8
13
6
2
40
13
12
32
17
Energy
12
22
6
Other
12
30
15
Total
212
1316
565
CAST Confidential
Oracle Oracle
Forms ERP
1
Software ISV
Business Serv.
ABAP
5
1
6
1
5
1
280
127
84
4
4
1
1
2
4
3
77
59
33
39
25,356
31,076
2
1
5
4
5
25,570
2
10,740
1
35,971
2
11,671
5,742
28
24
705,935
5
3
Performance
Security
Transferability
Changeability
Lines of Code
.31
.60
.58
.62
.15
.22
.36
.37
.00
.27
.13
-.09
.55
.00
Pearson
Correlation
Coefficients
TQI
Robustness
Correlation Among Software Characteristics
Robustness
.85
Performance
.57
.31
Security
.61
.60
.22
Transferability
.78
.58
.36
.27
Changeability
.75
.62
.37
.13
CAST Confidential
.55
.07
6
Distributions of Quality Characteristic Scores
Robustness
Performance
Changeability
Transferability
Security
Demographics
were not reported
for all apps. Only
J-EE had enough
apps reporting
demographics to
support statistical
analyses
CAST Confidential
7
4
No Differences in Structural Quality by Sourcing Choice
Robustness
In-house
Outsourced
Changeability
Performance
In-house
Outsourced
Security
In-house
Outsourced
Transferability
In-house
Outsourced
n = 224
n = 277
All F-tests insignificant
df = 1, 499
In-house
Outsourced
CAST Confidential
In-house
Outsourced
8
CMMI Level 1 Delivers Lower Structural Quality
Robustness
Level 1
Level 2
Level 3
Changeability
Performance
Level 1
Level 2
Level 3
Security
Level 1
Level 2
Level 3
Transferability
Level 1
Level 2
Level 3
n = 23
n = 26
n = 32
All F-tests signif.
df = 2, 78 p < .01
Size was insignif.
Level 1
Level 2
CAST Confidential
Level 3
Level 1
Level 2
Level 3
9
5
Agile/Waterfall Mix Best, No Method Worst
Robustness
Agile
Mix
None Other Water
Changeability
Performance
Agile
Mix
None Other Water
Security
Agile
Mix
None Other Water
Transferability
Agile
n = 57
Mix (Agile, Water) n = 46
No method
n = 21
Other
n = 36
Waterfall
n = 60
All F-tests signif.
df = 4, 215; p < .02
Agile
Mix
None Other Water
Agile
Mix
CAST Confidential
None Other Water
10
Apps for More Users Have Higher Structural Quality
Robustness
< 500
501 - 5000
> 5000
Changeability
Performance
< 500
501 - 5000
> 5000
Security
< 500
501 - 5000
> 5000
Transferability
< 500 users n = 50
501 - 5000
n = 37
> 5000 users n = 101
All F-tests signif.
df = 2, 185; p < .02
< 500
501 - 5000
CAST Confidential
> 5000
< 500
501 - 5000
> 5000
11
6
Summary of Global Trends
 No differences based on sourcing choice
 Shoring choice can make some differences
 CMMI Level 1 delivers lower structural quality
 Agile/waterfall mix exhibits higher structural quality
 Apps for more users have higher structural quality
 Security is an aspect of software quality
CAST Confidential
12
7