Reducing the Risk of Patching
Akbar Abdrakhmanov, Asif Iqbal, Wen Chen,
Jay Parlar, Chris George
McMaster University
in partnership with
Legacy Systems International
November 2011
1
Background
• We have a large system consisting of
Oracle database, E-business Suite and
the Customer’s Application software.
Customer’s Application
E-Business Suite
Oracle DB
2
Background
Customer’s Application
Forms, report
generators,
website, etc.
E-Business Suite
Oracle products
Oracle DB
3
Problem
• Patches are applied periodically that change or
upgrade the E-business Suite and the Oracle
Database.
• Do patches change the Customer’s Application
software behaviour?
• Reduce the risk of patching!
4
Size of the problem
• E-Business Suite contains
• almost 230,000 classes
• Almost 4 million functions (methods)
• The dependency relation we describe later has
almost 14 million edges.
• A patch may contain several thousand files.
5
Current Approach
Minimise risk by running all tests
 Risk remains that an application will
change or fail
Testing essentially blind
 Expensive: money and time
6
Solution
• Patch Analysis: Identify those places in the
database and the E-business Suite changed by
a patch.
• Impact Analysis: Identify those places in the
Customer’s Application software that may
access the changed places found by patch
analysis.
• Focused Testing: Select only tests relevant to
those places.
7
Benefits
• Reduce the risk
• Improve the effectiveness
• Reduce the cost
8
Current Testing Approach
Test Suite
Customer’s Application
Application
functions
E-Business Suite
E-Business
suite
functions
Database
objects
Oracle Database
coverage
of the test cases
9
Proposed Testing Approach
Customer’s Application
Red color indicates
direct changes caused
by the patch
E-Business Suite
Oracle Database
Yellow color indicates
places that depend
on the direct changes
indicates
dependency on
the affected places
10
Proposed Testing Approach
Focused
Test Suite
Customer’s Application
Red color indicates
direct changes caused
by the patch
E-Business Suite
Oracle Database
Yellow color indicates
places that depend
on the direct changes
indicates
dependency on
the affected places
11
Process: tool-based
Step 1: Patch Analysis: use tools to find
detailed changes in database and E-business
Suite. ( and )
Step 2: Impact Analysis: use tools to trace
back from changes to potentially affected
Customer’s Application functions. ( )
Step 3: Focused Testing: test affected
functions only.
12
Step 1: Patch Analysis
Patch Analysis: fine-grained
• An Oracle patch consists of newer versions of
older files or new files that will eventually go into
the system after the patch has been applied.
• So the granularity of an Oracle patch is at file
level, i.e., even if only a single line of a 1000 line
file has to be changed, it will replace the whole
file with a newer version.
• Therefore we need to look inside patch files to
do fine-grained analysis.
14
Patch Analysis: Contents
• The contents of a patch can be analyzed
in two ways:
 From Oracle Application Manager (Web)
 From the command line
15
Patch Analysis: Patch Wizard
• The tool in the Oracle Application Manager
that deals with patches is called the Patch
Wizard.
 High level information
 Limited for automation, because it does not output
any summary file that lists the contents of the patch
 The only output is a web page
16
Patch Analysis: Auto Patch
• In normal mode, the patch is actually applied
and changes are made to the files.
• In test mode the patch is not actually applied but
we can still get information about what would
have happened if the patch had been applied in
normal mode.
17
Patch Analysis: Auto Patch
• Two useful log files
 Adpatch.log shows all the steps during the execution of
adpatch, i.e., everything that appears on the screen.
 Adpatch.lgi shows the details of which files were
replaced or newly introduced by the patch. This provides
the initial information we need, in a form we can input to
tools.
18
Patch Analysis: Auto Patch
…
Copying
Backing
Copying
Backing
…
file /home/oracle/…/ghfed08.sql
up d01/oracle/…/ghfed08.sql
file /home/oracle/…/ghittb08.sql
up d01/oracle/…/ghittb08.sql
..New file
..Old file
..New file
..Old file
19
Patch Analysis: Two Parts
• Having found which files would be modified or
introduced by the patch, we split the fine-grained
analysis into 2 parts:
• Changes in the Database
• Changes in the E-business Suite
20
Patch Analysis: Database
File types that potentially change the database are:
class
ctl
lct
odf
pkb
pl
pll
pls
sql
xdf
xml
- Java binary source code files
- SQL loader control files
- FNDLOAD configuration files
- object definition files
- package body files
- Perl source code files
- PL/SQL library files
- PL/SQL source code files
- SQL script files
- XML definition files
- XML data files
21
Patch Analysis: E-business Suite
File types that potentially change the E-business Suite are:
class
drv
fmb
jsp
pkb
pl
pll
rdf
wft
wfx
xml
- Java binary source code files
- Driver files
- Oracle Forms binary source code files
- Java Server Pages
- Package body files
- Perl source code files
- PL/SQL library files
- Oracle Reports binary source code files
- Oracle Workflow definition files
- Business Event System definition files
- XML data files
22
Patch Analysis: Changes to Database
Customer’s Application
E-Business Suite
are the objects
in the database
that are changed
by the patch
Oracle Database
23
Patch Analysis: Changes to Database
• SQL scripts are ASCII files which can be automatically
analyzed.
• Our tools parse and analyze the scripts to find out which
tables have been modified in the script through update /
delete/ insert into/ alter table commands.
Script: inv.sql
Tables Modified:
PAY_USER_COLUMNS_INSTANCES_F
24
Patch Analysis: Changes to Database
• Dependent changes
 There are foreign key dependencies among tables in a
database.
 Explicit change in a certain table might cause implicit change in
those dependent tables.
 So we need to find all dependent tables as well.
• Our tools can detect dependent tables.
Explicitly Changed Table: PAY_USER_COLUMN_INSTANCES_F
Used by:
PAY_USER_COLUMNS
25
Patch Analysis: Changes to Database
Customer’s Application
E-Business Suite
are the objects
in the database
that themselves are
not changed by the patch
but are dependent
on the database changes
Oracle Database
26
String String Analysis
Analysis
• We have found by patch analysis the changed
database objects, and use dependencies in the
database to find all (potentially) affected database
objects (1040).
• Connections between the E-business Suite and the
database take the form of strings which are SQL
commands constructed in the Java code and passed to
the database by special statements.
• We can find the database objects (tables, procedures,
etc) referred to if we can read these strings.
• Some strings are constructed explicitly; many are
assembled from parameters, variables, etc.
StrinString Analysis: Method 1
Analysis
1. Find functions where SQL statements are
executed (10,000)
2. Find the paths from these functions to the
functions of interest: functions in the
application layer mentioned in tests.
3. Pass the classes involved in each path to the
Java String Analyzer: generates regular
expressions.
4. Discover which such expressions can include
names of affected objects as substrings, and
hence identify affected functions of interest.
StrinString Analysis: Method 2
Analysis
The Java String Analyzer (JSA) proved too slow.
So a new approach …
1.Use JSA to analyse classes directly containing
SQL strings to find which are finitely generable
(8,000), reduce to those containing affected
database objects as substrings, and find their
functions of interest (20,000)
2.Regard the remaining 2,000 SQL interacting
functions as (for now) not analysable, and find
their functions of interest (17,000)
3.Combine 1 and 2 as potentially affected
functions of interest.
Patch Analysis: Changes to Suite
Customer’s Application
E-Business Suite
Changes to the e-Business
Suite are apparent as
changed Java bytecode files
listed by the patch analysis
tool
Oracle Database
Figure 10 Changes to E-business suite
30
Patch Analysis: Changes to Suite
Process:
1. Identify new and old bytecode files
2. Convert both to an appropriate format
3. Compare old and new
4. Calculate differences to identify changed functions
31
Patch Analysis: Changes to Suite
Bytecode files
Changed
functions
Reverse
Engineer
Diff
Appropriate
Format
Figure 11 Processing bytecode files
32
Step 2: Impact Analysis
Impact Analysis
Summary so far:
• Patch Analysis has identified changed tables
and changed functions ( and )
• The potential impact is in the customer’s
application, its functions of interest ( )
• Have answered two questions:
1. What functions (may) access changed
database objects? We term these
affected functions.
2. What functions of interest may access
affected functions?
34
Impact Analysis: Affected Suite Functions
• Changed functions in the E-business Suite
functions were found by patch analysis.
• We use impact analysis to find the functions of interest
potentially affected by the changed functions.
• (Same impact analysis used previously to find
functions of interest potentially affected by SQL
interacting functions.)
35
Impact Analysis: Affected Suite
Functions
• Create a “dependency” relation for the E-Business
Suite and Customer's Application
• Use the relation to trace from affected functions to
callers in the Customer's Application
• This identifies all potentially affected functions of
interest in the Customer’s Application
• This completes Step 2: Impact Analysis
36
Impact Analysis: find changed functions
F
Customer’s Application
E-Business Suite
E
I
D
Forward
C
B
Inverse
G
A
H
A is a changed
function. We trace its
callers to functions like
F, in the customer's
application.
37
Impact Analysis: Performance
Based on our prototype:
• Building the dependency relation: 7 hours.
• Building the call graph in memory: 2 minutes.
• Call graph search time for one function:
 Worst case – just less than 30 minutes
 Best case – less than a second
 Average – a few seconds
• Executed on a Quad Core 3.2 GHz machine with 32 GB
RAM running 64-bit Linux
38
Impact Analysis: Results
Functions of interest dependent on
• Changed E-business Suite functions:
•Analysed SQL interacting functions:
411
20316
•Non-analysable SQL interacting functions: 16856
Total potentially affected functions of interest: 26104
(There is considerable overlap in the three groups.)
This is just 1.6% of the 1.6 million “top callers” in EBS.
39
Step 3: Focused Testing
Current Testing Approach
Test Suite
Customer’s Application
Application
functions
E-Business Suite
E-Business
suite
functions
Database
objects
Oracle Database
Implicit coverage
of the test cases
41
Proposed Testing Approach
Customer’s Application
Red color indicates
direct changes caused
by the patch
E-Business Suite
Oracle Database
Yellow color indicates
places that depend
on the direct changes
indicates
dependency on
the affected places
42
Proposed Testing Approach
Focused
Test Suite
Customer’s Application
E-Business Suite
Oracle Database
43
Example Client Testing Result
Focused
Test Suite
Customer’s Application
Test Results
unchanged
No tests in
test suite
E-Business Suite
Test Results
changed
Oracle Database
No tests
necessary
44
Result
• Automatic analysis of patches to find
potentially affected functions of interest.
• Approach is conservative: more
important not to miss anything than to
completely minimise tests.
• Testing focused on those functions
 improved risk analysis
 more effective than blind testing
 reduced cost
45
Thank you!
46