Lucas Jellema (AMIS, The Netherlands)
THE VERY, VERY LATEST IN
ORACLE DATABASE DEVELOPMENT
Oracle Open World 2012, San Francisco
Public Expertezed Session – Thursday 29th November 2012
THE VERY VERY VERY LATEST…
<Secret Code>
TOM KYTE TO THE RESCUE…
THE DATABASE
IN MODERN ARCHITECTURES
Oracle Open World 2012, San Francisco
NO SQL
THE TOP-3 EARNING EMPLOYEES
• What can you say about the result of this query with
respect to the question: “Who are our top three
earning employees?”
A.
B.
C.
D.
Correct Answer
Sometimes correct
Correct if there are never duplicate salaries
Not Correct
IN-LINE VIEWS
TOP-N QUERIES IN 12C
• Last part of a query to be evaluated – to fetch only
selected rows from the result set:
select *
from
emp
order
by
sal desc
FETCH FIRST 3 ROWS ONLY;
– To select the next set of rows:
select
from
order
by
OFFSET
*
emp
sal desc
3 FETCH NEXT 4 ROWS ONLY;
TOP-N% QUERYING
• To query for a percentage of the result set (rather than
an absolute number of rows)
select *
from
emp
order
by
sal desc
FETCH FIRST 30 PERCENT ROWS ONLY;
• And the next batch
select *
from
emp
order
by
sal desc
OFFSET (0.3*(select count(*) from emp)) ROWS
FETCH NEXT (0.3*(select count(*) from emp))
ROWS ONLY;
BOTTOM-N QUERY IN 12C
• Return only the last three rows in the ordered result
set (in the proper order)
select *
from
emp
order
by
sal desc
OFFSET ((select count(*) from emp)-3)
ROWS
FETCH NEXT 3 ROWS ONLY
– or:
select *
from
( select *
from
emp
order
by
sal asc
FETCH FIRST 3 ROWS ONLY
)
order by sal desc;
IN-LINE PL/SQL FUNCTIONS AND
PROCEDURES
• Procedures are also allowed in-line
• In-Line Functions and Procedures can invoke each
other
WITH
procedure increment( operand in out number
, incsize in number)
is
begin
operand:= operand + incsize;
end;
FUNCTION inc(value number) RETURN number IS
l_value number(10):= value;
BEGIN
increment(l_value, 100);
RETURN l_value;
end;
SELECT inc(sal)
from
emp
SPECIAL ‘BUSINESS RULE’: DEFAULT VALUE
• The default values is the value that should be inserted
for a column when the client has ignored the column
– not provided a value nor indicated NULL
• The default value is applied prior to the execution of
the Before Row trigger
– So :new.<column_value> has the value that will be
inserted
– The Before Row trigger has no built in way to telling
whether the value was provided by the client or
supplied as default by the database
• Default value is typically used for auditing purposes
– Note: default values for columns exposed in UI should
be set in the client
COLUMN DEFAULT
• Columns can have default values
– Static or literals
– SQL expressions evaluating to a static
– Pseudo-columns like USER and CURRENT_DATE
• DO NOT USE SYSDATE! DO NOT USE USER!
– References to Application Context parameters
• sys_context(‘USERENV’, ‘IP_ADDRESS’)..
– Some funny value to let the before row trigger know
that the real (complex) default must be calculated
create table citizens
( name
varchar2(100) default 'John Doe'
, birthdate date
default current_date - 1
, city
varchar2(50) default
sys_context('KANE_CTX', 'DEFAULT_CITY' )
, zipcode
varchar2(8)
default 'XYXYXYXYXQQ'
)
NEW OPTIONS WITH DEFAULT VALUE
alter table emp
modify (sal number(10,2)
DEFAULT ON NULL 1000
)
alter table emp
modify (empno number(5) NOT NULL
DEFAULT ON NULL EMPNO_SEQ.NEXTVAL
)
create table emp
( empno NUMBER GENERATED BY DEFAULT AS IDENTITY
(START WITH 100 INCREMENT BY 10)
, ...
)
APPLICATION CONTEXT
• Memory area that enables application developers to
define, set, and access key/value pairs
Application
• Rapid access in SQL and PL/SQL
Context
Attribute
Attribute
Value
Value
Attribute Value
Pairs
select sys_context('USERENV', 'SESSION_USER')
from
dual
l_user:= sys_context('USERENV', 'SESSION_USER')
• Two Application Contexts
are always around:
– CLIENTCONTEXT and USERENV
APPLICATION CONTEXT APPEARANCES
• Per session (default)
– Stored in UGA, just like package state
• Globally Accessible (shared across all sessions)
– Stored in SGA
• Associated with a Client Identifier
– Attributes in a Globally Accessible Application Context
can explicitly be tied to the Client Identifier
– And are only accessible to sessions with that Client
Identifier
TYPICAL WEB ARCHITECTURE USING
CONNECTION POOL
JDBC Connection Pool
Session 1
Package A
globals
Session 2
Package B
Session 3
Package C
Session 4
PACKAGE STATE IS TIED TO DATABASE
SESSION
JDBC Connection Pool
Session 1
Session 2
Session 3
globals
Package A
globals
Package B
Package C
Session 4
PACKAGE STATE IS TIED TO DATABASE
SESSION – NOT WEB SESSION
JDBC Connection Pool
Session 1
Session 2
Session 3
globals
Package A
globals
Package B
Package C
Session 4
APPLICATION CONTEXT TO RETAIN
STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1
Package A
globals
Session 2
Session 3
globals
?
Package B
Package C
Session 4
APPLICATION CONTEXT TO RETAIN
STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1
Session 2
Session 3
USERENV
Package A
globals
USERENV
Global Context
globals
globals
Session 4
Package C
APPLICATION CONTEXT TO RETAIN
STATE FOR LIGHT WEIGHT END USERS
JDBC Connection Pool
Session 1
USERENV
Package A
globals
Session 2
Session 3
USERENV
USERENV
Global Context
globals
globals
Session 4
Package C
PACKAGE GLOBALS: THE STATE OF THE
PACKAGE IN A SESSION
• This state is lost when the package is recompiled
– That is undesirable in a highly available environment
Package
PACKAGE GLOBALS CAN BE REPLACED BY
APPLICATION CONTEXT
• The Application Context is untouched by
recompilation of the package
– All ‘globals’ in the application context retain their values
Package
Application Context
EBR TO KILL PLANNED DOWNTIME
(BECAUSE OF APPLICATION UPGRADE)
Application X
VERSION 1
Application X
VERSION 2
TIME TRAVELLING
FLASHBACK
•
•
•
•
Introduced in 9i
Based on UNDO
Initially only for recovery
As of 11g – Total Recall option with
Flashback Data Archive
– Controlled history keeping
• Look back into history
– Query trends (version history)
– Difference reporting
– Audit trails (Replace journaling tables)
• Require trick for transaction history: WHO?
• Also: when is the start of history?
OOW 2012 SESSION COMES TO THE
RESCUE
• CON8511 - Temporal Database Capabilities with the
Latest Generation of Database Technology
TOTAL RECALL - FLASHBACK DATA
ARCHIVE IMPROVEMENTS
• Complete schema evolution support: all table
definition, partitioning, and space management DDLs
are supported on FDA-enabled tables.
• The metadata information for tracking transactions
including the user context is now tracked.
– This could mean that journaling tables are now officially
deprecated
• And the current contents of journaling tables can even be
migrated to Flashback Data Archive
• Introduction of SQL 2011 Valid Time Temporal
Modeling
TOTAL RECALL
• Import and export of history
– Support for import and export using Data Pump for
FDA-enabled tables. Data Pump can now be used to
export and import an FDA-enabled base table along
with its schema-evolution metadata and historical row
versions.
• Construct and manipulate the Flashback Data Archive
– import user-generated history
• Restore points: Support for the use of named restore
points in AS OF and versions queries has been added.
• Total Recall will (in all likelihood) be part of every
edition of the database – including SE
VALID TIME TEMPORAL MODELING
• Validity (or effectivity) of facts recorded in a database
is frequently specified through dates or timestamps
– For example begin date and [derived] end date of a
price, membership, allocation, certificate, agreement
• This valid time can differ from the transaction time at
which a record is entered into the database
• Multiple entries with different, non-overlapping validtime periods can exist for a single entity
• In 12c the notion of Valid Time is introduced into the
Oracle Database
– The valid-time dimension consists of two date-time
columns specified in the table definition (create or alter)
– These Valid Time columns specify the period during
which a record is valid
– A table can have multiple valid_time markers
CREATING A TABLE WITH VALID TIME
DIMENSION
• Table with explicit valid time columns:
CREATE TABLE EMP
( employee_number NUMBER
, salary NUMBER
, department_id NUMBER
, name VARCHAR2(30)
, hiredate TIMESTAMP
, firedate TIMESTAMP
, PERIOD FOR user_time (hiredate, firedate)
);
• Table with valid time dimension and implicit columns:
CREATE TABLE EMP
( employee_number NUMBER
, salary NUMBER
, department_id NUMBER
, name VARCHAR2(30)
, PERIOD FOR contract_time
);
columns valid_time_start and valid_time_end
(TIMESTAMP) are added implicitly
VALID TIME AWARE FLASHBACK QUERIES
• Select all employees who were employed at a certain
moment in time
SELECT *
FROM
EMP AS OF PERIOD FOR user_time
TO_TIMESTAMP('01-JUN-2012 12.00.01 PM')
• Perform all queries for records that are valid at a
certain point in time
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time
( 'ASOF'
, TO_TIMESTAMP('29-JUL-12 12.00.01 PM')
);
• Return all records currently (session time) valid
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT');
• Return all records (default)
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');
DATABASE IN MODERN ARCHITECTURE
WS
Mobile
Business Tier
Enterprise
Service Bus
Cache/Grid
(L1, L2, L3)
Services
Database
Database
Standard
Application
s
Legacy
Application
s
MULTI TIER ARCHITECTURE
WS
Mobile
Business Tier
Enterprise
Service Bus
HTTP REST
HTTP SOAP
FTP/WEBDAV
JDBC
JPA (H/EL)
Services
Monitor,
Trace, Audit
Database
Authentication &
Fine Grained
Authorization
Cache/Grid
(L1, L2, L3)
DB QRCN
HTTP
JMX, JMX
Stored
Procedures
Encapsulation
Decoupling
Caching
Business Logic
APPLICATION ARCHITECTURE:
DRIVE APPLICATION FROM META DATA
• Agility
• Design Time at Run Time
• Define part of the application behavior and appearance
through meta-data (outside the base source code)
– The default settings are defined by developers and
deployed along with the application
– Read and interpreted at run time
– Manipulated and re-read
and re-interpreted at run time
Application
• Note: very similar to the way
the database operates:
– Data Dictionary is the
meta-data driving the
meta
behavior of the database
SEPARATE
BASE DATA AND CUSTOMIZED DATA
• If a value is changed during site-level implementation
– Or run time customization
• It should be kept apart from the base ‘meta-data’
– To prevent overwriting customized data when the new
release arrives
– To allow for (temporarily) reverting to base data
• A simple solution: the Complex View with two
underlying tables approach
– Note: Select…
For Update Of
ORIGINAL_NAME
is not allowed
IO trg
Customized
Values
New release
BaseValues
REPLACE THE ORIGINAL SINGLE TABLE
WITH A TWO-TABLE BASE/CUSTOM SPLIT
• rename <original> to <base>
• create table <customizations>
as
select * from base where rownum = 0
• create or replace view <original>
as
select * from <customizations>
union all
select * from <base> b
left outer join
<customizations> c
on (b.id = c.id)
where c.rowid is null
REPLACE THE ORIGINAL SINGLE TABLE
WITH A TWO-TABLE BASE/CUSTOM SPLIT (2)
•
create or replace trigger handle_insert_trg
instead of insert on original
for each row
begin
insert into <customizations> (col, col2,…)
values(:new.col, :new.col2,…);
end;
•
create or replace trigger handle_update_trg
instead of update on original
for each row
begin
update <customizations>
set col = :new.col, …
where id = :new.id ;
if sql%rowcount = 0
then
insert into <customizations> (id, col, col2,…)
(select id, :new.col, :new.col2 from base where id = :new.id);
end if;
end;
VERY SIMILAR TO THE ARCHITECTURE OF
PLUGGABLE DATABASES
New release of
Oracle Database
ROOT
PDB
APPLICATION ARCHITECTURE: NO SQL
• NO SQL
– Complex SQL is
hidden away inside
the database
– Cache to not have
to query all the time
from the database
– … and to not take
the overhead of a
commit for not so
important data
NO
– Process first – in
memory, on
middle tier
(BigData and CEP) and only persist
what is useful
Web Browser
SQL
JEE Application Server
RDBMS
SQL
QUERY RESULT CHANGE NOTIFICATION
• Continuous Query Notification:
– Send an event when the result set for a query changes
– Background process calls PL/SQL Handler or Java
Listener or OCI client when the
Java
commit has occurred
Listener
– Event contains rowid
of changed rows
• Used for:
– Refreshing specific
data caches (middle
tier, global context)
– (custom) Replication
PL/SQL
CONTINUOUS PROCESSING OF DATA
STREAMS USING CQL
• Aggregation, Spot deviation, Match on complex
patterns
WHO IS AFRAID OF RED, YELLOW AND BLUE
• Table Events
– Column Seq number(5)
– Column Payload varchar2(200)
SOLUTION USING LEAD
• With LEAD it is easy to compare a row with its
successor(s)
– As long as the pattern is fixed, LEAD will suffice
with look_ahead_events as
( SELECT e.*
,
lead(payload) over (order by seq) next_color
,
lead(payload,2) over (order by seq) second_next_color
FROM
events e
)
select seq
from
look_ahead_events
where
payload ='red'
and
next_color ='yellow'
and
second_next_color='blue'
FIND THE PATTERN RED, YELLOW AND
BLUE
• Using the new 12c Match Recognize operator for
finding patterns in relational data
SELECT *
FROM
events
MATCH_RECOGNIZE
(
ORDER BY seq
MEASURES RED.seq AS redseq
,
MATCH_NUMBER() AS match_num
ALL ROWS PER MATCH
PATTERN (RED YELLOW BLUE)
DEFINE
RED AS RED.payload ='red',
YELLOW AS YELLOW.payload ='yellow',
BLUE AS BLUE.payload ='blue'
) MR
ORDER
BY
MR.redseq
,
MR.seq;
MATCH_RECOGNIZE FOR FINDING
PATTERNS IN RELATIONAL DATA
• The expression MATCH_RECOGNIZE provides native
SQL support to find patterns in sequences of rows
Table
Source &
Where
Match_
Recognize
Process
and Filter
Select &
Order By
• Match_recognize returns Measures for selected
(pattern matched) rows
– Similar to MODEL clause
• Match Conditions are expressed in columns from the
Table Source, aggregate functions and pattern
functions FIRST, PREV, NEXT, LAST
• Patterns are regular expressions using match
conditions to express a special sequence of rows
satisfying the conditions
DID WE EVER CONSECUTIVELY HIRE THREE
EMPLOYEES IN THE SAME JOB?
• Find a string of three subsequent hires where each
hire has the same job
• Order by hiredate, pattern is two records that each
have the same job as their predecessor
SELECT *
FROM
EMP
MATCH_RECOGNIZE
(
ORDER BY hiredate
MEASURES SAME_JOB.hiredate AS hireday
,
MATCH_NUMBER() AS match_num
ALL ROWS PER MATCH
PATTERN (SAME_JOB{3})
DEFINE
SAME_JOB AS SAME_JOB.job = FIRST(SAME_JOB.job)
) MR
THE SHOPPING ALGORITHM
THE SHOPPING ALGORITHM
•
shopForItem Item ( String itemName) {
driveToShop;
Item item = buyItemAtShop ( itemName);
driveHomeFromShop;
return item;
}
GET THIS WEEK’S GROCERIES
getGroceries Item[] ( String[] shoppingList) {
Item[] items = new Item[ shoppingList.length];
for (int i=0; i < shoppingList.length; i++) {
items[i] = shopForItem (shoppingList[i]);
}
return items;
}
PENSION FUND – SEPTEMBER 2012
Employer
Participants
Job & Benefits
<
>
FETCHING THE DATA OF THE PENSION
FUND FOR THE WEB APPLICATION
<
>
select *
from
employers
where id = < 324>
1 record
100s records
select *
from
participants
where employer_id = < 324>
10s records
select *
from
benefits
where participant_id = <#>
REPORTING ON MANY EMPLOYERS
select *
from
employers
100s records
1 query
10k records
select *
100s queries
from
participants
where employer_id = <#>
100k records
select *
from
benefits
10k queries
where participant_id = <#>
APPLICATION ARCHITECTURE –
BULK RETRIEVE
• Have the database bulk up the data retrieval
• Return Ref Cursor, Types and Collections or
JSON/XML
Benefits Package
select *
from
employers
where id in <some set> select *
from
participants
where employer_id in <some set>
select b.*
from
benefits b join participants p
on (p.id = b.participant_id)
where p.employer_id in <some set>
APPLICATION ARCHITECTURE –
SERVICE ENABLING
WebLogic Server
Database
Native DB
WebService
HTTP
EPG
Java/JEE
PL/SQL
package
SOA Suite
JDBC
AQ
View
Oracle
Service Bus
Other
(Email, FTP/File,
XMPP/Chat)
XML
DB
Chat/IM XMPP
Server File/FTP Server
Email Server
Table
XML/JSON
Relational/Oracle Type
JEE Server
Database
11g Native DB
WebService
HTTP
10g
EPG
http
Java App
SDO
WS
Oracle
Service
Bus
DB
Adapters
SOA
Suite
ADF BC
/SDO WS
XML
XML & XSD
JAX-WS
JPublisher
WS
Ref Cursor
JDBC
AQ
8i
AQ
JMS
JMS Queue
JMS
EJB
File
FTP
JSON/ CSV
EJB/JPA
Pojo
UMS
Other
(Email, FTP/File,
XMPP/Chat)
Chat/IM XMPP
Server
File/FTP Server
Email Server
XML
PL/SQL
package
Types & Coll
Types
utl_file,
BFILE,
URITYPE
9i XML
DB
View
Table
BUSINESS RULES
• Data Oriented Rules or Data Constraints
• Declarative support in database
– For referential integrity
• Order must be for a Customer
– For attribute and tuple rules
• Salary must be numeric,
• Hiredate may not be in the future,
• End date must come after begin date
• No declarative support for complex data rules – across
multiple records and tables
– A department in France may not have less then 20%
female employees
– Order items of type weapon may not be part of an
order that ships around Christmas
BUSINESS RULES –
WHERE AND HOW TO IMPLEMENT
• Criteria:
– Safe
– Well performant
– Reusable and maintainable
– Productive to implement
• Options
– Client side
• JavaScript
– Middle-tier
• Java, Enterprise Service Bus
– Database
• Constraints and triggers are statement level – i/o
transaction level
RDBMS NOT ALWAYS EXCLUSIVELY
ACCESSED THROUGH ONE LAYER
SOA, ESB,
WebServices
Batch Bulk
Processes
Standard
Applications
Database
Data Replication &
Synchronization
Legacy
Applications
11G VIRTUAL COLUMNS
• Add columns to a table based on an
expression
– Using ‘real’ columns, SQL Function and User Defined
Functions
– No data is stored for Virtual
Columns, only meta-data
VIRTUAL
– Virtual Columns can be
indexed
alter table emp
ADD
(income AS (sal + nvl(comm,0)))
UNIQUENESS RULES
USING VIRTUAL COLUMNS
• Business Rule:
– Not more than one manager per department
alter table emp
add constraint only_one_mgr_in_dept_uk
unique (one_mgr_flag)
alter table emp
ADD
( one_mgr_flag as
( case when job ='MANAGER'
then deptno
end
)
)
CHALLENGE: ORDERS BELONG TO A
CUSTOMER IN ONE OF TWO TABLES
• The Orders table contains Order records for
customers – either Dutch or Australian customers
• These customers are stored in two different tables
• Can we implement referential integrity to ensure that
the order’s customer exists?
OZ_CUSTOMER
ORDER
Country
Customer_Id
….
?
Id
Name
DUTCH_CUSTOMER
Id
Name
USING VIRTUAL COLUMNS
IN FOREIGN KEY RELATIONS
• A foreign key can be created on a Virtual Column
– That means for example we can have a single column
with some id
– And two virtual columns with CASE expressions that
produce NULL or the ID value
– With Foreign Keys on the Virtual Columns
ORDER
Country
Customer_Id
Dutch_id (VC)
Australian_id (VC)
OZ_CUSTOMER
Id
Name
DUTCH_CUSTOMER
Id
Name
USING VIRTUAL COLUMNS
IN FOREIGN KEY RELATIONS
alter table orders
add (australian_ctr_id as
(case country
when 'OZ'
then customer_id
end))
ORDER
Country
Customer_Id
Dutch_id (VC)
Australian_id (VC)
alter table orders
add (dutch_ctr_id as
(case country
when 'NL'
then customer_id
end))
alter table orders
add constraint odr_ocr_fk
foreign key (australian_ctr_id)
references oz_customer (id)
OZ_CUSTOMER
Id
Name
DUTCH_CUSTOMER
Id
Name
alter table orders
add constraint odr_dcr_fk
foreign key (dutch_ctr_id)
references dutch_customer
(id)
FOREIGN KEY SHOULD ONLY REFER TO
CERTAIN RECORDS USING VC
• Foreign Key can reference a UK based on a Virtual
Column
• That allows a ‘conditional foreign key’ or a foreign key
that can only reference specific records in the
referenced table
– Only refer to Women in the PEOPLE table for the
Mother Foreign Key
– Only refer to Values in the Domain Values table for the
Domain Name == ‘COLORS’
RESTRICTED FOREIGN KEYS USING
VIRTUAL COLUMNS
alter table domain_values
add (color_value as
(case domain_name
when 'COLORS'
then domain_value
end))
CARS
ID
Make
Type
Color
Year
alter table domain_values
add (country_value
as
alter
table domain_values
(case domain_name
add (country_value
as
whendomain_name
'COUNTRIES'
(case
then'COUNTRIES'
domain_value
when
end))
then
domain_value
end))
DOMAIN_VALUES
Id
Domain_Name
Domain_Value
Color_Value
Gender_Value
OrderStatus_Value
Country_Value
ShipmentMethod_Value
alter table cars
add constraint car_clr_fk
foreign key (color)
references domain_values
(color_value)
LACK OF WATERTIGHTNESS
IN TRIGGER BASED RULE VALIDATOIN
VALIDATION
•
Statement time validation means:
DML in different session
DML
validation
•
•
•
More DML
Commit
validation
To prevent leakage we should validate at commit time
– Logically correct as transaction is the logical unit
– Effects from other sessions between statement and
commit are taken into account
However: Oracle unfortunately does not provide us with
a pre-commit or on-commit trigger
Workarounds:
– Dummy Table with Materialized View On Commit Refresh
and Trigger on Materialized View
– Do a soft-commit by calling a package to do the actual
commit – that will first do transaction level checks
• Supported by a deferred check constraint that is violated by
each operation that potentially violates a business rule
SAFE SOLUTION: USE CUSTOM LOCKS
• Prior to validating a certain business rule for a specific
record – acquire a custom lock
– That identifies both Rule and Record
– Using dbms_lock
DML in different session
DML
validation
More DML
Commit
validation
• When a record is being validated for a certain rule,
other sessions have to wait
• The commit (or rollback) releases all locks
• Validation in a different session will include all
committed data
SUMMARY
•
•
•
•
•
•
•
•
•
Inline Views
Defaulting
Application Context
Flashback and the time dimension
NoSQL means smart SQL
– Cache refresh driven by change notification
– Streaming analysis before persisting
Decoupling galore
– Bulk retrieval
– Service enabling
Business Rules
EBR
12c promises even more