PUG Challenge Americas
– 2012 Remove
Click toPerformance
edit MasterBottlenecks
title style in Your SQL Connections Data Connectivity Performance Best Practices
Steve Pittman, Architect
OpenEdge SQL
1
© 2012 Progress Software Corporation. All rights reserved.
Disclaimer
The future features described in this presentation
are under consideration by Progress Software and
are not commitments for future products,
technologies, or services.
Any forward looking statements represent our
estimates only as of today and should not be relied
upon as representing our estimates as of any
subsequent date. While we may elect to update
forward looking statements at some point in the
future, we specifically disclaim any obligation to do
so, even if our estimates change.
2
© 2012 Progress Software Corporation. All rights reserved.
Agenda
Where did all that data come from?
What can we do?
A systematic approach to performance
An OpenEdge perspective
Questions?
3
© 2012 Progress Software Corporation. All rights reserved.
A Brief Audience Survey
# using ODBC
– Tools – Crystal?
# using JDBC
– Tools – DbVis?
# using Java
Java appservers
– WebLogic?
4
© 2012 Progress Software Corporation. All rights reserved.
Humans are information machines
5
Humans are now storing over 988 Exabytes
of information (at the end of 2010)
15 petabytes of information generated every
day
Scientific, Engineering, Financial, Social – all
areas of our lives continue to generate and
store the world’s data at an increasing rate.
© 2012 Progress Software Corporation. All rights reserved.
Making this data useable
6
© 2012 Progress Software Corporation. All rights reserved.
Performance trends in 2010-11 were
centered around making this data
more quickly accessible (and more
useful).
Sorting, cataloging, querying,
inserting…all these are required for
making the data mean something –
and it has to be fast!
Performance trends in 2012
Virtualization made efficiency a way of life
Concurrency will gain steam in 2012
– Research on multi-core has been building
– New tools by chip makers will help
– Additional Language constructs and
compiler directives will come
– Java application development frameworks
– Application servers – WebLogic, JBoss
7
Larger buffers and algorithm changes will
take place to fully utilize 64 bit processors.
© 2012 Progress Software Corporation. All rights reserved.
How did we get here?
Multi-tenant db
2012
1982
The advance of information technology (hardware and
software) gives you a dizzying array of choices on
how to store your data….
…and not everyone has the same opinion.
8
© 2012 Progress Software Corporation. All rights reserved.
A typical Enterprise App Architecture
Application UI
Buffers, I/O, etc..
Business Logic &
Data Access Code
Driver
Network efficiency
Database
Database Settings
9
© 2012 Progress Software Corporation. All rights reserved.
Time is of the essence
Large amounts of data
require long processing
times – small percentage
gains result in large time
savings
Performance tuning steps
remain the same
10
© 2012 Progress Software Corporation. All rights reserved.
So, what can we do to keep up?
Aggressively analyze all pieces of your
application architecture.
Just follow the PANDDa.
11
© 2012 Progress Software Corporation. All rights reserved.
Where Performance Bottlenecks Lie
P.A.N.D.Da
Platform
Application Code
Network
Driver
Database
Which element will give the most
payback?
12
© 2012 Progress Software Corporation. All rights reserved.
Platform
Consider native calls into the Platform and
whether you can do them more efficiently.
Varying client platform performance levels
–
–
–
–
Windows
Linux: 32bit, 64bit
Unix
Scale up, scale out
What are your optimization options? Are
tools available? Ex. Intel Parallel Studio
Consider Resource Usage when the
Platform is Virtualized!
13
© 2012 Progress Software Corporation. All rights reserved.
Platform – Virtualization
On a machine that has much less available Memory
and/or CPU, this graph will look different.
14
© 2012 Progress Software Corporation. All rights reserved.
Memory Usage Scalability – from the Lab
Still consumes
less Memory!!
Application #5
Application #4
Application #3
Application #2
Application #1
15
© 2012 Progress Software Corporation. All rights reserved.
Virtualization Update
Updated Performance Lab to scale Virtual
machines to measure VM Impact.
VM1
DD
1
3559.7
2
3
VM2
Comp
DD
VM3
Comp
Total
VM3
Total
2305
11741.8
7394.9
41.60%
49.40%
88.00%
58.80%
4155.9
7006
3162.7
18982.7
11347.6
48.30%
44.40% 121.50%
67.30%
4846.4
8117.6
3631.4
24172.8
13285.4
61.80%
70.80% 123.50%
82.00%
9809.8
5191
8593.5
3699
28048.8
14219.6
81.00%
89.00% 132.30%
97.30%
11031.7
5678.6
9163.5
3832.2
31061.4
15078.7
95.20%
94.30% 139.10% 106.00%
4493.3
11259.7
4146.8
35218.3
13066.2
169.30% 167.90% 171.50% 169.50%
12986.4
6312
12401.2
4451.7
38185.9
16399.9
127.10% 105.70% 178.60% 132.80%
13851.1
5820.3
13224.6
3259.7
40794.1
15397.8
117.10% 138.00% 305.70% 164.90%
6230.1
14601.7
6489
14135.6
3905.4
43167
16624.5
131.60% 125.00% 262.00% 159.70%
6294.1
15316.7
6277
14648.6
3329.3
45194
15900.4
142.00% 144.00% 340.00% 184.20%
3848.1
2575.2
5974.4
4029
6002.3
7778.9
4807.6
8276.3
4
9645.5
5329.6
5
10866.2
5567.9
6
11921.4
4426.1
12037.2
7
12798.3
5636.2
8
13718.4
6317.8
9
14429.7
10
15228.7
+
+
Comp
DD
Comparison
4334
2514.7
DD
=
Comp
VM1
VM2
The Competition needs 3 virtual machines to achieve the same
performance as 1 of our virtual machines!
16
© 2012 Progress Software Corporation. All rights reserved.
Virtualized Optimizations give you options!
Increase throughput
Add Progress | DataDirect
6277rps
6294rps
3329rps
Total = 15900 rps
15229rps
15317rps
14649rps
Total = 45194 rps
17
© 2012 Progress Software Corporation. All rights reserved.
Machine Consolidation
Application Code & Infrastructure
Faster communication
– Caching & Object Reuse
(avoiding comm!)
Increase Parallelism
– Threading, Concurrency
– Network async streams
– Not for basic reporting!
Minimize Slow operations
– Disk I/O, blocking operations
Re-architect when ?
– Reference architectures
18
Tools
– latest versions
© 2012 Progress Software Corporation. All rights reserved.
It’s like the Tower of Babel; we have
great infrastructure… but we still can’t
communicate
Network
Physical Location
Co-locate, Same country,
City, etc..
Security is expensive
How many Hops?
At what speed?
19
© 2012 Progress Software Corporation. All rights reserved.
The Network has much to tune…
20
© 2012 Progress Software Corporation. All rights reserved.
The Driver is the network
21
© 2012 Progress Software Corporation. All rights reserved.
Database
Utilize OpenEdge Tools if available
–
E.g., promon statement cache options
Data Layout
– Tune your SQL queries
– Eliminate unnecessary calls and scans
– Save MetaData
Memory tuning
– Buffer sizes, pre-fetch, etc.
Disk Usage
– Striping, storing more data in memory
22
Logging and health monitors – look in
the OpenEdge documentation
© 2012 Progress Software Corporation. All rights reserved.
OpenEdge sql techniques – better connectivity
Quick ordering
– “SELECT TOP”
Minimize round trips
– Stored procedures
Capturing preferred query form
– Views
Managing variable sized data
– dbtool for column width
– Network buffer memory impact
– Rows per network message
Managing concurrency effects
– Lockwait timeout
23
© 2012 Progress Software Corporation. All rights reserved.
SELECT TOP – Order and Time
db
TOP
ordering
App
24
© 2012 Progress Software Corporation. All rights reserved.
ORDER
BY
Sorting
machine
Optimized
ordering
Java Stored Procedure – logic on the server
App
Stored
Proc
SQL
db
25
© 2012 Progress Software Corporation. All rights reserved.
Push network
traffic
to server
Pushing best query elements into sql views
Views can be a toolbox of nicely working query components
– Pre-fab join relationships for naturally related tables
• The right tables with the right join predicates
• The right WHERE clause predicates to choose correct data
• Avoid those queries missing a vital WHERE predicate
Views can be a storehouse of totally built queries
– End users pick the view vs. end users built query from scratch
Accelerate those ad-hoc reports with views as building blocks
Views can provide security
– Expose only “need to know” tables, columns
– Screen out data which must be private or secured away
Simple to setup and use
–
–
26
CREATE VIEW myview AS <regular SELECT statement>
SELECT * FROM myview WHERE myview.region = ‘NW’
© 2012 Progress Software Corporation. All rights reserved.
Failing transactions – the problem
SELECT *
FROM customer
WHERE custnum
BETWEEN 10000 AND 20000
UPDATE customer
SET city = ‘Northboro’
WHERE custnum = 12345
12345,Smith Trucking, Westford,MA,02423,…..
Customer table
lockWaitTimeout expires
27
© 2012 Progress Software Corporation. All rights reserved.
Failing transactions – some help
Global lockwait timeout
– Environment variable PROSQL_LOCKWAIT_TIMEOUT
• export PROSQL_LOCKWAIT_TIMEOUT=30
• Significant shortcomings
JDBC Connection property
– “lockWaitTimeout”
– As of 10.2B04
String
url = "jdbc:datadirect:openedge://localhost:9878;" +
"databaseName=mys2k;lockWaitTimeout=60";
Connection conn = DriverManager.getConnection(url, ….. );
FUTURE: ODBC Connection property
– “lockWaitTimeout”, same format
– Target delivery in OE 11.2
– In ODBC Data Admin “Advanced” Tab
28
© 2012 Progress Software Corporation. All rights reserved.
Coping with column sizes
Power of the ABL
– ABL: size unlimited – TEXT
– Sql: size limited
– VARCHAR (61)
Result: “over-stuffed” columns
• OE sql error: “Column XXX has value exceeding it’s max length”
Server side fixes
– Data Dictionary
• Options - SQL Properties - Adjust Field Width
– OE SQL as DBA
• ALTER TABLE ALTER COL SET PRO_SQL_WIDTH …..
– dbtool
• High performance, multi-threaded db maintenance
• Menu choice: SQL Width Scan w/Fix Option
• Online schema updater
29
© 2012 Progress Software Corporation. All rights reserved.
Coping with column sizes - more
Client side fixes
– Find column which has size problem from error text
• “Column Name in table PUB.Customer has value
exceeding its max length or precision.”
– Change sql statement column reference to “substring()”
• SELECT Name, city, state, country FROM pub.customer ……
• SELECT SUBSTR( Name, 1, 84) AS ‘Name’, city, state,
country, ……
– OR, truncate over-stuffed value to column defined size
• SELECT SUBSTR( Name, 1) AS ‘Name’, city, state, country,
……
30
© 2012 Progress Software Corporation. All rights reserved.
Coping with column sizes – maxing out
App
Table1 Table2 Table3
result set result result
set
set
rows
rows
rows
Query
network buffers
32K * 100 cols * 50 rows = 160M
31
© 2012 Progress Software Corporation. All rights reserved.
db
New Challenges & Research in 2012
More 2012 Enterprise Performance Research
– Caching, Concurrency, Cloud deployment
Mobile devices
– 3g to 4g to wiMax
– Hardware upgrades coming fast
Services still the norm – what’s next?
–
–
SaaS
PaaS
SQL vs NoSQL
“Cloud”
32
© 2012 Progress Software Corporation. All rights reserved.
In Summary
33
Humans continue to create
copious amounts of data at
breakneck speed. This data
must be cataloged, sorted, and
queried – the latest
improvements in data access
middleware will ensure this
remains scalable for your
applications in the enterprise!
R&D trends in 2011 give way to
actionable development
changes in 2012. Follow the
P.A.N.D.Da model to keep your
application up to date!
© 2012 Progress Software Corporation. All rights reserved.
For More Information
Progress|DataDirect Website: www.datadirect.com
More than a Driver: http://web.datadirect.com/products/feature-overview/data-connectivity/anatomy/index.html
Progress|DataDirect Performance Data: http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html
Progress|DataDirect Virtualization: http://web.datadirect.com/solutions/virtualization/virtualization.html
Documentation: http://www.datadirect.com/support/product_info/proddoc_product/index.ssp
Contact Us: http://web.datadirect.com/contact-us/request_information.html
NA Tel Number: +1-800-876-3101
Intl Tel Number: +44 (0) 1753-218 930
34
© 2012 Progress Software Corporation. All rights reserved.
