PUG Challenge EMEA
2014 – Dusseldorf, Germany
Click to edit Master title style
Tales from the Audit Trails
Presented by: Mike Furgal
1
PUG Challenge Americas 2014
Introductions
•
Mike Furgal
– Progress employee from 1989
• Short time at Bravepoint from 2012 until 2014
– Progress OpenEdge Database Expert
– mfurgal@progress.com
2
PUG Challenge Americas 2014
Introduction - BravePoint
•
Managed Database Services
•
•
•
1100+ Databases
50+ TB in DB space
75,000+ connected users
Pro2 Replication
•
•
•
3
Real Time Replication
SQL target
600+ Deployments
PUG Challenge Americas 2014
Disclaimer
•
•
The techniques covered in this presentation are directed
at the users and administrators of OE applications.
Out of scope topics include:
– Establishing Trusted User Identity
– Adding Auditing Events to an application
4
PUG Challenge Americas 2014
Why Auditing?
•
Regulatory requirements…
–
–
–
–
–
•
•
5
SEC (Sarbanes Oxley)
FDA (CFR 21 Part 11)
HIPAA
Immigration (I-9s)
And more
Security
Peace of mind
PUG Challenge Americas 2014
Auditing Options
•
•
“Roll your own” ABL based solutions
OpenEdge Auditing
•
Let’s do a Poll:
–
–
–
–
6
Who has an audit system?
Who’s audit system is written in the ABL?
Who’s already using OE Auditing?
Who wishes they had auditing on their OE application?
PUG Challenge Americas 2014
“Roll your own” Solutions
– Typically use Replication or Database Triggers
– Pros
• Program in the ABL
• Very fine control
– Cons
•
•
•
•
•
7
May require access to application source code
Defeatable/insecure
High overhead
Complications (SQL access for example.)
Triggers are client based
PUG Challenge Americas 2014
OpenEdge Auditing
•
Integrated in the product since 10.1A
– No additional $ cost
•
Built into the Database Engine
– Low overhead
– Supports both ABL and SQL transactions
•
•
•
Secure and tamper proof
Tools for archiving and reporting
Unimpacted by ABL code constructs
– ie: DISABLE TRIGGERS
8
PUG Challenge Americas 2014
OpenEdge Auditing
•
Comprehensive Auditing Solution
– Table based Auditing
– Field Based Auditing
– Allows capture of Before and After versions of the data
•
•
9
Auditing includes Security so there is no tampering with
data
Some reporting capabilities built into the product
PUG Challenge Americas 2014
What does OE Auditing track?
• Changes to...
– The Database Schema
– The Application Data
– Application Defined Events
– The Security (New users, deleted users)
– Database Encryption
– The Audit Policies
10
PUG Challenge Americas 2014
Where is the data stored?
• Auditing adds tables to your production DB
– Audit trail data is stored in:
• _aud-audit-data
• _aud-audit-data-value
– Audit policy and controls are stored in:
• _aud-audit-policy
• _aud-event
• _aud-event-policy
• _aud-field-policy
• _aud-file-policy
11
PUG Challenge Americas 2014
What does basic Audit Data look like?
_Audit-data-guid
|"/unygEGMpaXiEXSzdJPDfQ"
_Database-connection-id
|"/unygEGMpaXiEXSzOlqlcw"
_Client-session-uuid
_User-id
_Audit-date-time
|""
|"root"
|2013-10-02T18:06:08.208-04:00
_Audit-event-group
_Db-guid
_Transaction-id
|""
|"4f5kvLayZrXiEXKzHE+akA"
|403
_Transaction-sequence
_Event-id
_Event-context
|0
|5101
|"PUB.Customer^F84"
_Application-context-id
_Event-detail
|""
|"Credit-Limit^F5^F12345^F22222"
_Audit-custom-detail
|""
_Audit-data-security-level
_Data-seal
12
|0
|""
PUG Challenge Americas 2014
What is in the _Event-detail field?
•
It can be a little…
"Credit-Limit^F5^F12345^F22222"
•
Or a lot….
Cust-Num^F4^F71^F^GName^F1^Fpocket billiards
co.^F^GAddress^F1^F44 Saunders
Ave.^F^GAddress2^F1^F^F^GCity^F1^FPhelan^F^GState^
F1^Fca^F^GCountry^F1^FUSA^F^GPhone^F1^F(818) 6664063^F^GContact^F1^FLeon Aida^F^GSalesRep^F1^FKIK^F^GComments^F1^F^F^GCreditLimit^F5^F5000^F^GBalance^F5^F0^F^GTerms^F1^FNet3
0^F^GDiscount^F4^F50^F^GPostal-Code^F1^F92371^F"
13
PUG Challenge Americas 2014
Enable Audit on the Database
14
•
•
Shutdown and backup database
Add storage areas for audit data and indexes
•
Enable auditing (with indexes inactive!)
PUG Challenge Americas 2014
Define Audit Policies
•
15
Use Audit Policy Maintenance Tool in GUI OpenEdge
PUG Challenge Americas 2014
Determining Auditing Requirements
•
•
•
•
•
•
16
What type of things do you want to audit?
How long do you keep your audit data?
What performance impact can you tolerate?
How does auditing fit into your disaster recovery plan?
Who can access or manage your audit rules and data?
What type of reporting/inquiry is required?
PUG Challenge Americas 2014
Roles and Responsibilities
•
The auditing system requires an administrator
– Best practice for this position has it as separate person from
the DBA
•
The administrator designates who can:
– Administer the audit rules
– Report on audit data
– Archive or delete audit data
•
17
Requires use of OE security
PUG Challenge Americas 2014
The Audit Data
•
The audit tables have 27 indexes defined
– Best practice is to have most turned off in production
– We’ll see why in a moment
•
•
•
18
Effective reporting requires that these indexes be active.
Having the audit data in an archive database allows for
reporting without negatively impacting production
OpenEdge provides tools to securely move data into the
archive database.
PUG Challenge Americas 2014
Overhead of Auditing
•
Customer example: QAD System
– Real world audit policies
– Inventory Load (36.15.1&.2)
•
Tested 5 scenarios:
–
–
–
–
–
19
Baseline without auditing
Auditing w/indexes inactive
Auditing w/indexes active
Auditing w/indexes inactive using value table
Auditing w/indexes active using value table
PUG Challenge Americas 2014
BI Logging
BI Mb Logged
60
50
40
30
BI Logged
20
10
0
QAD No Auditing
20
QAD Audit No-Index
QAD Audit w/Index
QAD Audit/Value no
Index
QAD Audit/Value
w/Index
PUG Challenge Americas 2014
Record Creates & Locks
Creates and Record Locks
450000
400000
350000
300000
250000
Creates
Locks
200000
150000
100000
50000
0
QAD No Auditing
21
QAD Audit No-Index
QAD Audit w/Index
QAD Audit/Value no Index QAD Audit/Value w/Index
PUG Challenge Americas 2014
Overhead of Auditing on DB Size
•
Keeping audit history in production may not seem
significant
5 GB database
Application Data
5gb
15gb
Audit Data
22
PUG Challenge Americas 2014
Overhead of Auditing on space
25 GB Database
•
25gb
As the database grows Audit data
takes up an immense amount of
space
75gb
25 GB of Data
2
•
Archiving out audit data keeps the
database at an appropriate size
23
25gb
This will generate a lot of BI/AI activity
PUG Challenge Americas 2014
Archive Commands
24
•
To periodically archive data out of production ready to
load into the archive database
•
To load the exported audit data into the archive
database
PUG Challenge Americas 2014
Caveat
It is not possible to truncate an area that
contains Audit data.
OpenEdge 11.2
proutil <db> -C auditreconfig [ tablearea <name>
] [ indexarea <name> ]
Recreates a new Audit Area and moves all audit
data to the new area
25
PUG Challenge Americas 2014
Reporting from Audit Data
•
Things to consider:
– Do you report off production?
– Do you report off archive
– Do you report from both?
•
Timeliness of the incident being analyzed
– Can alerts be set up?
– This is a “roll your own” activity
26
PUG Challenge Americas 2014
OE Reporting Options
27
PUG Challenge Americas 2014
DEMO
•
•
•
Start with a Sports database
Enable Auditing
Use Audit Policy Maintenance Tool
– Audit customer table
– Audit changes to customer max-credit
•
•
•
28
Create a workload on db
Polling process watching for changes to max-credit in
_aud-audit-data
Change the max credit and see what happens.
PUG Challenge Americas 2014
DEMO
29
PUG Challenge Americas 2014
Caveats
•
Make sure you use deactivateidx on the production database
audit tables
– Spelling counts!!!!
– If you rebuild all indexes all 27 audit indexes become active!!!
• Index rebuild causes worse performance
•
You need to retune both BI and AI
– Think about OE Replication and AI size impacts
•
•
Don’t forget your audit policies during a dump/load!
Managing 2 databases (production and archive)
– Backups
– Space management
– Recovery Plans
30
PUG Challenge Americas 2014
Conclusions
•
•
•
OE Auditing is a powerful tool
Knowing what needs to be audited may not be obvious
Use an iterative approach
– Audit less than you need to start
– Ramp as needed
•
•
31
Make sure you have a solid data management plan
Report and alert as needed
PUG Challenge Americas 2014
Thank You!
Questions?
32
PUG Challenge Americas 2014