Alberta Reliability Standards
Compliance Working Group Meeting
June 19, 2012
Minutes are noted on the summary slide for each section
Agenda Topics
1. Introduction
2. Presentation of discussion plan, method and scope
3. Presentation on compliance monitoring audit process issues
4. Initial discussion on solutions
5. Meeting debrief
Plan
• Review feedback and observations obtained from previous year’s audit experience
• Provide input on possible solutions to identified process issues
• Timelines
• 1st meeting (today)
• CWG review issues
• Gather input from CWG
• 2nd Meeting – DRAFT (end July or early August)
• AESO presents solutions based on first meeting input
• CWG refines solutions
• 3rd Meeting – FINAL (End August or early September)
• AESO presents updated solutions as per 2rd meeting refinements
• 4th meeting if required (End September or early October)
• Publish guidelines – one month following final meeting
Method, Scope
Method
• Go over the issues in order of the compliance monitoring audit process
sections (guideline process map)
• As a group: discuss each issue focusing on: impact, broadness and provide
input on possible solutions
• Feedback will be used by the AESO to update the audit guideline
Scope
• Scope is limited to:
• The compliance monitoring audit process
• Existing, clearly defined issues.
• Existing constraints (legislation and mandate)
• Issues outside of the scope will be tabled and brought forward at a later
date or other forum.
Process sections
Identified Compliance monitoring audit issues (13)
(numbers 7.x refer to the audit guideline sections)
Process Section
Process Section
Issues
Issues
3
7.7
Review for
completeness
3
Publish audit
schedule
0
7.8
Assess audit
evidence
0
7.2
Changes to audit
schedule
1
7.9
Issue draft report
0
7.3
Workshop
/Notification
3
7.10
Post-audit
conference
0
7.4
Pre-audit conference
0
7.11
Review draft audit
report
1
7.5
Gather audit
evidence
0
7.12
Issue final audit
report
0
7.6
Submit audit
evidence
2
7.13-14
Referral to MSA and
retain evidence
0
GEN
Information /
Guidelines
7.1
Defined Issues
General: Information/Guidelines
1. Applicability Assessments
Participants/CMP
- Process change to reflect updated applicability sections
- Complete
2. Acknowledgement and Consent
Participants/CMP
- Simplification to meet business need
- End August
3. Guideline overly complex
- Need to define specific issues
- CWG Input
Participants
Defined Issues
Audit Guideline - 7.2: Change to audit schedule
4. Request for deadline extensions
- Requests for deadline extensions were unreasonable
- CWG Review (from December 2011 CWG)
CMP
Defined Issues
Audit Guideline - 7.3: Workshop / Notification
5. Lessons learned and timing
Participants
- When can lessons learned be shared?
- CWG Input
6. Publish generic index template
Participants
- Can a generic index be published in advance?
- Related to #9
7. Asset identification
- Need to identify an asset list prior to audit
- CWG Input
Participants/CMP
Defined Issues
Audit Guideline - 7.6: Submit audit evidence
8. Bad index of evidence files
CMP
- Difficulties in finding evidence
- Created standard index template (Attachment 4)
- Related to #9 (from December 2011 CWG)
9. Audit index VS RSAWs
Participants/CMP
- What method can be used for indexing evidence?
- CWG Input
Defined Issues
Audit Guideline - 7.7: Review for completeness
10. Initial review too detailed
CMP
- Some assessment was done during completeness review
- CWG Review (From December 2011 CWG)
11. How can we help entities better manage information requests?
Participants/CMP
- Should we proposed tentative timelines at this stage?
- CWG Input
12. Additional request content is sometimes unclear
Participants
- Requests are not always clear resulting in multiple follow-up
- CWG Input
Defined Issues
Audit Guideline - 7.11: Review draft audit report
13. Re-issue of draft report
- Extent of commenting on re-issued draft report is unclear
- CWG Input
CMP
Additional issue brought forward by a participant
NEW: Difficulties with overlap of self-certification and audit
- What are the consequences of self-certifying compliance if
there is a subsequent draft audit report showing a suspected
contravention?
- Discussion was held at the meeting with follow-up at the next.
Discussion on issues
WORKSHEETS
Each worksheet will show:
1. Title
2. Background
3. Current status
4. Possible constraints
After a quick review of the issue we will begin discussions:
1. Gather thoughts on impact and broadness
2. Discuss possible solutions
Discussion on issues
SUMMARY
At the end of the discussion, we will:
1. Write a summary of the main points for all to see
2. Validate that the summary contains all pertinent discussions
NOTE: It is important that we validate the summary as these points
will be used to create the solutions that will be presented at the 2nd
meeting.
General
3
Guideline overly complex
Background
During an end of compliance monitoring audit survey a market participant stated that
the guideline was not used because it is overly complex.
Current status
“Overly complex” is unclear and we need input
Possible Constraints
The purpose of the guideline is to define the procedures for planning, executing, and
reporting with respect to each compliance monitoring audit. It has the dual purpose of
providing policy/rules and procedures (“how to”). During a compliance monitoring
audit one is more likely to simply look for the procedure which is currently mixed with
the policy/rules. It is possible to change the guideline however, both elements must
remain.
3
Audit Guideline overly complex
Our thoughts
The guideline(s) could be changed as follows:
1. Separate policy/rules from the “How to”
2. Create a simplified “how to” section
3. Use FAQs for missed items
- Update annually with FAQ transfer to guideline
4. Roll-out update to all participants (newsletter)
5. Send to new participants
Group discussion… What are the issues, impact and broadness?
3
Audit guideline overly complex
SUMMARY
The CWG:
• Agrees with the approach to split the “how to” and policy
• There is always room for improvement in the audit
guidelines
Audit guideline - 7.2: Change to audit schedule
4 (Review)
Requests for deadline extensions were unreasonable
Background
Some MPs requested extensions of all or most deadlines and did not provide sufficient
reasons. Guideline states “reasons beyond the market participant’s control” which is
not clear. Most reasons were related to internal scheduling or change of staff.
Current status
As discussed at the December 2011 CWG:
• The guideline is looking for “exceptional” reason.
• Entity needs to build a case
• We will communicate intent through compliance audit training sessions
• We will change guideline to provide better guidance
Audit guideline - 7.3: Workshops / Notification
5
Lessons learned and timing
Background
We have been asked by entities to share lessons learned. “Lessons learned” are shared
during our audit training sessions - however, this is only useful to scheduled
participants which may not be timely for all.
Current status
“Lessons learned” are provided during the audit training of a scheduled entity.
Possible Constraints
“Lessons learned” cannot be identified in the audit report - under rule 12, only
suspected contraventions are identified.
5
Lessons learned and timing
Our thoughts
As discussed at the December 2011 CWG meeting:
•
We have modified training to focus on market participant needs
•
Emphasis is placed on key areas (index, pointing to evidence, use of
attestations and relating evidence to requirements/measures (location within a
file)
Further to this, we will:
• Include a “lessons learned” module at the Stakeholder Information Sessions
• Provide a review of expectations at RSAW training
• New: Ad hoc
• Old: Request stakeholder input on interest and timing
Group discussion… Does this work?
5
Lessons learned and timing
SUMMARY
The CWG:
• Agrees that a module at the stakeholder information
sessions for lessons learned on the general audit process
would be valuable
• There is a need for RSAW sessions to go over auditor
expectations:
• Start with statistically problematic
• When possible group by standards areas
• Possibly include AESO technical SME at the session
• Suggest once a year for problem standards
Audit guideline - 7.3: Workshops / Notification
6
Publish generic index template
Background
Entities are requested to fill out the index template when submitting audit evidence.
The index is sent out as an attachment to the notification letters. Some entities
requested that a copy of the index template be published well in advance of the audit.
Current status
We provide a copy filtered by entity type as an attachment to the notification letter. A
generic (non-filtered) copy is provided upon request.
Possible Constraints
This issue is related to #8 (difficulties for auditors in finding evidence) and with #9 (use
of RSAWs or Index)
6
Publish generic index template
Our thoughts
We have no issues with publishing a generic index – however, since the need for this
depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed
as one issue.
Group discussion… Does this work?
6
Publish generic index template
SUMMARY
The CWG:
• Agreed to discuss this issue with issue #9
Audit guideline - 7.3: Workshops / Notification
7
Need to identify an asset list prior to audit
(Can extend to relationships and other information)
Background
The need for information depends on individual standard requirements and on entity
structure. As an example, a participant requested that we provide a list of assets that
will be included in the audit.
Current status
Information is requested and obtained during the audit.
Possible Constraints
Since the need for information varies by standard and by entity structure, requests for
lists cannot be generic. As an example, an entity may have different CIP procedures for
different plants within their organisation – in such a case, a list of operating personnel is
required for each plant (the asset) so that the yearly provision of the procedure can be
verified.
7
Need to identify an asset list prior to audit?
Our thoughts
Lists are not limited strictly to assets. The lists are required to perform certain
assessments. Requests depend on individual standard requirements and entity
structure which may include but is not limited to:
• Functional entity type (who owns, who operates and who performs PP function)
• Asset list to verify inclusion or exclusion
Possible solutions:
• Introduce pre-audit questionnaire – similar concept to WECC.
• More information could be requested in the RSAW.
Group discussion… Impact (timing), broadness, alternatives?
7
Need to identify an asset list prior to audit?
SUMMARY
The CWG:
• Is in favour of using a pre-audit questionnaire
• There will be a possible need for training
Audit guideline - 7.6: Submit audit evidence
8
Bad index of evidence files
Background
Based on the 2011 Q2 and Q3 audit experience, it was found that the suggested index
format as described in appendix B of the audit guideline, did not provide clear
guidelines on what was required. Further to this, entities were allowed to submit a
variety of index styles. Having a variety of index structures led to difficulties for auditors
to find the actual evidence files and prolonged the audit schedule by necessitating
requests for clarifications and re-submittal of additional evidence.
Current status
As per the December 2011 CWG meeting, we now provide a index template filtered by
entity type as an attachment to the notification letter. A generic (non-filtered) copy is
provided upon request.
Possible Constraints
The auditors need the entity to point to the evidence document/page for each
applicable requirement.
8
Bad index of evidence files
Our thoughts
Since the need for an index depends on our decision to use RSAWs or not, we propose
that 6, 8 and 9 be discussed as one issue.
Group discussion… Does this work?
8
Bad index of evidence files
SUMMARY
The CWG:
• Agreed to discuss this issue with issue #9
Audit guideline - 7.6: Submit audit evidence
9
Audit index VS use of RSAWs
Background
Issue numbers 6 and 8 outlined the difficulties that the auditors were having in finding
evidence documents because of bad indexes. Entities suggested that perhaps the RSAW
could be submitted. A pilot project enabling submission of both the RSAW and index (as
back-up) was provided to requesting entities for 2012 Q1 and Q2.
Current status
As per the December 2011 CWG meeting, we now provide an index template filtered by
entity type as an attachment to the notification letter. A generic (non-filtered) copy is
provided upon request. As a pilot project, RSAW submission with index back-up is
permitted when requested.
Possible Constraints
The intent is that the index or RSAW be used to point the auditor to the evidence
document/page related to the requirement/measure of the standard. For efficiency,
only one method should be used.
9
Audit Index VS use of RSAWs
Our thoughts
The pilot project of using RSAWs was quite successful.
• There were no major difficulties in finding evidence.
• Most submitted RSAWs used imbedded document links which proved to be quite
efficient.
• In one case the link did not work however the document was found using the
index back-up. Alternatively an additional request could have been made.
• The “Description” column was sometime used to give or state evidence. The intent
of the column is to describe the evidence (such as paragraph or page number).
• The purpose of using the RSAW should be the same as the index – to point the
auditor directly to the evidence. The advantage of the RSAW is that the
requirement, measure and assessment step are all aligned in one document and
page row.
Group discussion… Index VS RSAW? Impact and broadness?
9
Audit Index VS use of RSAWs
SUMMARY
The CWG:
• Agrees with using the RSAW instead of the index template
• Notes:
• Language in RSAW could be clearer
• Separate the request from auditor functions
o Side note: New RSAW are formatted this way
Audit guideline - 7.7: Review of completeness
10 (Review)
Initial review too detailed
Background
(Internal issue) The completeness review went beyond a simple check that all
documents were received and usable. Some assessment was being performed during
the completeness review and then repeated in more detail during the assessment
phase. Some entities were confused by the similar nature of requests during the
completeness review and assessment phases.
Current status
As discussed at the December 2011 CWG:
• The guideline will be changed to state “Administrative review” so as to align
with language used in self-certification
• We will limit the “administrative review” to the intended checks
• We will better communicate the intent through training workshops
Audit guideline - 7.7: Review of completeness
11
How can we help entities better manage information requests?
Background
Some participants have asked us to give them an idea of when to expect additional
information requests in order to better plan SME resources during the audit assessment
phase.
Current status
Although auditors try to provide an estimate of when additional requests will be
available, it is not part of the process to provide an audit assessment schedule
Possible Constraints
Information request timelines may depend on volume of material which is not known
until the initial “administrative review” is completed. Once the volume is known, a
tentative schedule of when additional requests for each standard type is possible –
however, there are too many variables to be able to determine in advance when the
draft or final report will be completed.
11
Need to establish an additional request schedule
Our thoughts
• Establishing a schedule of when information requests will help entities and us to
better prepare internal resources
• Any schedule would be on a tentative basis and could not be guaranteed
Possible inclusions in process:
• After the “administrative review”, the AESO will provide a tentative schedule of when
the entity can expect information requests.
• The entity would provide any possible constraints
• The AESO will make “best effort” to accommodate entity needs.
Group discussion… Work for you? Impact and broadness?
11
Need to establish an additional request schedule
SUMMARY
The CWG:
• Agrees with the approach of providing a schedule for
additional requests:
• Schedule is a best guess
• Group by standard area
• Possibly update tentative schedule on known changes
Ex: Where it conflicts with the constraints
Audit guideline - 7.7: Review of completeness
12
Additional request content is sometimes unclear
Background
A participant stated that additional requests are not always clear and require additional
follow-up to be understood. This leads to difficulty in tracking requests, confusion and
inefficiencies. Further to this it is not clear how an entity can request clarifications.
Current status
Process is not sufficiently detailed.
Possible Constraints
Requests are made so that the auditor is satisfied that all available evidence has been
received. Multiple requests may still be necessary.
12
Additional request process not efficient
Our thoughts
Establishing a more detailed process may help clarify request content.
• Auditors will use a template for requests
• Template to include all key information
• Standard / Requirement
• Background of request and any related documents
• Entities to use the template to respond
• Space provided for response
• Space provided to attach documents
Group discussion… Work for you? Impact and broadness?
12
Additional request process not efficient
SUMMARY
The CWG:
• Agrees that using a template is good
• Entity should communicate any need for clarifications and
the AESO will re-word the request if required.
Audit guideline - 7.11: Review draft audit report
13
Extent of commenting on re-issued draft report
Background
The guideline allows for re-issue of the draft report: “In the event a re-evaluation of
previously submitted audited evidence is deemed appropriate, and results in revised
audit findings or significant changes to the draft audit report, a new draft audit report
will be issued and the registered market participant will be provided ten business days
to review and submit their comments in writing to the compliance monitor.” It is unclear
however as to what extent comments may be provided.
Current status
No changes have been made to section 7.11 of the guideline.
Possible Constraints
None identified
13
Extent of commenting on Re-issued draft report
Our thoughts
Comments on a re-issued draft report should be limited to items that have changed.
The guideline wording would be changed to:
“In the event a re-evaluation of previously submitted audited evidence is deemed
appropriate, and results in revised audit findings or significant changes to the draft audit
report, a new draft audit report will be issued and the registered market participant will
be provided ten business days to review and submit their comments on the changes in
writing to the compliance monitor.”
Group discussion… Work for you? Impact and broadness?
13
Extent of commenting on Re-issued draft report
SUMMARY
The CWG:
• Agrees with the approach of adding “on the changes” to
this section of the audit guideline.
Next steps
• We will send power point summaries to all CWG members
• Create solutions based on today’s results
• Set next CWG meeting
• End July / Early August
• Date: Will send out survey to members on possible dates
• Time: Afternoon, mid-week is good.
• Purpose
• AESO to present solutions based on first meeting input
• CWG to refine
Thank You
Any questions or comments?