PRISM MARS
Comparison of PRISM and MARS
A comprehensive technical summary that compares two LOLE assessment
tools: PJM’s Probabilistic Reliability Index Model (PRISM) and General
Electric International, Inc.’s Multi-Area Reliability Simulation (MARS).
PJM Resource Adequacy Analysis Subcommittee (RAAS)
February 17, 2011
DOCs # 599848
© PJM Interconnection 2011. All rights reserved
Table of Contents
Executive Summary ...................................................................................................................................... 3
Introduction ................................................................................................................................................... 6
Background ................................................................................................................................................... 7
Reliability Criterion … 1 day in 10 years ...................................................................................... 7
Early Reserve Study Models: GEBGE, PTI’s MAREL and GE-MARL / GE-MARS ........................... 8
Comparing GEBGE with MAREL ................................................................................................... 9
Current Reserve study Models .................................................................................................................. 10
Two–Area Model: PJM PRISM ................................................................................................... 10
Multi-Area Model: GE-MARS ..................................................................................................... 11
Complementary Aspects of PRISM and MARS ........................................................................... 14
Ongoing LOLE Assessment Work ............................................................................................... 15
Model Calculation Processes .................................................................................................................... 17
PRISM Calculation Processes ..................................................................................................... 17
MARS Calculation Processes ...................................................................................................... 23
Model Calculation Comparisons: Observations ......................................................................... 33
PRISM’s ability to automatically solve to a specified LOLE level ............................................... 34
Historical Continuity .................................................................................................................. 34
PJM’s Recent Comparison Efforts ............................................................................................................ 36
Access to models: ARC screen display and GUI layout .............................................................. 38
#1 Comparing the Models ......................................................................................................... 41
#2 Comparing the Calculations Performed................................................................................ 41
#3 Comparing the Available Standard Output .......................................................................... 41
Comparison of Attributes Documentation ................................................................................................ 42
Table 1 – Model Attributes / Application Comparison Matrix .................................................. 42
Table 2 – OUTPUT Comparison Matrix ...................................................................................... 50
Table 3 – DATABASE Modeling Relationship Matrix ................................................................. 56
Summary of Tables 1, 2, & 3 Comparisons ................................................................................ 62
Resource & Cost Assessment ................................................................................................................... 64
Frequently Asked Questions (FAQs) ........................................................................................................ 73
Interregional Assessments ........................................................................................................................ 83
Glossary....................................................................................................................................................... 85
References................................................................................................................................................... 94
© PJM Interconnection 2011. All rights reserved
Page 1 of 136
APPENDICES ............................................................................................................................................... 97
Appendix A – MARS Model vs. PRISM Model Overview PJM Staff presentation ..................... 98
Appendix B – Translation from PLOTS to LOD-UNCY table ..................................................... 103
Appendix C – Load Modeling Comparison Issues .................................................................... 105
Appendix C1 – Load distribution granularity .......................................................................... 105
Appendix C2 – Baldwin Paper................................................................................................. 107
Appendix C3 – ISO-NE’s Comparison of Westinghouse Model and MARS ............................. 110
Appendix D – MARS public solution techniques ..................................................................... 115
Appendix E – Transparency of process to perform LOLE calculations .................................... 122
Appendix F – Details of Input Parameter Requirements......................................................... 127
Appendix G – Items for possible future investigation ............................................................. 136
© PJM Interconnection 2011. All rights reserved
Page 2 of 136
Executive Summary
For many years, PJM has conducted its annual resource adequacy modeling and Installed Reserve Margin
(IRM) studies using their in-house, two-area model: the Probabilistic Reliability Index Model (PRISM)
program. By comparison, PJM’s neighboring control areas (most notably the New York Independent
System Operator (NYISO), the Independent System Operator of New England (ISO-NE) and the Midwest
Independent System Operator (MISO)) utilize a multiple-area program, the Multi-Area Reliability Simulation
(MARS) that was developed by General Electric International, Inc. (GEII).
Resource adequacy models apply both deterministic and probabilistic methods with varying degrees of data
requirements and various input / output. By nature of its inherent complexity, multi-area assessments
involve more operational specific and rigorous data requirements than does a two-area model. Multi-area
assessments also require more time, computing horsepower and resources to evaluate and interpret the
results.
The PJM Capacity Market is a $6+ billion annual enterprise. As such, the determination, characterization
and quantification of resource adequacy is important to all PJM Stakeholders. Over the past four decades,
resource adequacy modeling has typically involved the modeling of interconnected areas. Such programs
can employ either a one- or two-area modeling approach or more expansive multiple-area modeling.
Several members on the PJM Reserve Adequacy Analysis Subcommittee (RAAS) have long encouraged
PJM to adopt MARS in place of PRISM while other members believe that PRISM already fills the bill with
respect to modeling resource adequacy.
Despite the fact that both PRISM and MARS have been known and proven in the electric industry for
decades, there is a lingering question: Which is the better model for resource assessments?
Evaluation of this question began years ago – at the RAAS and its predecessor groups. By early 2010, it
was agreed that a detailed comparison was in order. The genesis of the report began with a simple
comparison table (similar to that of Table 1). Subsequent feedback from the RAAS participants thereby
prompted more comprehensive evaluation and the opportunity to document the history and evolution of
resource adequacy modeling.
PJM has used both PRISM and MARS since 2005, taking advantage of the complementary features offered
by both. This “blended” approach using both programs is regarded by PJM Staff as more beneficial than
using just one of the programs exclusively. Realizing that there is somewhat more intensive and rigorous
database requirements for MARS, PJM has relied more heavily on PRISM for its resource modeling needs.
This decision enables PJM to fully utilize their existing staff resources, structures and tools to achieve the
best technical results at the least cost.
This evaluation summarizes the models, calculations and output with detailed comparison of various
attributes, strengths and weaknesses. Cost estimates were also developed to provide a comparative
framework for needed resources.
The purpose of this report is to deliver an objective technical evaluation – not to endorse one
modeling method over another (per unanimous RAAS directive). In some cases, PRISM and MARS
provide complementary information that enhances the overall reliability study. This report is designed to
stimulate technical discussion and lay the groundwork for possible further study and future action items.
The intent of this evaluation is to provide an objective analysis and technical comparison of both PRISM and
MARS. This comparative analysis attempts to evaluate the consistency of: A) the models, B) the data and
calculations and C) the standard output. Key findings are summarized as follows:
A. Comparing the Models:
PRISM
PRISM is a two-area simulation model.
PRISM uses a probabilistic distribution to model load.
© PJM Interconnection 2011. All rights reserved
Page 3 of 136
PRISM uses probabilistic distributions for capacity modeling (as does MARS).
PRISM uses deterministic distributions for transmission system modeling (as does
MARS).
PRISM uses statistical input data requirements. This is a distinct advantage as PRISM
has relatively few statistical parameters that incorporate the full model.
PRISM uses a probabilistic forecast load model.
PRISM features a comparatively fast solution time; this advantage thereby allows more
time for additional sensitivity cases to more fully assess system impacts.
Running PRISM requires no additional PJM staff to complete a resource adequacy
assessment.
Unlike MARS, PRISM cannot perform Hourly assessments or determine metrics regarding
either Loss-of-Load Hours (LOLH) or Expected Unserved Energy (EUE).
MARS
MARS is a multiple-area model; this is a tremendous advantage.
MARS uses a deterministic load distribution.
MARS uses probabilistic distributions for capacity modeling (as does PRISM).
MARS uses deterministic distributions for transmission system modeling (as does
PRISM).
MARS can perform hourly calculations and include more direct Operational parameters.
Compared to PRISM, data collection and maintenance of inputs for MARS is relatively
time-intensive.
Being a Monte Carlo simulator, MARS requires longer solution times.
MARS is ably supports and continuously refined and updated by GE technical staff.
MARS has gained industry-wide acceptance and usage throughout North America.
MARS will therefore, be used to comply with any forthcoming North American Reliability
Corporation (NERC) Planning Committee (NERC-PC) reporting for new metrics, LOLH
and EUE.
B. Comparing the Data & Calculations:
The significance of the mantra “It is all about the data!” cannot be overlooked in Loss of Load
Expectation (LOLE) assessment work: Having high-quality data allows enables greater confidence,
correct interpretation of reported results, appropriate decision making and high accuracy of final
reported values. PJM’s database management conforms to standards and processes governed by
third-party audits and assurance that Best Practices are used in the underlying data systems.
PJM uses the same main graphical user interface (GUI) and underlying database for both
PRISM and MARS. Both programs are included in the Applications for Reliability Calculations
(ARC) process used by the PJM staff. Many data relationships are established to assist
automating a consistent model between both tools. (See Table 3 for further details.)
MARS contains more data input categories than does PRISM (see Appendix F). This allows
MARS more flexibility and the potential to perform assessments that PRISM cannot perform.
© PJM Interconnection 2011. All rights reserved
Page 4 of 136
But it also requires increased coordination among PJM Staff and more time to maintain and
update the data.
While PJM Staff is able to perform most Adequacy work in-house, increased MARS efforts
would likely increase member technical representatives’ responsibilities and efforts. This
dynamic can be witnessed by comparing MARS assessment work that is done by neighboring
RTOs, ISOs and EROs.
C. Comparing the Output:
MARS contains several output summaries not found in PRISM, including: load level,
Emergency Operating Procedure level, and Interface flows.
PRISM uses a database schema both for its input and output results. This allows a mapping of
the relationships between these data and is defined in an OLAP metadata process that allows
assessment and reporting of several complex summaries.
Both PRISM and MARS offer many detailed outputs to perform LOLE assessments.
© PJM Interconnection 2011. All rights reserved
Page 5 of 136
Introduction
Resource capacity and reliability planning assessments are performed each year by PJM and its
neighboring RTOs, ISOs and EROs to ensure the reliability of the interregional bulk power system. These
comprehensive, time-intensive evaluations are critical to the planning process. Each region and reliability
entity that performs such resource adequacy studies does so with various tools and models at their disposal.
Modeling techniques generally utilize combinations of deterministic and probabilistic systems (such as
Monte Carlo simulation methods), depending on the modeling parameters, assumptions and inputs.
PJM Interconnection (PJM) currently uses the two-area Probabilistic Reliability Index Study Model (PRISM)
for reliability and resource adequacy modeling. Developed by PJM, PRISM uses cumulative probability
functions involving numerical techniques that combine (convolve) a probabilistic load distribution with a
probabilistic resource distribution.
Several of PJM’s neighbors, most notably the New York Independent System Operator (NYISO), the
Independent System Operator of New England (ISO-NE), the Midwest Independent System Operator
(MISO), and the SERC Reliability Corporation, use the General Electric International, Inc. (GEII) Multi-Area
Regional Simulator (MARS) program. MARS’ core calculations are based on sampling using the proven
Monte Carlo solution technique. MARS analyzes multi-area generation systems with explicit recognition of
unit and system operating considerations, rules, and constraints that influence system reliability indices.
MARS utilizes a random number generator to facilitate calculations with many replications performed to
satisfy a chosen solution criteria of standard error.
PJM’s Resource Adequacy Analysis Subcommittee (RAAS) reviews the modeling and analysis techniques
used in the reserve requirement annual study and Capacity Emergency Transfer Objective (CETO).
Reporting to PJM’s Planning Committee (PC), the RAAS charter also includes supporting capacity model
review. Several members expressed interest in using MARS as PJM’s primary modeling tool.
PJM’s use of MARS is increasing due to the capabilities inherent in a multi-area Monte Carlo solution tool.
These efforts address Industry topics such as Wind unit modeling, Demand Response (DR) assessments,
and collaboration with neighboring regions staffs in performing intraregional assessments.
While PRISM remains PJM’s primary resource adequacy evaluation tool, PJM is also licensed to use MARS
– and does so in the conduct of the overall Resource Requirements Study (RRS) report. In recent years,
PJM conducted comparative analysis of PRISM and MARS but until now, these efforts were not welldocumented. With the recent expansion of PJM’s regional footprint and increasing emphasis on
interregional planning activities, the RAAS determined it was worthwhile and timely to provide a more
comprehensive, fully-documented comparison.
The intent of this report is to provide comparative information of PRISM and MARS with respect to inputs,
assumptions, system requirements, required resources, outputs and results. This analysis strives to present
viewpoints useful for evaluating the technical aspects of the tools so that objective comparisons can be
made.
© PJM Interconnection 2011. All rights reserved
Page 6 of 136
Background
Reliability Criterion … 1 day in 10 years
Although it is difficult to determine when the first published material appeared describing reliability criterion,
the first notable group of papers on this topic appeared in 1947. In that year, several key papers were
presented, namely those by Calabrese1, Loane & Watchorn2, Lyman, Seelye and others. These documents
proposed the basic concepts upon which some of the methods in use at the present time are based.
Concepts such as the “loss of load method” and the “frequency and duration” approach were explored at
3
that time. Until that time, reserve capacity (planning reserve) was deterministically established to be in the
range of 18 to 24 percent.
In particular, the analysis by Loane & Watchorn used forced outage rates of generators in conjunction with
outage probabilities (valid under the assumption of exponential failure density and other approximations)
that resulted in a computed loss of load probability (probability of curtailment) of 0.32e-03.
The Calabrese and Watchorn papers, when taken together, presented similar ideas and philosophies. As
such, the resulting daily expectation of load loss can be combined, using a simple average, which yields an
approximate probability of curtailment of 0.4e-03.
Since using loads of working weekdays during the entire year in their model (~260 days/year), the
mathematical expectation of this risk became 0.4e-03 x 260 = 0.1 days per year … or 1 day in 10 years. As
other contemporary analyses eventually arrived at similar conclusions, the “1 in 10” evolved as an industry
standard for probabilistic bulk system planning, as the Loss of Load Expectation (LOLE).
Different load models gave different reserve margins, although they all used the same risk criterion, namely
one day in 10 years. There is considerable debate regarding risk level in technical journals and regulatory
proceedings. Some economists have long questioned the Bulk Electric System’s usage of the one day-inten year LOLE criterion. Notwithstanding such discussions, this risk criterion is ubiquitous and has gained
acceptance as an acceptable index for probabilistic resource adequacy planning. This acceptance ia also in
line with business and governmental officials’ expectations and requirements for the resource adequacy of
the bulk power grid.
From this historical backdrop, the bulk electric grid industry and related organizations have adopted the one
day (on average) in 10 years requirement – used for LOLE analysis in planning the Adequacy of the bulk
system.
The Northeast Power Coordinating Council (NPCC)4, the Mid-Continent Area Power Pool (MAPP)5 and the
6
previous NERC regions of the Mid-Atlantic Area Council (MAAC) and the Mid-America Interconnected
7
Network (MAIN) all conducted LOLE analysis to comply with this requirement. NPCC conducts LOLE
analysis to demonstrate its Member’s compliance with this requirement, as specified by the NPCC Bylaws.
1
Giuseppe Calabrese, “Generating Reserve Capacity Determined by the Probability Method” (March 25, 1947),
presented at the AIEE Midwest meeting of November 3, 1947 – see page 21 (which cites a 0.00046 probability of loss
of load)
2
E.S. Loane and C.W. Watchorn, “Probability Methods Applied to Generating Capacity Problems of a Combined Hydro
and Steam System” (August 14, 1947), presented at the AIEE Midwest meeting of November 3, 1947 – see page
1651 (which cites a 0.00032 probability of failure to carry peak daily load)
3
Roy Billinton and Ronald Allan, “Reliability Evaluation of Power Systems” (1984), Plenum Press, New York – page 7
4
NPCC Regional Reliability Reference Directory #1 “Design and Operation of the Bulk Power System (December 1,
2009) - section 5.2, page 9
5
MAPP Loss of Load Expectation (LOLE) Study, MAPPCOR and MAPP Composite System Reliability Working Group
(December 30, 2009); http://www.mapp.org/ReturnBinary.aspx?Params=584e5b5f405c567900000002cb
6
7
Mid-Atlantic Area Council (MAAC) Document A-1, MAAC Reliability Principles and Standards (March 30, 1990)
MAIN Guide #6, Generation Reliability Study 20045-2014 (September 27, 2005)
© PJM Interconnection 2011. All rights reserved
Page 7 of 136
Currently the ReliabilityFirst Corporation (RFC)8 has this criterion specified in their region’s Standard BAL502-RFC-02 as well as Midwest Reliability Organization (MRO)9 in their Standard RES-501-MRO-01.
Early Reserve Study Models: GEBGE, PTI’s MAREL and GE-MARL / GE-MARS
With the 1 day in 10 years LOLE reliability criterion having gained broad acceptance, engineers and
reliability professionals sought to develop simulation models. In the decades since, various modeling tools
were developed for one-area, two-area and multi-area applications. Although using the same risk criteria
these different models also yield different reserve margins 10.
One such early model to gain recognition and industry acceptance was the GEBGE. Developed by General
Electric International, Inc. (GEII) in conjunction with staff from Baltimore Gas & Electric (BGE), “GEBGE” is a
two-area, probabilistic reliability model. Starting sometime in the mid-1960s, GEBGE became the primary
reliability (R) model used by PJM – until replaced by the later-generation PRISM program. A 1969 paper by
Baldwin11 provides the general specifics for how GEBGE was created in line with the philosophies
expressed by both the Calabrese and Loane & Watchorn studies. Around this same time, the Westinghouse
two-area model also gained recognition and industry acceptance (used by ISO-NE Staff and is similar in
nature to GEBGE).
GEBGE uses a probabilistic load model based on a daily peak distribution, aggregated on a weekly basis,
derived from historic loads and a forecasted growth rate. The daily peak distribution is represented by a 21point normal curve. GEBGE’s capacity model uses each generating unit’s capacity, forced outage rate and
maintenance requirements to develop a cumulative capacity outage probability table for each week. On a
daily basis, GEBGE calculates probability of every possible load level simultaneous with every possible
generation availability level. Individual week LOLEs are summed over the entire year to determine the
annual LOLE. The GEBGE calculations and solutions techniques served as the basis in the development of
PJM’s PRISM model.
From 1988-1991, the former New York Power Pool (NYPP) Resource Planning Advisory Subcommittee
(RPAS) initiated a project to develop a multi-area computer program using two different technical solution
techniques: 1) the analytical method (convolution) and 2) the Monte Carlo method.
In response, power systems software developer Power Technologies Inc. (PTI) developed a prototype multiarea analytical method called the Multi-Area Reliability (MAREL) program. PTI’s approach imposed no
theoretical limitations on the number of areas or topology of the interconnected network. MAREL considered
various types of generation, transmission interfaces, firm interchanges (contracts) and emergency operating
procedures (EOPs).
GE, meanwhile, took up the Monte Carlo method and developed the Multi-Area Reliability (GE-MARL). GEMARL used a sequential Monte Carlo simulation for modeling the effects of random events such as forced
outages. GE-MARL considered various types of generation, transmission interfaces, firm interchanges
(contracts) and emergency operating procedures (EOPs). GE-MARL was later renamed GE-MARS (MultiArea Reliability Simulation) once the program went into production.
8
9
ReliabilityFirst Corporation (RFC) region planning resource Adequacy Analysis, Assessment and Documentation,
Standard BAL-502-RFC-02 (December 4, 2008): http://www.nerc.com/files/BAL-502-RFC-02.pdf
Midwest Reliability Organization (MRO) Standard RES-501-MRO-01, planned Resource Adequacy Assessment
(December 29, 2007); http://www.midwestreliability.org/04_standards/approved_standards/mro_standards/RES-501MRO-01_Final_20071229_Clean.pdf
10
11
“PJM Generation Adequacy Analysis: Technical methods” (October 2003) available at:
http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/res-adeq/20040621-white-papersections12.ashx.
C.J. Baldwin, “Probability Calculation of Generation Reserves” (March 1969), published by The Westinghouse
Engineer, This paper is copyright protected but can be purchased online at Infotrieve; article information accession
number 00434361 (800-422-4633; www.infotrieve.com).
© PJM Interconnection 2011. All rights reserved
Page 8 of 136
In 1991, ESEERCO evaluated the PTI-MAREL and GE-MARL programs.12 The NYPP ran various
simulations for single and multiple area and six-area regions. While MAREL and GE-MARL produced similar
results for the single and six-area representations, the GE-MARL program was able to solve the eleven and
fifteen-area simulations. NYPP therefore concluded that GE-MARL was the superior product.
Comparing GEBGE with MAREL
In 1993, the PJM Load and Capacity Working Group (L&CWG) conducted a comparison of the PTI-MAREL
application and GEBGE.13 Concerns over the possible shortcomings of the two-area model prompted this
comparison. From the outset, it was PJM’s intent that MAREL was not intended to replace GEBGE but
rather to complement GEBGE. The LCWG report made the following key observations and conclusions:
12
13
It was determined that the PJM Reserve Requirement should be calculated with GEBGE. The
multi-area MAREL results indicated that no accuracy was sacrificed by modeling a single-area
World as was done by GEBGE.
The benefit of modeling a multi-area World became apparent only when reserves were reduced
from their forecasted level of 26.6% down to 10%. Under this condition, load diversity and
transmission constraints within the World were judged to affect the reserve levels to meet a 1 day
in 10 years LOLE.
Since PJM installed capacity (ICAP) accounting is predicated on 1 day in 10.00 (hundredths
accuracy), the multi-area complexity of MAREL prevented it from obtaining a solution to this level of
accuracy.
Unlike GEBGE, MAREL lacked the capability to automatically adjust load levels to solve to a userspecified Reliability Index (RI).
LCWG recommended to purchase MAREL pointing out some compelling rationale:
o
It was thought that future conditions in the electric utility industry (such as increased wheeling
and Non-Utility Generation (NUG) activity) would require multi-area reliability assessments.
o
MAREL included several features not found in GEBGE, including the modeling of firm
contracts, demand side management and modeling of emergency operating procedures
(EOPs).
o
MAREL could be used (the only tool at that time) to verify GEBGE results.
o
Unlike GEBGE, MAREL calculated EUE in addition to LOLE.
Ultimately, PJM elected to not purchase MAREL. At some point thereafter, PTI discontinued further
development of the MAREL software.
NYPP Reliability Evaluation Task Force, “Multi-Area Generation Reliability – Phase I Testing Report” (October 22,
1991), ESEERCO Project EP87-28.
PJM Load & Capacity Working Group, “Evaluation of PTI’s Multi-Area Reliability Program MAREL” (November 1993)
© PJM Interconnection 2011. All rights reserved
Page 9 of 136
Current Reserve study Models
Two–Area Model: PJM PRISM
Developed by PJM, the Probabilistic Reliability Index Study Model (PRISM) is owned and maintained by
PJM – with PJM Members having the rights and ability to review its technical attributes. PRISM is the
primary modeling tool used for conducting PJM’s resource adequacy studies. PRISM is classified as a “twoarea” model that simulates the PJM RTO and areas adjacent to PJM’s footprint, called the “World”.
The modeling, core calculations, and underlying techniques used by PRISM are based on the GEBGE
model. PRISM was developed on a Web-based platform. GEBGE remains as a legacy system based on a
FORTRAN platform.
PRISM calculations use a cumulative probability function involving numerical techniques that combine
(convolve) a probabilistic load distribution with a probabilistic resource distribution. A given load distribution
point is “looked up” in the available generation array, the cumulative probability distribution table, resulting in
a loss-of-load value that – when greater than zero (10-11), is weighted by the appropriate probability for this
numerical load occurrence.
PRISM can model either a single region or two regions can be modeled together linked by a chosen tie size.
The interface tie size (or “pipe” size) represents aggregate transmission facilities capabilities at peak
conditions between the study regions.
PRISM requires a forecast probabilistic representation of the daily peak load and generation resources. The
tie size can be considered probabilistically, with PRISM enhancements needed to make this an automated
process. The numerical methods used to implement the LOLE calculations using the cumulative probability
distributions are considered not practical for more than six areas14.
Due to its use of statistical parameters and application of probability theory using numerical methods,
PRISM is considered a planning tool – and not well-suited for use in operational assessments.
The ISO-NE planning staff has used the similar, single-area Westinghouse Model for LOLE assessments in
the ISO-NE region. (Appendix C-3 includes a description of ISO-NE’s Westinghouse model.)
Probabilistic resource adequacy criteria require models to determine compliance with the criteria, while
deterministic criteria require simple event tracking and measuring. Loss-of-load expectation measures the
probability that the load will exceed the available generating capacity. Load, used in resource adequacy
assessments, can be modeled in several ways:
Model all hourly loads deterministically, or 8,760 hours per year. The resultant metric is termed
loss-of-load expectation (LOLE), and is expressed in either hours/year, or in days per year. A
days-per-year metric can be calculated with a Monte Carlo model. In such a model, a Monte Carlo
replication results in one of two outcomes: either load lost in a day or it is not lost. The sum of the
days per year that load is lost over all 8,760 hours is represented by LOLE in days per year.
Models that compute LOLE with all hourly loads, typically a deterministic set of values, can also
compute a system’s EUE which represents the summation of the expected amount of load lost for
each hour of the year.
Model only daily peak loads. Usually just week-day peak loads are modeled, or 260 hours per year
(52 weeks x 5 days per week x one peak hour per day).
All daily peak loads (365 days per year) can be modeled, if there is non-trivial risk on the
weekends. Assessments are needed to determine if weekday risk is appropriate for modeling the
entire week’s risk.
LOLE is related to the Loss of Load Probability (LOLP) but is different. The LOLE, a mathematical
expectation term, is explained further in reference number 22, showing the relationship of LOLE to
14
NYPP Reliability Evaluation Task Force, “Multi-Area Generation Reliability – Phase I Testing Report” (October 22,
1991), ESEERCO Project EP87-28 – See pages S4-S6.
© PJM Interconnection 2011. All rights reserved
Page 10 of 136
LOLP.15 However in several technical discussions these two terms are used interchangeably. Care
should be taken in comparing this paper’s use of these technical terms with a more general use in
other papers.
The term Loss of Load Hour (LOLH) is the probability that the peak load will exceed the capacity in
a single hour.
The Peak Load Ordered Time Series (PLOTS) load model is based on the hourly loads for the
regions in the Study, typically using about 12 years of history. Hourly loads are required for the
entire region. The historic hourly loads are transformed into a set of forecast, probabilistic values
having a weekly mean and standard deviation.16 These forecast, probabilistic values are used for
each week in the Study.
Multi-Area Model: GE-MARS
The Multi-Area Reliability Simulation (MARS) program is owned and licensed by the General Electric
International, Inc. (GEII), and is sometimes referred to as “GE-MARS”. MARS’ core calculations are based
on sampling using the proven Monte-Carlo technique. In this, a random number is used to seed the
calculations with many replications performed to satisfy a chosen solution criteria of standard error (based
on individual replications).
MARS performs a chronological hourly simulation of the system, comparing the hourly load demand in each
area to the total available generation in the area, which has been adjusted to account for planned
maintenance and randomly occurring forced outages. Typically a solution option is chosen to consider only
the daily peak load (Maximum Hourly integrated value). Areas with excess capacity will provide emergency
assistance to those areas that are deficient, subject to the transfer limits between the areas.
MARS utilizes many tables (50) for discrete entries involving many modeling specifics for various categories
including: general solutions, interfaces, loads, maintenance for both load and generation, modifications to
hourly load and generation shapes, generation resources, emergency operating levels, and reserve sharing
among areas.
MARS’ generation resource modeling can use either a transition state matrix or forced outage rate. The load
model uses a discrete 8,760 hourly load pattern, typically chosen by assessment of historic patterns. MARS
can calculate LOLE, LOLH and EUE. .
MARS modeling can also be utilized with operational assessments – but it is still primarily regarded as a
planning tool that can assess adherence to probabilistic criteria (outage event one day, on average, in 10
years)
MARS calculates, on an area and pool basis, the standard reliability indices of daily and hourly LOLE,
expected unserved energy and time-correlated indices such as frequency and duration of outage. The
program can also calculate the expected number of days per year that various emergency operating
procedures would need to be implemented – by those staff operating the grid.
Typical MARS applications include:
Resource adequacy assessments
Installed capacity (ICAP) requirements
Locational capacity requirements
Benefits of load diversity assessment
Benefits of reserve sharing
Tie-line effectiveness
15
Power System Reliability Evaluation, Mathematical Expectation - page 12, Gordon and Beach, Science Publishers, –
1970 – by Roy Billinton and Reinvent Legacy Software with SAS, the Web, and OLAP Reporting, Appendix B –2007 –
available at: http://www.qlx.com/whitepapers/25-2008.pdf .
16
Re-invent a Legacy System with SAS, the Web and OLAP reporting, November 13, 2007, 2007 Northeast SAS user
group conference – Applications Big and Small, Paper Number 10. http://www.lexjansen.com/nesug/ , Appendix:
Download the code (2369 KB).
© PJM Interconnection 2011. All rights reserved
Page 11 of 136
Expected need for implementing emergency operating procedures (EOPs)
Reliability impact and capacity value of intermittent or variable resources such as wind, hydro,
thermal and solar energy.
A sequential Monte Carlo simulation forms the basis for MARS software. The Monte Carlo method provides
a convenient input specifications, versatility, and easily expandable program that can be used to fully model
varied generation and reserve sharing options. MARS software makes the following reliability indices
available on both an isolated (zero ties between areas) and interconnected (using the input transfer limits
between areas) basis:
Daily LOLE (days/year)
Hourly LOLE (hours/year)
Loss of Energy Expectation (LOEE) (MWh/year)
Frequency of outage (outages/year)
Duration of outage (hours/outage)
Need for initiating Emergency Operating Procedures (EOPs, as days/year and hours/year)
PJM is licensed to use MARS and uses this program to enhance the analytical capabilities of PRISM. PJM
uses the MARS program for developing case sensitivities, coordinating study models and results with
neighboring regions, participating in interregional studies, and performing the Winter Weekly Reserve Target
analysis.
PRISM is used to calculate the LOLE of up to two interconnected systems with a single transfer link. PRISM
is used to calculate the Forecast Pool Requirement (FPR) and the Demand Resource (DR) factor (DR
Factor).
Comparative Strengths
The strengths of PRISM as a two-area convolution model include:
PRISM’s relatively fast speed of calculations
PRISM provides historic benchmarking with Operational experience for the PJM footprint
PRISM has reduced input requirements due to using statistical parameters
PRISM is consistent with PJM Agreements and Operational practices.
PRISM provides confidence in accuracy of technical results directly applied in the
Marketplace.
PRISM promotes efficiency and effectiveness of PJM user staff and PJM support staff.
Mitigates tedious repetitive tasks.
17
The strengths of MARS as a multi-area Monte-Carlo model include:
MARS can model Operation specific actions and events.
MARS can model many areas and incorporate different modeling for each area such as 1)
reserve sharing, 2) invocation of Emergency Operation events, 3) load patterns and
distributions, and 4) transmission path flow coordination.
MARS can determine energy related indices such as LOLH and EUE values.
MARS can model various impacts using hourly assessment calculations and methods,
such as intermittent resources, demand side management programs, and load shedding.
17
PJM Generation Adequacy Analysis: Technical Methods - October 2003 – developed by PJM Staff; available at:
http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/res-adeq/20040621-white-papersections12.ashx. - See Section 2 of this White Paper.
© PJM Interconnection 2011. All rights reserved
Page 12 of 136
Comparative Strengths
The strengths of PRISM as a two-area convolution model include:
PRISM’s relatively fast speed of calculations
PRISM provides historic benchmarking with Operational experience for the PJM footprint 18
PRISM has reduced input requirements due to using statistical parameters
PRISM is consistent with PJM Agreements and Operational practices.
PRISM provides confidence in accuracy of technical results directly applied in the
Marketplace.
PRISM promotes efficiency and effectiveness of PJM user staff and PJM support staff.
Mitigates tedious repetitive tasks.
The strengths of MARS as a multi-area Monte-Carlo model include:
MARS can model Operation specific actions and events.
MARS can model many areas and incorporate different modeling for each area such as 1)
reserve sharing, 2) invocation of Emergency Operation events, 3) load patterns and
distributions, and 4) transmission path flow coordination.
MARS can determine energy related indices such as LOLH and EUE values.
MARS can model various impacts using hourly assessment calculations and methods,
such as intermittent resources, demand side management programs, and load shedding.
18
PJM Generation Adequacy Analysis: Technical Methods - October 2003 – developed by PJM Staff; available at:
http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/res-adeq/20040621-white-papersections12.ashx. - See Section 2 of this White Paper.
© PJM Interconnection 2011. All rights reserved
Page 13 of 136
Complementary Aspects of PRISM and MARS
Throughout this report, the term “complementary” is used to characterize usage of both PRISM and MARS
in reliability studies. As was mentioned in the Background section, the 1993 assessment by PJM’s L&CWG
(the predecessor to RAAS) clearly stated its intention to use a multi-area tool in concert with a two-area
convolution tool. Since that time, the PJM Agreements, Operational procedures and LOLE technical
methods have not significantly changed to alter that over-arching philosophy.
Below is a recap of “complementary” tasks that each program provides to the other:
Complementary tasks that PRISM provides to MARS:
1.
Since MARS can require significant time to reach a result, PRISM can be used to quickly
determine LOLE results. This enables more time for fine-tuning and other analytical work.
2.
The variability of MARS’ Monte Carlo calculation results can be minimized but not eliminated19.
Conversely, PRISM’s non-variable, direct table lookup method can be used to verify the
MARS’ results.
The PJM Agreements and Operational procedures currently in place reflect PRISM results –
and does not require event-driven detailed, operational and transmission values (as required
by MARS). PRISM’s use in planning assessments can reduce the resources required to reach
the desired result (i.e. FPR, DR Factor and CETO values in load deliverability tests).
3.
4.
MARS results are based on operational input data parameters. PRISM can help evaluate,
measure, and assist coordination with operational experience and MARS input parameters.20
5.
The MARS load model parameters are critical in CETO assessment work. Since PRISM uses
a forecast probabilistic load model, it can assist MARS in evaluating the choice in: 1)
deterministic load model parameters, 2) distributions used in the MARS Table, 3) in02 file, 4)
LOD-UNCY and 5) LOD-MTAR. These deterministic distributions, especially for smaller PJM
RTO sub regions, have been identified as additional development efforts for any MARS CETO
assessment work.
6.
PRISM can independently verify and help measure the MARS results. This can be done by
evaluating the solutions on a very detailed basis, and at every solution point used (See
Appendix E).
7.
MARS is the tool used by our neighboring regions and control areas. PRISM can assist these
interregional study efforts by translating MARS modeling and results into the parameters used
in the PJM Planning division assessments.
19
Refer to NYPP Reliability Evaluation Task Force - Phase I Testing Report (October 22, 1991) – See page S-8.
PJM Generation Adequacy Analysis: Technical Methods - October 2003 – developed by PJM Staff; available at:
http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/res-adeq/20040621-white-papersections12.ashx. - See Section 2 of this White Paper.
20
© PJM Interconnection 2011. All rights reserved
Page 14 of 136
Complementary tasks that MARS provides to PRISM:
1.
MARS can assess the single-area World model used in PRISM. (It is important to note that
single-area modeling was tested and evaluated to be appropriate, in the comparing GEBGE
with MAREL section.21)
2.
MARS can assist assessment of intermittent resources such as Wind, Solar and Hydro due to
its various tables and hourly patterns. Such assessments can also include Demand
Resources (DRs). More development work would be needed for these efforts.
3.
Operational issues such as transmission constraints, reserve sharing events, and assessment
of load management events can be performed.
4.
MARS can provide weekly distributions and resulting LOLE for the winter period that is
consistent with the annual PRISM patterns. This function can be used in setting the Winter
Weekly Reserve Target used by the Operation division staff for scheduling unit planned
outages over the 13-week winter period (December through February).
5.
MARS can independently verify and benchmark PRISM results. This can be done by
evaluating the solutions on a very detailed basis, and at every solution point used. See
Appendix E.
6.
MARS is used extensively by PJM’s neighboring regions. Using MARS therefore helps
facilitate interregional coordination and planning efforts.
7.
MARS can assist in evaluation of the impact of high ambient conditions on generation
resources by using its UNT-DERT table.
8.
MARS can be used to comply with any new, forth coming requirements, per the NERC
Planning Committee directives. MARS can report the identified new metrics, LOLH and EUE.
In reporting for EUE, the forecast energy, per the annual PJM load forecast report and other
specifics will need attention. These two metrics have been historically not used and reported
within the electric Industry for bulk power systems.
Ongoing LOLE Assessment Work
Reserve Requirement Study (RRS) – PJM Staff annually conducts a Reserve Requirement Study
(RRS). These efforts involve seven or more staff and several professional disciplines for a
successful completion. These efforts are required to:
o
Establish the needed Reliability Pricing Model (RPM) Market parameters: the FPR,
and associated Demand Resource DR Factor (DR Factor)
o
Perform load deliverability tests for PJM’s Regional Transmission Expansion Plan
(RTEP)
o
Demonstrate compliance with RFC Standard BAL-502-RFC-02.
The RRS also determines the Winter Weekly reserve target used in Operations. These
assessment activities serve to coordinate efforts between the Planning, Operations and Market
Divisions. The efforts for the RRS include:
o
21
A data completion effort that is developed for each study showing resources,
estimated hours, duration, and target completion dates (Using a tracking
spreadsheet).
See the related discussion in the “Evaluation of PTI’s Multi-Area Reliability Program (MAREL)” by the Load and
Capacity Working Group (November 1993).
© PJM Interconnection 2011. All rights reserved
Page 15 of 136
22
o
A quality data source is established before any analysis can be started. The data
sources involve both the internal and external regions in the model, have more than
20 specifics tasks, involve the LOLE modeling Team, the load forecasting Team, and
the GADS administrators.
o
PJM’s Information Technology Services (SAS, Oracle, Web (Java), Database
Administrators, and Network systems) and Operations staff assist supporting
compilation of the generation model with involvement by the Generator Owners to
review their units’ data.
o
These efforts can take approximately 480 Full Time Equivalent (FTE) hours to
complete and typically span over three to four months. Experienced PJM Resource
Adequacy and Planning Department staff oversees and perform the primary review of
these efforts.
LOLE Analysis – The RRS analysis to be performed, after the data has been finalized, is
discussed and assignments agreed to among the LOLE analysis Team.
o
PJM’s Resource Adequacy Planning (RAP) Department staff performs these studies,
coordinating with other Planning, Markets and Operation Division staff as results and
observations become apparent.
o
The RAAS and PC are part of this initial review. The intent is to quickly identify any
issues that are seen as significant drivers so that a thorough assessment and
reporting can be made.
o
Typically there are over fifty separate assessment items involved in the analysis,
resulting in a one hundred page report (see 2010 RRS reference).
o
These analysis efforts and related reporting can take approximately 680 hours and
span over three to four months, primarily performed by four RAP staff members.
Coordination and review with the RAAS and PC are important steps in this process.
Capacity Emergency Transfer Objective (CETO) Assessments - Related analysis efforts also
involved CETO assessments for the 24 Locational Deliverability Areas (LDAs)22.
o
CETO assessments use a given RRS model as the basis for its assessment work.
These efforts involve the LOLE Team, Load Analysis Team (LAT), Transmission
Planning Staff, and Generation Interconnection Queue staffs.
o
The CETO efforts are performed for two primary efforts: 1) To address the
requirements of the RPM Market Place and 2) To address the needs of the RTEP
process overseen by the PJM Transmission Expansion Advisory Committee (TEAC)
stakeholder group.
o
The amount of CETO related assessment work has increased six fold in recent years
and continues to increase.
o
The CETO assessments currently are performed during the entire calendar year, with
concentrated efforts in the December to February time periods, and recently the June
to August time periods.
o
There are at minimum 96 required CETO assessments in each year, for various
delivery years. There can also be ad-hoc assessment work based on generation
termination requests and state commission inquiries.
o
The approximate total efforts are 730 Hours, for the LOLE Team, but other
department staffs are affected as well. The coordination with the other departments
and divisions is critical in the LOLE Team’s ability to perform the required
As discussed in the Reliability assurance Agreement (RAA), Schedule 15 and listed in PJM Manual 14B, Study Area
definitions – Zonal and Global (pages 49-50).
© PJM Interconnection 2011. All rights reserved
Page 16 of 136
assessments with high data quality and organization being a significant impact in the
process.
The Resource and Cost Assessment section of this paper provides further details concerning the LOLE
assessment efforts.
Model Calculation Processes
PRISM Calculation Processes
The calculations used in PRISM use forecast statistical parameters for both the load and generation
distributions. PRISM simulates “the joining” of a given load distribution point with the associated generation
distribution point (Click here for joining calculation reference). This joining is performed by a “lookup” table
approach using the cumulate probability (Cum Prob) distribution of the generation distribution.
The distributions used are both forecast and probabilistic. The first step is to build the cumulative probability
23
array, based on the individual generation unit forced outage rates. This calculation is widely documented .
The creation of this Cum Prob table is the most calculation-intensive aspect of the method used in PRISM,
done for each week in the model. The table is created using the binominal expansion method11. Once the
Cum Prob distribution pattern is determined for the generation model, the load model distribution is used to
look up the associated Loss-of-Load Probability (LOLP). This lookup is when the actual load distribution is
“convolved” with the generation distribution.
A graphical depiction of the numerical look-up method is shown in Figure 1, as an illustration only (again the
“joining” reference). The red bars of Figure 1 depict when there is risk of load exceeding the available
generation; this results in a loss-of-load event with an associated probability. Each load level has a defined
probability of occurrence and the red bars occur at significantly high loads, but with a low probability of
occurrence.
In the context of PJM Adequacy calculations the term “convolution” is used to depict the relationship
between two input distributions of load and capacity and the resulting loss of load distribution. In terms of
formal Mathematics, “convolution” (as used in Digital Signal Processing) does not have the same meaning
as convolution used in this report. However, science and engineering problems approach this single concept
from two different directions: 1) by considering a system in terms of what its impulse response looks like or
2) considering the system as a set of weighting coefficients. Familiarity with both views allows one to toggle
between directions one and two24.
Convolution plays a major role in both the capacity table approach and the load approach. The convolution
method illustrated for both approaches is called the recursive method.25 In this paper convolution is used to
convey to a technical audience, but yet non-mathematician audience, the relationship between two input
distributions of load and capacity and the resulting loss of load distribution. The load model derived from the
hourly load (Daily Peak) curve is thereby “convolved” (joined) with the generation system model for
computing the LOLE26.
23
Refer to Roy Billinton’s book, “Power System Reliability Evaluation”, Gordon and Beach, Science Publishers, or a
simple example shown in Appendix B of the Paper titled “Reinvent Legacy Software with SAS, the Web, and OLAP
Reporting” available at the following link: http://www.qlx.com/whitepapers/25-2008.pdf .
24
The Scientist and Engineer's Guide to Digital Signal Processing , By Steven W. Smith, PhD, copyright © 1997-2006
by California Technical Publishing – http://www.dspguide.com/pdfbook.htm
25
Dr. James McCalley’s course notes, module PE.PASU19.5 on Generation adequacy evaluation, Convolution
techniques, item U19.7.3 on page 43. http://www.ee.iastate.edu/~jdm/ee653/ee653schedule.htm
26
Dr. Chanan Singh, course notes: Electrical Power System Reliability, part3 Discrete Convolution Method, page 30,
copyright 1995, http://www.ece.tamu.edu/People/bios/singh/coursenotes/part3.pdf
© PJM Interconnection 2011. All rights reserved
Page 17 of 136
Figure 1 – Load distribution and Cumulative Probability Capacity Distribution
depicting PRISM calculations
Illustration of PJM RTO Convolution - Daily (during Peak Week)
Load Probability & Load LOLE
0.18
1.00
14
0.90
13
15
0.80
0.14
12
0.12
0.70
16
0.60
0.1
0.50
11
0.08
17
0.40
0.06
0.30
10
18
0.04
Cumulative Probability of Generatioon Availability
Daily Peak Load Probability of Occurance
0.16
0.20
09
0.02
19
0.10
08
01
02
03
04
06
05
20
07
21
0
Megawatts ( Load and Generation ) Zero to
Daily Load Probability
Load LOLE
22
23
24
25
26
0.00
∞
Generation Cumumulative Probability (Right Y Axis)
Figure 1 shows that as the load (in megawatts) increases, there is higher likelihood for loss of load (i.e. loss
of load occurs at much higher values than the mean). For each level of probable load (shown as the blue
bars) there is an associated cumulative probability of available generation (black dashed line). The
calculations mirror a continuous distribution (probable load), approximated by the blue bars.
Figure 2 – Extreme high loads: Detail of green oval
Illustration of PJM RTO Convolution - Daily (during Peak Week)
Detail: Load Probability & Load LOLE for "Extreme High" Loads
0.20
0.18
Daily Peak Load Probability of Occurance
0.16
0.14
19
0.015
0.12
0.10
0.01
0.08
20
Bars 21 - 26 reflect Equal
values for LOLE and Load
Probability.
0.06
18
0.005
0.04
21
15
16
0
0.02
22
17
23
24
25
Megawatts ( Load and Generation ) Zero to ∞
Daily Load Probability
Load LOLE
© PJM Interconnection 2011. All rights reserved
Generation Cumumulative Probability (Right Y Axis)
Page 18 of 136
26
0.00
Cumulative Probability of Generatioon Availability
0.02
Figures 1 and 2 are for illustration, but 21 points are used in the calculations. The number of points used
was due to practical considerations of speed and accuracy. Therefore twenty one points are used for each
daily peak lookup.
The red bars indicate when a loss of load state (LOLE) occurs –when the load is excessively large (which
rarely occurs and shown in green oval –see bars 19 and beyond). The cumulative probability of available
generation is low at these excessively large loads. The daily peak load probability of occurrence scale is
shown on the left Y axis. Example calculations used to determine the load model lookup value (into the
cumulative probability array), is shown in Table 1.
For the extremely high-load levels encircled by the green oval, the red LOLE bars increase because of the
higher cumulative probability of unavailable generation –at least until a certain higher load value is reached.
Figure 2 shows a magnified view of the green oval area, extreme load that have LOLE states. After
reaching an LOLE peak (in 19th bar from left), the red bars taper off due to the diminishing likelihood of
higher loads. (I.e. Such very high loads do not often occur while the generation probability (of unavailability)
saturates at a value of 1.0 –generation probability (of availability – dashed line) declines as megawatts
increase.
As generation unavailability saturates (approaching a value of 1), generation resources will not be able to
serve that load level (generation availability approaches zero). Even though there is a much greater risk of
generation unavailability as load increases, the LOLE is reduced because of the very small chance of that
higher load occurring.
It is important to note that the blue bars of this graph include PJM RTO loads up approximately 200,000
27
MW. There is a higher risk of generation unavailability (until saturation when a total generation is less than
load) as the load increases, but a lower likelihood that a higher load will occur.
Figure 3’s probability of per-unit load column (last column on right) correlates with the left Y axis in Figure 1
(Probability of occurrence). Figure 3’s Daily Peak Load distribution (MWs) column is analogous to the X
axis of Figure 1. Figure 3’s values are used to perform a table look-up into the generation’s cumulative
probability array to determine the probability of the available generation. This probability of generation
unavailability (reciprocal of availability –right Y axis), is multiplied by that load’s probability of occurrence (left
Y axis) to calculate the final LOLE for that day’s load.
For a given daily peak in a given week, using two summer months’ peak week (July and August) for these
examples, the following data can be derived as Load Model Lookup Values (Figure 3A & 3B):
Figure 3’s 21-point calculated load values use the equation shown at the bottom of Figure 3. Each Daily
Peak Load distribution value is used to “lookup” the associated cumulative probability in the generation
distribution (i.e. the load is thus “joined” with the generation). By multiplying the generation’s cumulative
probability value by the associated load probability the LOLE value is determined. If an LOLE value is less
than 10 -11 it is assumed to be zero. The daily LOLE, from the 21 points (points 4 through 24 in Figures 3A &
3B), are summed for all weeks in a delivery year to determine the annual LOLE.
The calculation of the available generation cumulative probability is performed using a binomial expansion
function. An example of the binomial expansion is shown on the second page of reference number 12 , and
Appendix B of reference number 15 .
27
PJM 2010 Load Forecast Report shows a 50/50 peak for the 2014 forecast delivery year of
163,093 MW.
© PJM Interconnection 2011. All rights reserved
Page 19 of 136
Figure 3A – Load Model Lookup Values for Daily Peak distribution- JULY
Key Parameters:
Final Week Peak Frequency (WKPKFQ) Mean =
1.0000
Final WKPKFQ Standard Deviation =
Total Sigma =
Forecast Error Factor (FEF) =
0.0703
0.0710
0.01
Daily EWM (July Peak Week) =
Daily Per-Unit EWM -of annual peak (July Peak Week) =
Daily Per-Unit Mean (on annual peak basis -Bar # 14) of EWM =
1.08173
1.000000
0.924448
Annual Peak Load (MW) =
EWM (for each day) =
Daily Mean Load (Bar # 14) =
163,093
1.000000
0.924448
1.0000
MW
163,093
150,771
EWM = Expected Weekly Maximum : Used for Daily Peak in that week.
Bar #
Probability of
Per-Unit Load
(given)*
# Sigma
from Mean
(given)
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
0.000033
0.000145
0.000638
0.002351
0.007273
0.018940
0.041400
0.076080
0.117490
0.152480
0.166340
0.152480
0.117490
0.076080
0.041400
0.018940
0.007273
0.002351
0.000638
0.000145
0.000033
-4.20
-3.78
-3.36
-2.94
-2.52
-2.10
-1.68
-1.26
-0.84
-0.42
0.00
0.42
0.84
1.26
1.68
2.10
2.52
2.94
3.36
3.78
4.20
Annual
Daily Per-Unit
Peak Load
Mean of EWM
(MW)
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
0.924448
<= Original
WKPKFQ value
Total
Sigma
Per-Unit Load
Daily Peak Load
Distribution (MW)
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.0710
0.648844082
0.676404499
0.703964917
0.731525334
0.759085751
0.786646168
0.814206585
0.841767003
0.869327420
0.896887837
0.924448254
0.952008671
0.979569089
1.007129506
1.034689923
1.062250340
1.089810758
1.117371175
1.144931592
1.172492009
1.200052426
105,811
110,307
114,803
119,299
123,795
128,291
132,787
137,283
141,779
146,275
150,771
155,267
159,763
164,259
168,755
173,251
177,747
182,243
186,739
191,235
195,731
Sigma from Mean Load Calculation:
Daily LOAD = Annual Peak Load (MW) x Per-Unit Mean EWM x (1 + (# Sigma from Mean) x Total Sigma)
Per-Unit Load = (Mean of EWM * (1 + (# of Sigma from Mean) X Total Sigma)
* Probability of Per-Unit Load values are given and taken from the 21-point Standard Normal distribution table.
The values in this example (Figure 3A & 3B) are the final adjusted mean and standard deviation
values, after adjustment to match the forecast monthly peak shape. The fifteen step calculation
process for the original WKPKFQ mean and standard deviation values is shown in Appendix A of
reference number 15.
© PJM Interconnection 2011. All rights reserved
Page 20 of 136
Figure 3B – Load Model Lookup Values for Daily Peak distribution- AUGUST
Key Parameters:
Final Week Peak Frequency (WKPKFQ) Mean =
0.9529
Final WKPKFQ Standard Deviation =
Total Sigma =
Forecast Error Factor (FEF) =
0.0719
0.0726
0.01
Daily EWM (August Peak Week) =
Daily Per-Unit EWM -of annual peak (August Peak Week) =
Daily Per-Unit Mean (on annual peak basis -Bar # 14) of EWM =
1.03250
0.954494
0.880876
Annual Peak Load (MW) =
EWM (for each day) =
Daily Mean Load (Bar # 14) =
163,093
0.954494
0.880876
0.968141
MW
155,671
143,665
EWM = Expected Weekly Maximum : Used for Daily Peak in that week.
Bar #
Probability of
Per-Unit Load
(given)*
# Sigma
from Mean
(given)
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
0.000033
0.000145
0.000638
0.002351
0.007273
0.018940
0.041400
0.076080
0.117490
0.152480
0.166340
0.152480
0.117490
0.076080
0.041400
0.018940
0.007273
0.002351
0.000638
0.000145
0.000033
-4.20
-3.78
-3.36
-2.94
-2.52
-2.10
-1.68
-1.26
-0.84
-0.42
0.00
0.42
0.84
1.26
1.68
2.10
2.52
2.94
3.36
3.78
4.20
Annual
Daily Per-Unit
Peak Load
Mean of EWM
(MW)
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
163,093
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
0.880876
<= Original
WKPKFQ value
Total
Sigma
Per-Unit Load
Daily Peak Load
Distribution (MW)
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.0726
0.612444339
0.639287529
0.666130718
0.692973908
0.719817097
0.746660287
0.773503476
0.800346666
0.827189856
0.854033045
0.880876235
0.907719424
0.934562614
0.961405803
0.988248993
1.015092182
1.041935372
1.068778562
1.095621751
1.122464941
1.149308130
99,858
104,239
108,620
113,000
117,381
121,762
126,142
130,523
134,903
139,284
143,665
148,045
152,426
156,807
161,187
165,568
169,949
174,329
178,710
183,090
187,471
Sigma from Mean Load Calculation:
Daily LOAD = Annual Peak Load (MW) x Per-Unit Mean EWM x (1 + (# Sigma from Mean) x Total Sigma)
Per-Unit Load = (Mean of EWM * (1 + (# of Sigma from Mean) X Total Sigma)
* Probability of Per-Unit Load values are given and taken from the 21-point Standard Normal distribution table.
The final week peak frequency (WKPKFQ) mean and standard deviation values are determined
in a separate process. These final values are an input into the calculation example shown in
Figure 3. Part of the determination of these weekly values, is an adjustment to match a
forecasted monthly peak shape, given in the PJM load forecast report. The original WKPKFQ
mean and standard deviation values28 are adjusted so that the final WKPKFQ monthly peaks
match the forecast monthly peak shape. The 52 week values, for the non monthly peaks, keep
their original ratio relative to their adjusted monthly peak. The final and original WKPKFQ values
are shown at the top of Figure 3A for comparison purposes.
28
2010 PJM Reserve Requirement study (RRS), Table II-2. Week number 12 is the August Peak.
Week number 10 is the July peak (annual peak).
© PJM Interconnection 2011. All rights reserved
Page 21 of 136
The Total Sigma calculation from PJM Manual 20 is:
Total Sigma = (Standard Deviation 2 + Forecast Error Factor 2)0.5
The process for the calculations to determine an IRM for compliance to the Standard, are shown in Figure 4.
PRISM uses a proxy for adjusting the capacity distribution, assuming that an adjustment in the load level is a
good proxy for individual unit adjustments to the capacity. Assessment in the NPCC CP-8 WG demonstrated
that adjustments to load gave almost identical changes as equal adjustments to capacity.29
The load shape is adjusted for the entire deliver year in this process until an LOLE (sum of individual daily
peak LOLPs) is at a Reliability Index (RI) of 10 (RI = 1 / LOLE).
Figure 4 – Installed Reserve Margin (IRM) Weekly Load Profile
LOLE Studies
Installed Reserve Margin
The Peak Load line is statistically manipulated until the
1-day-in-10-years criterion is met.
3/28/07
©2007 PJM
©2008 PJM
29
www.pjm.com
41
1
Northeast Power Coordinating Council Tie Benefits Methodology, by Glenn Haringa and Philip Fedora, November 5-6,
2008, Best LOLE Practices meeting held at California ISO offices, Agenda item 8. See slide 12, last bullet.
© PJM Interconnection 2011. All rights reserved
Page 22 of 136
MARS Calculation Processes
The publicly available solution method and techniques are summarized in Appendix D, as documented by
GE Power Systems. The solution information is given further details from the complete information in
Appendix D – and is based on the RAAS drafting team’s interpretation, observation and feedback from GE
Power Systems Staff, and participation in other LOLE Industry groups’ efforts (See Interregional
Assessments section).
MARS is a Multi-area generation system reliability model based on sequential Monte Carlo simulation. It was
developed in the late 1980’s by a project initiated by the New York Power Pool (NYPP) (refer to the Early
Reserve Study Models section of this report.) MARS uses a transportation algorithm to model flows
between areas. A transportation model requires separate assessment of the interconnected facilities and
operational practices to determine the interface transfer (pipe) size limits between the areas in the model.
The reliability calculations are performed in MARS by determining the capacity available each hour from
each unit on the system based on three primary model characteristics:
1.
2.
3.
Unit rating and capacity states
Scheduled planned outages
Random forced outages
The objective is to determine the area margins, details of which can be listed in output file 10, for each hour
which can be expressed as:
Margin = Capacity - Load
This algorithm’s solution order is:
1)
2)
3)
4)
5)
Accumulate statistics for isolated indices
Calculate flows between areas (if needed) and resulting area margins
Accumulate statistics for interconnected indices
Proceed to the next hour
Proceed to next replication
A further overview of the algorithm’s order for solution is shown in Figures 5A, 5B and 5C, a flow chart of the
MARS processing of the Master Input File (MIF). Note that the typically solution option for large system
studies in step 4, is to proceed to the next daily peak.
The solution step to calculate emergency assistance and resulting area margins, in Figure 5c, in the solution
process is the most computational intensive step. If dynamic interface limits are specified they are evaluated
in this step with a potential to significantly increase the solution time.
The MARS solution process is performed as shown in Figures 5A - 5C, on a given model basis. Typically a
chosen seed number is given for the solution results to be repeatable for auditing and verification purposes.
In this system- wide solution for large regional models, the default is to share assistance between areas on
an equitable basis. This approach mitigates individual areas’ LOLE states, without priority. This avoids
deciding which areas have priority over others in receiving assistance. MARS allows specifying a priority list
for allocating known assistance priorities among the areas that comprise the large regional model (Table
RES- PRIO).
The solution is unique for each model, based on the Master Input File’s Table entries and specifications.
Figures 8a- 8C show how the resulting Standard Error (SE) reduces by running more replications. The LOLE
metric result, for any specified Margin state, is consistent with how the SE improves with more replications.
The right (secondary) axis on Figures 8A-8C show the LOLE metric after replications of 1, 50, 100, 200, 500,
1000, 1500, and 2000 –demonstrating how it “converges” to a final unique solution.
© PJM Interconnection 2011. All rights reserved
Page 23 of 136
Figure 5A – Overview MARS solution
method
© PJM Interconnection 2011. All rights reserved
Figure 5B – Monthly
data preparation loop in solution method
Page 24 of 136
Figure 5C – Overview of Monte Carlo simulation
The Monte Carlo simulations usually specify a “random number seed” to start the solution process – with
defined convergence criteria for the Standard Error. The seed number allows for repeating the exact solution
in future sensitivity assessments.
The Standard Error is the average difference of all simulations performed. By making many simulations, this
solution method tends to saturate at a given level of Standard Error. If the given level of Standard Error is at
or below the convergence criteria, the simulation is deemed fully solved and all output reporting is generated
for the simulations.
A graphical depiction of the solution process is shown in Figures 5A-5C with an example resulting
Probability Density Function (PDF) of this process shown in two diagrams in Figure 6A and 6B. The
Standard Error of the individual replication results, sum of square average from mean, is shown in Figures
8A-8C.
© PJM Interconnection 2011. All rights reserved
Page 25 of 136
Figure 6A – Sample Probability Density Function (PDF)
Figure 6B – Example PDF and associated Cumulative Density Function (CDF)
Histogram
100
120.00%
90
100.00%
80
Frequency
70
80.00%
60
50
60.00%
40
40.00%
30
20
20.00%
10
0
0.00%
Bin
Frequency
Cumulative %
Figures 6A and 6B depict the PDF of the model in the Monte Carlo Simulations. They are a function of a
continuous variable, the generation outage set, such that the integral of the function over a specific region
yields the probability that its value will fall within the region.
© PJM Interconnection 2011. All rights reserved
Page 26 of 136
MARS models and simulates Load Forecast Uncertainty (LFU) by use of the LOD-UNCY table. This table
allows for each daily peak, 7 peaks each week, and 10 values to represent the distribution of the daily peaks
so that all possible loads, with the applicable probability of occurrence, assessed in the calculations. This
allows the assignment of probabilistic characteristic to the 8760 deterministic hourly load values defined in
the in02 file.
Figure 7 shows an example of this LFU and the associated calculated LOLE. Industry groups have identified
that the high probability risk, for models in North America, are in the high load values that have a low
probability of occurrence. (See Appendix C3 for further details concerning discussion of this topic.)
Figure 7 – Impact on Load Forecast Uncertainty (LFU) on LOLE
Figures 8A through 8C show the Standard Error (SE) achieved over a 2000 replications. These results are
from an Inter-regional Industry model that had over 25 areas, which included dynamic interface transfer
limits, thermal unit capacity derations, and seven load forecast uncertainty probability values, and six
Emergency Operation Procedure levels.
© PJM Interconnection 2011. All rights reserved
Page 27 of 136
Figure 8A – Standard Error (SE) of Monte Carlo Simulation
0.500
0.006
0.450
Standard Error (SE)
0.350
0.004
Trial 38
0.300
Trial 46
Trial 70
0.250
0.003
Tolerance
Trial 74
0.200
Trial 45
LOLE-Trial 38
0.002
0.150
0.100
Loss-of-Load Expectation (LOLE)
0.005
0.400
0.001
0.050
0.000
0.000
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Number of Replications
The vertical axis(Y axis) is the standard error while the horizontal (X axis) is the number of replications.
Figure 8B – Standard Error of Monte Carlo Simulation – Low SE
0.09
1.800
0.085
0.08
1.750
0.07
Standard Error (SE)
0.065
0.06
0.055
1.700
Trial 51
0.05
Trial 52
0.045
Trial 72
0.04
Trial 75
0.035
Trial 76
1.650
Tolerance
0.03
LOLE -Trial 75
0.025
0.02
1.600
0.015
0.01
0.005
0
1.550
0
200
400
600
800
1000
1200
Number of Replications
© PJM Interconnection 2011. All rights reserved
Page 28 of 136
1400
1600
1800
2000
Loss-of-Load Expectation (LOLE)
0.075
Figure 8C – Standard Error of Monte Carlo Simulation – High SE
1
0.001
0.001
Standard Error (SE)
0.8
0.7
0.001
0.6
Trial 47
LOLE
0.5
0.001
0.4
0.000
0.3
0.2
0.000
Loss-of-Load Expectation (LOLE)
0.9
0.1
0
0.000
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Number of Replications
The Standard Error from the mean is defined as the standard deviation of the distribution of sample
averages. From the Central Limit Theorem it is known that sample averages are approximately normally
distributed. The Mean of the normal distribution of sample averages is µ, which is also the mean of the
population. The standard deviation of the distribution of sample averages is [σ/n]1/2, where σ
is the standard deviation of the population. All iterations are compared to this average mean value, with the
mean changing as the simulations are performed until the Standard Error value is at or below the define
criteria. MARS defines a default Standard Error value of 0.05 with some Industry studies using a value of
0.025.
Note that even after 2000 replications, the Standard Error is still on a trend downward for Figure 8C. In this
model there were 76 various trials or area summaries that could be assessed concerning the solution criteria
for the best value to use for Standard Error. There can be a large variation between the various areas’
Standard Error, compare the standard Error values between Figures 8A-8C. This indicates that proper
assessment and Engineering judgment needs to be used to specify the solution criteria and to choose which
area should be used to define a successful final solution.
It is evident from observing Figures 8A – 8C that there is a “law of diminishing returns” for running many
MARS replications. The accuracy of the LOLE results may not be improved by performing thousands of
replications –in this case more than 2000 replications. Engineering judgment and experience must be used
to choose the best trade off of computer resources, resource cost and accuracy of results.
The right (secondary) axis on Figures 8A-8C show the LOLE metric after replications of 1, 50, 100, 200, 500,
1000, 1500, and 2000 –demonstrating how it “converges” to a final unique solution. An orange line was
generated, for the LOLE values of Figures 8A-8c, between these points for illustration purposes only.
© PJM Interconnection 2011. All rights reserved
Page 29 of 136
Figure 9 – Confidence Interval
Figure 9 is a modified version of a Confidence Interval graph that regularly appears in the NYSRC’s Annual
IRM Study30. (This graph represents a Standard Error of 0.025 occurring after 2,173 iterations.)
The Monte-Carlo Simulations can be summarized by Figures 8A-8C. This is based on a given Standard
Error to define the convergence criteria and typically requires many thousands of replications to reach a
solution. Figure 9 depicts the Confidence Interval of the thousands of replications performed. Typically
LOLE results are measured against a probabilistic measure, in this case 0.1 outage/delivery year. This
outage rate is reported at a defined Emergency Outage Procedure level.
The 0.1 outage per delivery year is viewed as a practical equivalent of the value of 1 loss of load event, on
average, every 10 years. It comes from a defined Industry Standard (BAL-502-RFC-02) that is viewed as a
requirement to demonstrate compliance. Further details of the source of this 1 day in ten years are
discussed in the background section of this paper.
An advantage of this calculation approach is that by changing the assumptions used for the discrete event
driven parameters, in the various input tables, one can observe the change in the confidence level and
LOLE results. The confidence level for simulations is generally a known technical measure and understood
for reporting purposes. The many replications are needed as the mix of generation resources are varied in
each replication, to value the loss of load probability for the value of load chosen in the input tables (LODDATA, LOD-MTAR, LOD-UNCY). These replications are a function of the continuous variable, the
generation outage set. The deterministic nature of the load necessitates the use of the Monte-Carlo
simulation method to choose the generation outage set, iterating into the solution.
30
NYSRC-ICS; “New York Control Area Installed Capacity Requirements for the Period May 2010 through April 2011 Technical Study” Report (December 4, 2009) - New York State Reliability Council, LLC (NYSRC) Installed Capacity
Subcommittee (ICS) – see page 25
© PJM Interconnection 2011. All rights reserved
Page 30 of 136
Figure 10 – Area Diagram
MRO-USA
Independent Electric System
Operator
1570
1200
New York Independent
System Operator
750
7500
Western
MA
6500
RFC
2660
PJM
7500
Eastern
MA
300
5700
PJM
7500
810
Rest of PJM
3000
1015
1500
PJM
Central
MA
8400
8400
MA = Mid-Atlantic
2000
1700
2330
200
3000
ReliabilityFirst
Other ECAR &
MAIN
1500
1350
1200
PJM
5500
1000
Non-PJM
1525
NYISO
140
Midwest Reliability
Organization
1000
Independent System
Operator of New
England
IESO (Ontario)
90
550
ISO-NE
1650
1300
1700
1500
1500
1600
ENTERGY
TVA
Tennessee Valley
Authority
2500
1000
3200
3500
© PJM Interconnection 2011. All rights reserved
Page 31 of 136
2500
Southern
2300
Non-PJM
VACAR
Virginia Carolinas
1100
550
February 16, 2009
2000
3500
2500
Figure 10 shows an example diagram of possible individual areas that could be modeled. Note that the New
York (NYISO), New England (ISO-NE) and Ontario (IESO) areas can be represented in further detail (at
least 10 subareas comprise each region shown). The calculations are done on an area basis, for each load
level specified, at each EOP level on an hourly basis. If this determines that an area needs assistance to
avoid an LOLE state for this hour, assistance is considered from the other areas in the model. See Figures
5A- 5C for further details on the solution process.
The assistance, determined in the calculation process to mitigate LOLE states, which the surrounding areas
can give, is a function of: 1) the individual given area of interest’s LOLE state (need for assistance) and 2)
the resource allocation assigned to each of the surrounding areas. The LOLE calculations attempt to
mitigate any area of interest’s LOLE states, using the defined resource allocation assistance (i.e. RES-PRIO
and RES-POOL). See Appendix F for further discussion concerning MARS input Tables.
Figure 11 – Expected need for Emergency Operating Procedures (EOP)
Figure 11 shows possible invocations of six EOP levels as the IRM is varied. As IRM increases, the number
of times a given EOP is invoked decreases. These are representative values only, not based on any model
or assessment performed. However this type of reporting is typical for MARS LOLE calculations and
assessments. The reduction in invocation values, from Step 1 (Operating Reserves) to Step 6 (Appeals for
Public Curtailment), is representative of how the system is controlled in the day to day operation of the bulk
power grid by the Control Room Dispatcher Staff. There are many more Control Room Dispatcher actions
than the limited number and aggregated steps shown for LOLE simulation.
© PJM Interconnection 2011. All rights reserved
Page 32 of 136
Model Calculation Comparisons: Observations
31
Both PRISM and MARS can perform their calculations on daily peaks. They use the hourly load
data for this dally peak, the maximum of the hourly integrated value for that day, used in the LOLE
calculations.
MARS can perform calculations, for each of the EOP levels, for each hour in its model. This is a
recent option, available in the current MARS version 3.00.
During 2002-2004, PJM compared the underlying generation distribution patterns used in the
calculations. MARS uses a transition state matrix distribution while PRISM uses a two-state
distribution. It was shown that the transition state matrix distribution were essentially identical to the
two state distribution as used in the calculations. This assessment was for the Mid-Atlantic region.
On a weekly basis, PRISM typically evaluates 105 values (21 points for each of 5 week days) while
MARS can evaluate up to 70 values (10 points in LOD-UNCY table for 7 days). Typical Industry
models for MARS use 7 values in the LOD-UNCY table. A difference in this is that PRISM uses the
same 21 values across all 5 weekdays during a given week while MARS uses different values for
each of the 7 peak demands (assuming that a given week does not have the same peak for
multiple days).
The evaluation of the Expected Weekly Maximum (EWM) and its consistent nature with the load
forecast has been documented in the 2009 RRS (pages 17 and 18). However additional effort is
pending to document and compare the use of MARS’ load model with the load forecast model.
Once this documentation is completed a comparison of PRISM and MARS LOLE load modeling
could be further investigated and assessed.
The Load Model in PRISM is a forecast probabilistic model, aggregating daily peak values for each
week of a delivery year (from June 1 through May 31). MARS uses a deterministic 8760 hourly
loads, which are typically historic in nature. The PRISM solution method requires a forecast
probabilistic daily peak load model. Both load models can be adjusted to be consistent with
forecast demands (See Appendix C3).
The PRISM calculation can be considered a direct solution; i.e. – there is no Confidence Interval
around a given IRM value with the calculated value the exact solution. The IRM value represents
the exact value for a given model’s assumptions and distribution characteristics.
MARS is useful for seasonal, monthly, and weekly assessment work. Due to PRISM’s use of a
probabilistic load model, it is useful for delivery year and seasonal assessments.
PRISM can automatically iterate into the determination of the IRM to comply with the BAL-502RFC-02 Standard. MARS requires additional manual intervention for this determination. MARS,
used in the NYSRC study, uses the Unified Method approach which adjusts the generation
resources applying a “TAN 45 criteria” for determining the compliance IRM value.31
Please see the section in this report concerning the early reserve study model about historic testing
of computational engines. Those reports, by the NYPP and PJM LCTF, performed a technical and
comprehensive evaluation of the solution methods that still apply today. The full reports are
available to the RAAS members, on a restricted private SharePoint site basis, so that individual
members can review and dialog concerning application of these reports on the current Industry
issues of interest.
New York State Reliability Council LLC NYSRC Policy, No 5-3, procedure for establishing New York Area Installed
Capacity Requirements, November 16, 2009)
© PJM Interconnection 2011. All rights reserved
Page 33 of 136
PRISM’s ability to automatically solve to a specified LOLE level
An advantage that PRISM has over other Industry computer applications is that it can automatically iterate to
a user-specified reliability index (RI). This can greatly aid the efficiency of PJM’s RAP staff, allowing
members to focus on assessment and analysis of model impacts rather than tedious, repetitive manual
tasks. This advantage has been known for years, and was cited in the 1993 report that compared MAREL
to GEBGE. This specific comparison and discussion also applies to the comparison of MARS to PRISM.
The following was cited in the Conclusion and Recommendations section of the November 1993 Load and
Capacity Working Group (L&CWG) report comparing the multi-area application MAREL to GEBGE. (The full
details of this report are posted on the private RAAS SharePoint site.)
“Unlike GEBGE, MAREL lacks the capability to automatically adjust load levels to solve the case to
a user-specified reliability index. Manually solving to a 1 in 10 reliability index is a tedious process
that requires numerous iterations with various load levels. The LCWG was forced to accept a
reliability level between 1 in 9 and 1 in 11 (due to resource limitations). Conducting reliability
studies with MAREL would, therefore, require much more time and effort than conducting the
studies with GEBGE.”
Typically, assessment work desires to solve to either a 1 day in 10 criteria … or a more stringent 1 day in 25
criteria. The automatic Reliability Index (RI) solution ability allows efficient assessments to any LOLE criteria
of interest. This is such an important aspect of the calculation method that the letters RI in the name,
PRISM, point to this automatic RI solution ability.
PRISM uses an analytical solution technique similar to PTI’s MAREL program. However it is purposely
limited to only two areas, so that its computational speed advantage is not compromised. Also the
requirement of many areas modeled in a study, does not apply to the PJM RTO due to the Agreements in
place for the PJM Membership.
The assessment need is for establishing a single RTO-wide reserve requirement, currently measured at the
unforced capacity (UCAP) level by the FPR. This single RTO-wide reserve value is critical in the Operational
methods used by the PJM dispatchers. In addition, again due to the structure of the Agreements in place,
the PJM assessment makes an assumption that any megawatt of generation can serve any megawatt of
load; i.e., there are no transmission system bottlenecks or constraints.
Through the TEAC Stakeholder group and the RTEP process, this assumption that any megawatt of
generation can serve any megawatt of load, is tested and satisfied in the load deliverability tests – the CETO
and the Capacity Emergency Transfer Objective Limit (CETL) – performed to satisfy the RPM capacity
market place. Therefore, one needs to ascertain any differences in the Agreement and Operation structure
in place for any RTO, State, or Control Area to completely understand why the Monte Carlo or analytical
(convolution) method makes the best choice.
Historical Continuity
The calculation algorithms used and deployed in PRISM are the same as those used in the GEBGE
program. The difference between PRISM and GEBGE is that GEBGE is based on text files and historic
FORTRAN coding methods (Not modular) while PRISM uses current SAS modular coding methods, data
base schemas, and java based GUIs.
While PRISM and GEBGE arrive at the same calculated results, the maintenance and ability to
accommodate changing Industry needs (including those to address Operations and Markets issues) is much
more efficient and less costly for PRISM. Because the underlying calculation methods and application of
statistics and probability theory are essentially the same between GEBGE and PRISM, it seems apparent
that the conclusions and overall intent of the comparison between GEBGE and MAREL are applicable to
PRISM as well. It is interesting to note in the 1993 detailed report that,”The evaluation was a result of an
investigation of the SRTF in September of 1987 which assessed general reliability methodologies.”
This indicates that a thorough and comprehensive evaluation needs sufficient time to reach conclusions and
that various phases can provide insights and information to the parties involved to evaluate and decide on
support for identified next steps (i.e.; it took 7 years).
© PJM Interconnection 2011. All rights reserved
Page 34 of 136
The report discussion summary indicates that there was general interest in acquiring the MAREL program
although it was noted that MAREL would not replace GEBGE but, rather, be used in conjunction with it. This
is the same philosophy in place today – using PRISM (an improved GEBGE) with MARS (an improved
MAREL).
Although there have been many and significant changes in the bulk electric Industry since this evaluation,
the drafting team is not aware of any fundamental technical changes to the calculations and algorithms used
so that the rationale of complementary tools is still valid (i.e. the solution techniques developed in the 1970s
and 1980s are still used in today’s Industry methods). Please see the section Complementary Tools for
further details.
© PJM Interconnection 2011. All rights reserved
Page 35 of 136
PJM’s Recent Comparison Efforts
During 2002-2004, PJM Staff conducted rather detailed comparisons between PRISM and MARS. These
efforts were the first steps in implementing and integrating MARS into the database schemas and analysis
that are an integral part of the overarching Applications for Reliability Calculations (ARC) environment. PJM
Staff hired a GE consultant to assist and help develop a method to:
1.
2.
3.
Compare the models
Compare the calculations performed
Compare the available standard output
The goal of this work was to determine the best ways to consistently assess the two applications.
A major difference between PRISM and MARS is how load is modeled and assessed. This comparison
further revealed that:
32
Modeling comparisons should begin with the Capacity Model, holding all the various load model
options to a given, flat unchanging shape.
Capacity model items of interest include PRISM’s two state variance and two-state EEFORd. Both
models need to represent all outages as captured in the GADS data. MARS can translate the
EEFORd values into its needed transition states using the UNT-FORS table. One identified
improvement for PRISM involves modeling transition table-like values.
The generation model values in the UNT-MXCP, UNT-CAPS, UNT-DATA, MNT-UNOP can be
matched identically to those used in the PRISM model (see appendix C3, slide 9, as an example of
this kind of comparison). Development of an automated process for this is an identified future effort.
PJM uses the MARS UNT-DERT table to model the same units used for the 2,500 MW ambient
derate in PRISM. This modeling can be identical.
When modeling generation resources for PJM’s Mid-Atlantic region, PRISM and MARS gave
identical results (difference in LOLE ~.001). This was shown by the Project Team, however the
documentation of this effort was not a Project deliverable and this intermediate step was not saved.
When this work was first performed, MARS had a rounding limitation that could report LOLE to the
nearest 0.001. However, MARS has now been upgraded to show the LOLE to 10 -12 decimals (or
more).
MARS’ deterministic hourly load shape (the in02 file); having 8760 values received much attention.
A conclusion was that these hourly loads could be a “design year”, not necessarily from a single
given historic year. From a comparison standpoint, developing a design year had several
advantages. Adjustments to this hourly shape needs appropriate consideration with the interaction
with the other load models characteristics. However the design year concept has not gained
acceptance by other regional groups. Another valid method to address this is to develop the LODUNCY values based on the 8760 hourly load values, as describe in Appendix C3, slide 15.
In 2004 a comprehensive assessment was made concerning what values are appropriate to use in
the in02 hourly load shape file, so that consistent LOLE values can be assessed. The 2002 historic
hourly loads, for the PJMRTO, were judged to be appropriate for use in LOLE studies. However the
selection of the given calendar year may introduce a modeling consistency issue throughout the
entire set of MARS load tables (See Appendix C3).32
The MARS LOD-DATA and LOD-MTAR tables can be identically matched to the corresponding
PRISM parameters, annual peak load (Load forecast Table B1 & B2) and monthly peaks (Load
Forecast Table B6).
ISO-NE load model processing for selection of LOLE load models.
© PJM Interconnection 2011. All rights reserved
Page 36 of 136
When comparing the load model, we can use one MARS table and the appropriate PRISM
parameters to make comparisons. However, all the various table values need to be evaluated and
assessed as a package to determine the best, most consistent modeling approach. Moreover, the
need to evaluate four tables makes for a complex and potentially time-consuming effort. The MARS
tables involved in the load model translation include:
o
o
o
o
o
LOD-MTAR
LOD-UNCY
LOD-DATA
In02 hourly load shape
Convergence Index used in the solution (CNV-CRIT) LOLE.
The LOD-UNCY table values in MARS are consistent with the PRISM PLOTS load models by
translating the WKPKFQ load model parameters, on an Expected Weekly Maximum (EWM) basis,
to the 7 probabilities typically used by the NPCC CP-8 WG. A spreadsheet calculation is used for
this and shown in Appendix B.
PRISM reports results after the invocation of Load Management (Demand Resources and Energy
Efficiency), based on the PJM Load Forecast Report Table B-8, but before a Voltage Reduction.
This corresponds to the MARS EOP level 3, which occurs just before a Voltage Reduction.
The EOP levels used are:
1)
2)
3)
4)
5)
6)
Operating Reserves
Curtailable Loads
30-Minute Reserves
Voltage Reduction
10-Minute Reserves
Public Appeals.
All levels must be determined, by separate evaluation, for each area modeled.
MARS can be used in conjunction with PRISM to perform LOLE assessments that need:
o
o
o
o
o
Less than annual factor/output considerations such as the Winter weekly Reserve
Assessments.
Hourly assessments such as items addressed by CURTAIL and Load management
duration (How many hours?) and Adequacy value of interruption.
Assessment of Demand Management impacts as it increases to be a significant
portion of the PJMRTO peak load and/or a given LDA’s peak load.
Assessment of Intermittent resources, due to hourly nature of Wind facilities.
Coordination of modeling/assessments with other regions.
Refer to the NPCC Long Range Adequacy Overview discussion of this paper, in the Interregional
assessment section. This is an example of how MARS reporting can be consistent with other
PRISM and RTEP related planning assessments.
In the short term, identified investigation items include:
33
The NYISO’s use and documented Unified Methodology using the Tangent 45 inflection point
calculation to determine the IRM in the MARS assessment. The New York Control Area Installed
Capacity Requirements dated December 5, 2008, pages 27-28 and dated December 4, 2009,
Table A-4 page 28 submitted by the New York State Reliability Council, LLC installed capacity
Subcommittee.33
Investigation into the method and techniques used by the ISONE staff to compare their
Westinghouse model with MARS. Refer to the April 28, 2010 presentation on this topic, at the
NERC LOLEWG meeting (http://www.nerc.com/filez/lolewg.html ).
New York State Reliability Council LLC NYSRC Policy, No 5-3, procedure for establishing New York Area Installed
Capacity Requirements, November 16, 2009)
© PJM Interconnection 2011. All rights reserved
Page 37 of 136
Investigation into the method and techniques used by the MISO staff to determine the IRM for their
region.
Access to models: ARC screen display and GUI layout
After logging into the secure, intra-net web based application, the following is the main selection screen for
the Applications for Reliability Calculations (ARC) environment. This environment conforms to the PJM ITS
standards and process that require a change management process for both the database and the
application. There is a robust system to move changes from the development platform to the Test platform
and finally to the Production platform. The response time and reliability of these systems is paramount for
the PJM staff to perform all activities related to the Adequacy studies.
The Week Peak Frequency system is how one specifies and creates the forecast, probabilistic load models
(PLOTS) used by PRISM. The ARC system Administrative system has five categories of activities which
include defining user access and twenty five database administrative functions. The “View Production
Information Warehouse (PIW) Status” option allows a quick summary of all database elements to identify
when the latest update was performed.
© PJM Interconnection 2011. All rights reserved
Page 38 of 136
The main MARS GUI screen is shown below. All options are indicated on the left, with all input options
perform by the selection task drop down button. The GEMARS Help link at the top of the page allows for
specific and detailed on-line user instructions and examples. The user can do searchable inquires and
default specific information is available in a point-and-click environment. The underlying database schemas
and tables are the “engine” that allow the GUI selections to create the Master Input File (MIF) file necessary
to make assessment runs. There is an understanding between the user and support staff that some text
editing is appropriate, as further development work is necessary to fully automate all GUI selections with the
database schema.
© PJM Interconnection 2011. All rights reserved
Page 39 of 136
The main PRISM GUI screen is shown below. All options are indicated at the top selection categories, with
all input details selected in the underlying GUI screens. The Help selection allows for specific and detailed
on-line user instructions and examples. The user can do searchable inquires and default specific information
is available in a point- and-click environment. The underlying database schemas and tables are the “engine”
that allow the GUI selections to create the PRISM and GEBGE text files necessary to make assessment
runs.
© PJM Interconnection 2011. All rights reserved
Page 40 of 136
#1 Comparing the Models
Table 1 provides a detailed comparison between PRISM and MARS and their relative strengths
and weaknesses. Of the many parameters considered, there are four prominent items to mention
from this table: a) Technology, b) Underlying Data Requirements, c) Planning Horizons and d)
Transmission System Model.
a)
Technology – PRISM uses state-of–the-art technology to improve data integrity,
quality results and user efficiencies.
b)
Underlying Data Requirements – PRISM uses a relatively limited set of statistical
parameters while MARS uses numerous event-driven, data-intensive tables. See
Appendix F for further details.
c)
Planning Horizons – Typically PRISM has long-range planning capabilities while
MARS has a comparatively shorter-term planning horizon. This is due to the
intrinsic nature of the operational parameters used in the MARS modeling. Note
that the MAPPCOR and CAISO do publish planning study results using MARS
for a ten year period.
MARS’ use of Operational related inputs present a long-range assessment study
issue as Operation staff is focused on shorter duration periods. These
operational parameters need feedback and assistance by the Operation’s staff.
Values in the long-range efforts might necessitate more sensitivities and
variations to capture uncertainty.
d)
Transmission System Model – PRISM can evaluate up to two areas at one time
whereas MARS has the capability of evaluating 75 (or more) areas.
#2 Comparing the Calculations Performed
PRISM uses a numerical method to simulate a general convolution method by an array look-up into
a cumulative probability table of unit outages. MARS uses a Monte-Carlo method that iterates on a
deterministic load shape simulating different generating unit outages. All things being equal, the
solution time for PRISM is markedly faster than that for MARS.
PRISM enjoys an efficiency advantage as it is able to automatically solve to a specified LOLE
criteria, saving the user from repetitive, tedious efforts. Investigation into other ISO techniques
would seek solutions to address this efficiency issue in the MARS processing.
#3 Comparing the Available Standard Output
Table 2 shows the comparison of the PRISM and MARS output files. One of the obvious
differences is that the PRISM output is in SAS tables and structures while MARS is a flat text file.
The “Text Pad” tool is used to cut and paste data from the text files while SAS can export data
directly into an Excel spreadsheet. Both methods are easy to extract data for reporting and creating
summaries. Both tools have many summaries and reported values that are useful in LOLE
assessment work.
The PRISM output does have the ability to be used in an OLAP, and by ARC’s use of a GUI and
underlying database schema assures consistency among the two models being reported.
© PJM Interconnection 2011. All rights reserved
Page 41 of 136
Comparison of Attributes Documentation
This section includes comparison of strengths and weaknesses and commentary on specific modeling attributes between PRISM and MARS.
Table 1 addresses attributes and application for each tool. Table 2 discusses eleven specific output comparisons with commentary. Table 3
discusses thirteen specific comparisons of database modeling relationships with commentary.
Table 1 – Model Attributes / Application Comparison Matrix
PRISM
Comparison
Strength
MARS
Weakness
PJM
Developer /
Owner
Developed in-house by PJM Staff;
relatively easy to make revisions and
modifications.
PJM is not staffed nor has the time and
resources to provide in-house subject
matter experts to provide PRISM
training and support to external users.
PRISM based on SAS and Database Schemas
Technology
Newer technologies allow for
improved data assessment and
analysis of results
Requires robust underlying information
systems – as a “stand-alone system”,
could be cost-prohibitive.
PJM – however, external requests have been made for the
predecessor application GEBGE
Current Users
Several LSE staffs use and
contribute to the methods deployed
in PRISM. Full and complete
documentation is available for
participant review.
© PJM Interconnection 2011. All rights reserved
Only PJM and ISO-NE staff use a
similar calculation approach. This
allows the opportunity to clarify and
produce concise answers to detailed
technical questions.
Page 42 of 136
Strength
Weakness
General Electric International, Inc. (GEII)
GE supplies a “turn-key” modeling
product and provides quality training and
excellent support to licensed MARS
users.
MARS based on FORTRAN and “flat text” files
Well-known programming structure;
enables easy, quick, manual changes.
Allows for efficient delivery of
executable code for updates that secure
proprietary intellectual property rights.
Older technologies are less flexible;
difficult to upgrade without wholesale
system changes
Multiple licensees
MARS enjoys a wide user network; this
is potentially valuable for interregional
reliability determinations and Broader
Regional Market applications. MARS
enjoys the distinction of an LOLE
product with widespread use among
almost all of North America. Full and
complete documentation is available for
participant review.
PRISM
Comparison
Strength
MARS
Weakness
GADS, historic loads, operational experience, transfer limits
Underlying data
requirements
PRISM has a comprehensive but
more limited amount of data inputs.
The nature of the data inputs do not
rely on other resources to assist in
compiling the data.
The numerical methods used for the
cum Probability table allow a quick
and direct solution (~ 15 minutes).
There is no confidence interval as
one calculates the exact LOLE.
No ability to produce confidence
interval value. No ability yet to
produce output for all 21 numerical
method states.
Yes
Determine
Capacity
Benefit Margin
(CBM)
The advantage is that PRISM only
needs a single value. Once
determined, this is not a
maintenance or otherwise
burdensome task.
34
35
Reporting LOLE is a single value,
before invocation of Voltage
Reductions. Clarity of basis of LOLE.
GADS, historic loads, operational experience, transfer limits
MARS needs more specific Operations
related data. This might cause a
modeling concern if other groups need
to perform an assessment for this data
and there is a resource (manpower)
limitation.
Monte Carlo solution / Standard Error basis usually a 0.05 value.
Can report out many aspects of the
solution process, tracking each solution
with a good summary output of solution
details.
Possible slow solution (some examples
35
were up to 18 hours) but depends
upon processing speed, modeling
assumptions and number of
simultaneous areas evaluated.
Not typically determined for a region with many areas
Only a single value to represent paths
involved in simultaneous import limit.
After invoking load management. (Not disconnecting firm load)
Reporting of
LOLE results
Weakness
Useful in detailing specific event data
(GADS 97 card) and events seen in
operations.
Direct convolution method / Solution tolerance is 0.001
Solution Basis34
Strength
No ability to automatically assess
LOLE at other EOP levels. Need
assessment for each desired EOP
level.
A very detailed transmission pipe
representation.
The transmission pipe sizes are
required for the model. If significant
Load flow work is needed, this could
cause a delay or additional effort to
make appropriate assumptions.
at EOP margin states
Calculation and reporting LOLE at up to
10 EOP levels.
Need Engineering Judgment
concerning the best reported value,
when considering consistent
comparison. Allows defining levels
differently which can muddy the water
when comparing results among
regions.
Solution Basis is determined using a 2-CPU server running Windows 2003 Operating System with 18 gigabytes of RAM and CPU speed of 3.0 GHZ – as applied to large models,
such as those modeling the bulk of the Eastern Interconnection.
Eighteen-hour (18) run times were seen in detailed models using dynamic interface transfer limits, as this is part of the core calculations that requires the largest percentage of
solution time. Refer to Figure 5C for the Monte Carlo simulation process.
© PJM Interconnection 2011. All rights reserved
Page 43 of 136
PRISM
Comparison
Strength
MARS
Weakness
Two: 1) PJM RTO, 2) World
Simultaneous
Study Areas
Up to two areas. This can be an
advantage if proper consideration of
assumptions and its use can be
established to simplify modeling.
For some assessments, PRISM is
limited in assessment capabilities such
as evaluation of contributions from
more than one surrounding neighbor.
One transmission interface (pipe) between areas
Interface
Transfer Ties
One transmission tie, which allows all
pertinent assessment work to
happen outside of the LOLE model.
No probabilistic model for the
transmission tie. Only one transfer pipe
can limit modeling and coordination
with known operations practices.
Planning
Modeling Style
Due to its probabilistic load and
generation model, this is only a
planning division tool.
Increased development to coordinate
with Operation practices.
1 year to 11 years ahead
Planning
Horizon
Data Input
Requirements
36
PRISM offers wider time range.
PRISM’s use of statistical
parameters are useful in the time
frame
Primary use is not l close in seasonal
assessments or less than annual
evaluation efforts.
Three primary input categories-Load, Capacity, Tie Size
PRISM less complicated, less time
intensive
Does not have breadth and depth of
data inputs for reporting out many
different LOLE aspects.
Strength
Weakness
Multiple (up to 75 used)
MARS exhibits much greater flexibility;
virtually unlimited
Increased data requirements.
Multiple (up to 125 used)
MARS exhibits much greater flexibility
Increased data requirements.
Planning, Operational
MARS allows for Operational data to be
specifically modeled
Need for operational data, increased
effort and potential delays or use of
best estimates to ensure best quality
data.
6 months to 5 years ahead
MARS is very useful for close in
assessment work, such as seasonal
assessment that is 3 to 9 months in the
36
future.
MARS can be used to perform long
range assessments, but this
necessitates increased efforts for a
quality forecast data model
(Operational data such as transmission
limits, megawatt values for Emergency
Operation levels).
Many separate tables for modeling values (~50)
Many, many Tables, allowing a plethora
of detailed options that allow Operations
related assessment. Very good for close
in assessments (1-5 years).
Much more complicated, with model
needing all tables of interest before a
single run can be made. Might force
use of best approximations.
California ISO Planning Reserve Margin – 2010 – 2020, May 21, 2010 – 2020. http://www.caiso.com/279d/279ded0337f20.pdf and MAPPCOR December 30, 2009 LOLE
Study 2010-2019, look out for a 10 year period.
© PJM Interconnection 2011. All rights reserved
Page 44 of 136
PRISM
Comparison
Modeling
specifics of
interest
Strength
MARS
Weakness
PRISM models include: Weekly Standard. Normal distribution and Expected
Weekly Maximum (EWM); Forecast monthly peaks; Peak Load Ordered Time
System (PLOTS) file-Mean and STD; Sensitivities to assess EOPs and transfer
ability; Capacity model based on PJM Manual M-22 statistics; EEFORd; TwoState Variance; Unit maintenance; DR & EE; Solved load and 50/50 forecast;
Unit commercial probability; BTM and uncommitted resources
Many details can be modeled and
assessed.
PRISM allows all states (possible
loads) to be evaluated in a given
base case. This is a forecasted
model, with a basis in recent, historic
daily peak demand.
PRISM does not allow an easy, direct
correlation of load shape to a given
Operations period.
Describing and explaining the load
shapes is not straight forward and
requires considerable time, energy,
and a given level of expertise for a
complete understanding.
Based on 10-year historical data
Load Shape
Forecast Probabilistic shape.
Need to assess shapes and
association with other regions for best
selection of time period.
Magnitude ordered – using PLOTS Load Model Shape (Weekly
mean, STD)
Load Model
Shape
Forecast Probabilistic shape.
© PJM Interconnection 2011. All rights reserved
Weakness
MARS Tables includes: LOD-UNCY, LOD-MTAR, in02 file, EOP-DATA, INFTRLM, ELU-DIST, UNT-DERT, UNT-FORS, FCT-DATA, INF-DLYM, INF-DATA,
INT-ONLY, NUM-TRNS, MNT-OPTN, MNT-UNOP, UNT-MXCP, UNT-CAPS, UNTDATA, LOD-DATA, CNV-CRIT, GEN-AREA, GEN-POOL, GEN-CASE
Many details can be modeled and
assessed.
Probabilistic
Load Model
Type
Strength
Need to assess shapes and
association with other regions for best
selection of time period.
Page 45 of 136
Deterministic
MARS allows for specific operation
condition modeling. Explanation of
deterministic values used is rather
straightforward. Use of load forecast
uncertainty table places a weighting of
the probability a given load value is
expected to occur.
MARS Load Model is not a forecast
shape; it is based on historic events.
Needs periodic and independent
assessment work to determine best
historic shape to use. MARS also
needs separate assessment to
determine appropriate load forecast
uncertainty table values, for each area
in mode.
Committee selection of “Base Year” Load Shape
MARS can apply a forecast monthly
distribution and a load forecast
uncertainty to make the deterministic
shape in line with probabilistic resource
model in Monte Carlo simulations. See
Appendix C3.
Since MARS uses a deterministic load
shape, it is constrained by a subjective
and engineering judgment selection
process.
Chronologically ordered – Hourly Load Shape
Straight forward to implement, once a
year has been chosen.
Deterministic based on historic year. If
the shape is many years ago, one can
lose the relationship with current trends
and forecast shapes used in 50/50
forecast.
PRISM
Comparison
Strength
MARS
Weakness
Daily
Peak Load
Assessment
Load Diversity
Forecast Error
Factor (FEF)
Expected
Weekly
Maximum
(EWM)
PRISM modeling starts with the
same hourly loads; it moves to the
more critical daily peaks thus
streamlining any maintenance or
periodic review issues.
1) Simulation based on hourly loads
which capture historic diversity.
2) Consistent with Load forecast.
3) Lower resource requirements to
perform assessment.
A stakeholder and PJM staff
documented values and rational.
PRISM can only address daily peak
assessments. Cannot perform Hourly
assessment using LOLH or EUE.
Yes
1) Resources and expertise to perform
assessment
2) Limited to two area model.
3) No ability to capture dynamics
between sub regions that make up
world region.
Yes
Yes
Lack of increase for future delivery
years.
Documented using the 1st order
statistic in proper aggregating the
daily values into each week modeled.
Yes
Load Forecast
Uncertainty
(LFU)
These are determined from the
appropriate WKPKFQ model and
EWM values.
This directly corresponds to the
monthly load factor used in PRISM or
the 18 Control Card loads.
© PJM Interconnection 2011. All rights reserved
Weakness
Hourly Load Shape, MARS table: in02 file
Granularity can cause issue as hourly
shapes need periodic detailed
assessments.
MARS can use more granular approach
1) Able to simultaneously assess multi
area diversity. Inherent in the 8760
hourly loads (in02 file) data
modeling specification.
2) Can be made consistent with the
monthly peak load forecast.
Yes
1) Resource and expertise to perform
assessment work.
2) Must decide on consistent model
characteristics for a system wide
study.
Not typically done, but could be included.
Can be Incorporated in the LOD-UNCY
table
No way to determine specifics
concerning this modeling attribute.
Not needed for the daily peak
calculations. This uses a different load
model than PRISM, based on 8760
hours. See Appendix C3 for further
discussion (slides 10-15).
Needs coordination with probabilistic
load distribution if consistency and
result reporting is desired to be readily
available between the two tools.
Yes but needs further efforts
MARS table: LOD-UNCY
Use allows the load model shape to be
evaluated at various level of
probabilities.
Yes
Monthly
Forecast Loads
Strength
Page 46 of 136
MARS table: LOD-MTAR
Use allows the forecast monthly load
model shape to be adjusted to match the
50/50 forecast.
PRISM
Comparison
Strength
MARS
Weakness
No
Load
Management
Assessment
PRISM uses an application called
CURTAIL. This evaluates the
reliability benefit for each invocation
of a load management event.
Uses the EEFORd and a two state
model to represent either a full on or
full off scenario.
CURTAIL has limited documentation
and would need time to fully document
process and rational.
Requires GADS event data for 5 year
period.
Represented by a statistic
Forced Outages
Weakness
MARS can model and calculate hourly indices
Probabilistic
Capacity
Modeling
Strength
The PRISM modeling initiated the
Industry to consider including the
EFORd statistic into the IEEE
Standard regarding this matter, IEEE
STD 762.
MARS can assess things related to Load
Management duration, wind unit
performance, other intermittent
resources, etc.
Probabilistic
Uses transition states or Forced outage
rates to approximate transition states.
Yes
Basis for each unit’s forced outage rate
to determine unit availability.
Done outside of PRISM w/ supplementary GE program using
NUM-TRANS table
© PJM Interconnection 2011. All rights reserved
MARS allows non-EFORd values to be
input. Care needs to be made to make
sure all values are on the same basis
and consistence with IEEE STD 762.
MARS offers robust solution
Basis for each unit’s forced outage
rate to determine unit availability.
The primary unit two state
performance statistic, EEFORd, is
based on the EFORd. See PJM
Manual 22.
Transition Rate
Requires GADS event data for 5 year
period.
Represented by a table
Yes
Equivalent
Demand Forced
Outage Rate
EFORd
Typically this is performed by specific
sensitivity work.
Not available in PRISM yet. An
identified future enhancement.
Page 47 of 136
MARS table: UNT-FORS
Application included in MARS
PRISM
Comparison
Strength
MARS
Weakness
Strength
Weakness
Yes
Wind Unit
Performance
Other
Intermittent
Capacity
Resources
Modeled per the existing guidelines,
outlined in PJM Manual 21, Appendix
B
Yes
Need GADS event data for proper
modeling. Proxy is data turned into
GADS administrators over previous 3
year peak period.
Yes
Modeled per the existing guidelines,
outlined in PJM Manual 21, Appendix
B.
One transmission tie only. No issue
with data collection.
Need GADS event data for proper
modeling. Proxy is data turned into
GADS administrators over previous 3
year peak period.
Identified improvement could be to
model a probabilistic set of values.
Relatively Low
Modeling
Resources
(Staff)37
MARS offers robust solution. Modeled
per the existing guidelines, outlined in
PJM Manual 21, Appendix B.
Shorter runtime is a major
advantage for mitigating PJM staff
resource limitations.
37
Need GADS event data for proper
modeling. Proxy is data turned into
GADS administrators over previous 3
year peak period.
Deterministic
Variation of results, offers robust solution
and reporting.
Could cause delays or extra effort due
to resource limitations to determine
modeling values.
Higher
Data requirements are captured by
PJM’s RAP department. No known
resource limitations.
Typically ~ 5-15 minutes
Solution
Runtime38
Need GADS event data for proper
modeling. Proxy is data turned into
GADS administrators over previous 3
year peak period.
Yes
Deterministic
Transmission
System
Modeling
MARS offers robust solution for
intermittent resources. Modeled per the
existing guidelines, outlined in PJM
Manual 21, Appendix B
Potential delays and limitations of
quality data due to resource limitation
of non PJM RAP staff.
Long ( 30 min - 18-hour) runtime due to multiple iterations (~2500
runs)
Can be addressed by using a large
server network or running on a super
computer cluster.
Possible times of up to 18 hours, but
access to a large network environment
can serve to significantly reduce this to
about 30 minutes.
Reference Resource and cost Assessment Section for details. Specific assessment efforts vary. Depending on the assessment, PRISM or MARS might show lower resource
estimates.
38
Solution Basis is determined using a 2-CPU server running Windows 2003 Operating System with 18 gigabytes of RAM and CPU speed of 3.0 GHZ – as applied to large models,
such as those modeling the bulk of the Eastern Interconnection.
© PJM Interconnection 2011. All rights reserved
Page 48 of 136
PRISM
Comparison
Strength
MARS
Weakness
Solution based on size of Cumulative Probability Table
Runtime
Characteristics
Strength
Network complexity, binding issues can impact run time
If appropriate run time options are
selected, Results stored have plethora of
details for the specific calculations
performed in the Monte Carlo
simulations. See figures 5A-5C.
PRISM employs look-up table
approach
Yes
Hourly Data
Assessment
PRISM starts with the hourly loads.
Hourly loads subject to several
automated checks and OLAP review
to insure data quality. Possible
assessment work for neighboring
regions as the hourly loads for
select time period are mandatory.
© PJM Interconnection 2011. All rights reserved
Weakness
Combining all sub-regions that
comprise the World region assumes
close coordination of many control
areas. These control areas are
geographically diverse which might not
represent operational practices.
Page 49 of 136
Yes
Hourly data is used per separate
determination of best values for study.
Consider “Design Shape” instead of
selecting historic hourly loads.
Table 2 – OUTPUT Comparison Matrix
Output Comparisons
Table 2 shows eleven separate output summaries that PRISM produces, and the corresponding MARS
output files. These summaries indicate the results of the calculations, giving various aggregations and
details. These summaries are the “standard” output, or otherwise stated, this is the output for most PRISM
and MARS runs, using the most commonly used input parameters.
PRISM can output many more calculations details, by setting the “DEBUG” option before submitting a
PRISM case. The DEBUG option causes many more output files to be created for a given case. The
DEBUG option can cause use of more computer resources and require more solution time. Once the
DEBUG option is set, it will apply to all submissions in a given environment. This allows someone working
in the development environment to work on developing calculation changes, needing scrutiny of all the
details, while those in the Production environment can perform normal work and not be affected by the
additional DEBUG output. See Appendix E for further details.
The analogous items in MARS to the DEBUG option in PRISM, is the many and various input parameters in
the CNV-CRIT, GEN-TIME, and GEN-CASE tables.
© PJM Interconnection 2011. All rights reserved
Page 50 of 136
Table 2 – OUTPUT Comparison Matrix
Output File
PRISM Output
MARS Output
Comments & Comparisons
1. ALMTABLE (Demand Resources)
Active Load Management (ALM) refers to the
former term used for Demand Resources
(DR). The output is derived from four specific
runs:
1. Base
2. Insufficiency
3. Necessary
4. Initiation
Not available in MARS.
This is a group of specific and automated analysis
which would need to be developed using MARS.
All these are inputs into the CURTAIL
application.
This output reflects the reliability values of DR,
the maximum amount of DR that saturates its
reliability value at 10 interruptions, and the
number of times a given amount of DR needs
to be interrupted for its reliability value to
saturate
The Load Management calculation details are part
of the desire to treat the contractual obligations,
Operational practices and planning assessment on
a consistent basis.
The calculations measure the reliability benefit of
the number of LM interruptions.
Contractually the limit is 10 for the summer peak
season.
Historic calculations verify that the full year benefit
is similar (identical) to the summer only benefit.
There are several issues surrounding DR with
assessments and evaluations being performed to
discern any different and new approach due to
using Load Management in a resource construct
(Bids into Capacity market).
2. CUMPROBTABLE (Cumulative Probability of Outage)
The cumulative probability table is a twocolumn array, indexed by the amount of MWs
to be evaluated for an outage, typically the
columns represent: 1) daily peak load, and 2)
the associated cumulative probability. The
cumulative probability is based on the unit
characteristics of the model.
For a two-area run there are two separate
tables: Area 1(PJMRTO) and Area 2
(Neighboring Region).
© PJM Interconnection 2011. All rights reserved
Page 51 of 136
Output Table (OT) 10 (OT10) can be used to
determine the area margins, but this will only yield
the LOLE for each load level evaluated and the
megawatt amount of the capacity.
However, the distribution characteristics of the
aggregated unit total are unknown for each
replication.
These negative margin values, for each load level
and each EOP level can be outputted by specifying
the input parameters in the GEN-TIME table, "Print
hourly output" with caution given that this can
result in a very large output.
The PRISM margin values for any of the values
looked up in the cumulative probability table are
not saved or known outside the calculation
process as the determination is only if the load
value looked up can be served by available
generation, it is an "on" or "off" switch.
If running this with the PRISM DEBUG options, the
Cumulative probability table can be saved, but the
intermediate LOLE equation values for all points in
the distribution are not saved (yet).
PRISM Output
Output File
MARS Output
Comments & Comparisons
3. OUTAGE_SCHEDULE
Parts of the OT7 file output. Interestingly there are
53 weeks for this table showing the individual unit
planned pattern. This shows weekly categories:
o
Fixed daily maintenance
o
Partial fixed maintenance
o
Scheduled maintenance
o
Partial Scheduled Maintenance
o
Unit not yet installed
o
Unit Retired
Two MARS summaries, for each calendar year in
the study are shown.
One of the primary summaries of calculated
results showing the weekly:
o LOLE
o Mean of Load distribution after
Monthly adjustment
o Load distribution Standard Deviation,
o Expected Weekly Maximum
o Total Standard Deviation (after
applying 1st order statistic and
Forecast Error Factor(FEF)),
o Capacity amount on planned outage
o Reserves in percent and megawatts.
OT8 File Output gives detailed results, on a weekly
or Monthly basis for each calendar year
incorporating several summaries of these values:
o Weekly Indices, LOLE (Days), LOLE
(Hours), LOEE (MWh)
o Expected Number of Days at Specified
Margin States – Weekly
o Monthly Indices, LOLE (Days), LOLE
(Hours), LOEE (MWh)
o Expected Number of Days at Specified
Margin States – Monthly
o Interface Flows – Monthly
In a general sense the outputted values show the
necessary details.
A process is in place for both applications to
request and change the outputs as justification
and benefits are appropriate.
A given summary, due to the nature of the
difference between the analytic solution
(convolution) and the Monte Carlo solution might
be cost prohibitive so that a direct comparative
output can be made.
There are many output summaries in the OT7,
OT8, and OT11 files.
Shows annual LOLE and RI, separately for
two area model
OT7 file output gives detailed results on a weekly
or monthly basis, for several categories with the
"Weekly Reserves Summary" category of interest.
OT11 provides further detailed area results, on
weekly, monthly load level (LOD-UNCY), and EOP
level basis for each calendar year; summary
categories include:
o
For each LOD-UNCY level, Calculated
Indices (LOLE (days/yr), LOLE (hrs/yr)
LOEE, Frequency, Duration) for both
isolated and interconnect system.
o
For each LOD-UNCY level the expected
number of days at specified EOP margin
state
o
For each LOD-UNCY level, Import and
Export Interface flows
Listing of generation units that are unavailable
due to a planned outage for the 52 weeks in
each delivery year assessed in a given run.
4. PERIOD_PROBABILITY
© PJM Interconnection 2011. All rights reserved
Page 52 of 136
PRISM Output
Output File
MARS Output
o
o
o
o
o
o
Comments & Comparisons
Annual EOP usage for given LOD-UNCY
level
Weekly Indices (LOLE (days/yr), LOLE
(hrs/yr) LOEE, Frequency, Duration) for
given LOD-UNCY level load
Weekly Expected number of days at
specified EOP margin state for given load
level
Monthly Indices ( LOLE (days/yr), LOLE
(hrs/yr) , LOEE) for given load level
Monthly Expected number of days at
specified EOP margin state for given load
level
Monthly Import and Export Interface flows
for given load level.
5. PRISM_PARAMETERS
Identifies user name, runtime, input file name,
location of output file, various solution
parameters.
Useful for trouble shooting and debug efforts.
The various files used in the study are stored in the
same directory.
These can be found quickly by going to the
directory_Help of the ARC system and selecting
the GEMARS editable directories.
For MARS runs, there is no record of users as the
information is stored in the directories per the
Windows OS.
OT6 shows
o Number of replications in study
o Convergence Tolerance
o Standard Error
MARS Solution time can be determined, assuming
one saves the MIF input file just before
submission, by subtracting the time of the OT files
from the input MIF file.
6. PRISM_TIME
Indicates amount of time for each set of
analysis
7. SUMMARY TABLE
Primary output summary indicating:
o Solution load
o Installed Capacity
o Week of peak
o Installed Capacity (ICAP) Reserves
o Available reserves(after planned
outages),
o CBM value
o RI
© PJM Interconnection 2011. All rights reserved
OT 9 provides general results for each calendar
year, using several summary categories:
o Program Option Summary
o Interface Rating Data
o Area Capacities and Loads
o Calculated Indices - Both Isolated and
Interconnected LOLE as days per year,
hours per year,
o Frequency Duration (expected number of
days per year at specified margin states
o Interface Flows
o Annual EOP Usage - (expected number of
times per year)
Page 53 of 136
Output File
PRISM Output
MARS Output
Comments & Comparisons
8. SYSPAR (System Parameters)
SYSPAR indicates:
o
Installed Capacity (ICAP)
o
Load Distribution mean
o
Load Distribution Standard Deviation
o
ICAP Distribution Mean
o
ICAP Distribution Variance
o
Average Unit Size
o
Average Forced Outage Rate
o
Average Planned Outage (MW)
OT 7, OT 8 and OT 9 includes the underlying data
(EX individual unit) that are used in these
summaries.
However, the OT7 file does show aggregated load
model statistics on a weekly basis.
9. UNITDESCRIPTIONS (Generation Unit Descriptions)
List of all units in the case: name, unit number
assigned case number, geographical number
OT7 includes individual unit data statistical data in
three summaries:
1. Weekly Individual generation unit
maintenance and outage schedule
2. Available Energy by non-Thermal unit
type(Monthly)
3. Thermal Unit Summary (Summer & Winter
Ratings, FOR, planned weeks of outage)
10. UNITVALUESBYWEEK (Weekly Generation Unit Values)
List of all units in the case, by delivery year:
name, unit number, Variance, EEFORd, Adder
or change unit, rated value for each of 52
weeks, on or off status for each of 52 weeks.
© PJM Interconnection 2011. All rights reserved
Page 54 of 136
OT7 shows the individual unit data statistical data
in three summaries:
1. Weekly Individual generation unit
maintenance and outage schedule
2. Available Energy by Non-Thermal Unit
Type(Monthly)
3. Thermal Unit Summary (Summer & Winter
Ratings, FOR, planned weeks of outage)
These summaries are for the distributions for the
primary model input categories: Load, Generation,
and Transmission.
Some summaries might use a distribution,
perhaps a confidence interval applied, for
application in the MARS Monte Carlo solution.
Output File
PRISM Output
MARS Output
Comments & Comparisons
11. OLAP (On-Line Application Program)
A cube using an On-Line Application Program
(OLAP) uses ~ 80 specific datasets built from
the primary SAS input database tables and
resulting output files.
This allows many different and unique
combinations of the underlying dataset details
and allows efficient investigation and
verification of data quality of both the input
data and assessment results.
The key in this is the appropriate joining of the
load data, defined by sub zones, with the
generation resources, defined by individual
units, using Holistic techniques’ (HOLAP) to
correctly define complex relationships used to
build the database schemas.
This allows the planning engineer user to
display many facets of the data, in a point and
click GUI, otherwise not available without ITS
staff expertise and resources.
OLAP metadata summaries include:
o
Forecasted Loads and Reserves, by
subzone
o
Historical Loads, by study area
o
PRISM Reports
o
Generation Owner Web Comparison
Report
o
Peak Week Frequency reports
o
Generator class average comparison.
© PJM Interconnection 2011. All rights reserved
Page 55 of 136
MARS does not have this capability as it uses text
output files.
Online analytical processing, or OLAP
(pronounced /ˈo-lap/), is an approach to quickly
answer multi-dimensional analytical queries.
OLAP is part of the broader category of business
intelligence, which also encompasses relational
reporting and data mining.
The typical applications of OLAP are in business
reporting, and business process management
(BPM)
The term OLAP was created as a slight
modification of the traditional database term
OLTP (Online Transaction Processing).
Databases configured for OLAP use a
multidimensional data model, allowing for
complex analytical and ad-hoc queries with a
rapid execution time.
OLAP methods borrow aspects of navigational
databases and hierarchical databases that are
faster than relational databases.
The output of an OLAP query is typically
displayed in a matrix (or pivot) format.
The dimensions form the rows and columns of
the matrix; the measures form the values.
Table 3 – DATABASE Modeling Relationship Matrix
Table 3 shows thirteen categories related to the data base schemas in use. These schemas use SAS and
Oracle tables. Table 3 discusses the relationship between the two tool’s established data base items. It has
been said that “It is all about the data!”, and that still continues to be the case in conducting the various
Adequacy assessment work. An important and significant amount of effort is needed to ensure high data
quality. High data quality is a mantra for the PJM staff involved in the Adequacy study efforts. One of the
processing methods in place is comparing our SAS database with known detailed Industry data which might
be available by participation in Industry groups, with Industry consulting firms, or with public data sources.
© PJM Interconnection 2011. All rights reserved
Page 56 of 136
Table 3 – DATABASE Comparison Matrix
Database
File
PRISM Database
MARS Database
Comments & Comparisons
1. ALL Monthly Load Forecast
This database of the monthly peak load
forecasts is for each identified zone of the
PJMRTO or neighboring World regions.
The PJMRTO data is from the PJM Load
forecast report typically dated in January. The
neighboring World data is based on the NERC
Electric Supply and Demand (ES&D) report.
The PJMRTO data is posted on the PJM web
site as well, but the database has more
granularities and incorporates many
complexities not available in the public data
posting.
This database of the monthly peak load forecasts
is for each identified zone of the PJMRTO or
neighboring World regions.
The PJMRTO data is from the PJM Load forecast
report typically dated in January.
The neighboring World data is based on the NERC
ES&D report or publically available data posted on
other neighboring ISO web sites.
The PJMRTO data is posted on the PJM web site
as well, but the database has more granularity and
incorporates many complexities not available in the
public data posting.
The MARS data does require different summaries
in addition to those used in the PRISM model, due
to using different area cut-sets (the cut-set define
an area's boundary).
The LOD-UNCY-01 table can currently
accommodate monthly shape distribution values.
The PJM staff has asked for this input's granularity
to be improved so that either weekly or daily peak
distribution parameters can be specified.
The maximum number of points that can be used
in this distribution is 10.
The process to determine these points is
discussed in Appendix C and an example shown in
Appendix B.
The LOD-UNCY data is stored in the SAS tables,
on an area basis, but is currently manually inputted
into the MIF.
This data is very similar in structure between the
two models with both based on the same data
sources, either the PJM load forecast report or
NERC ES&D.
Some of the modeling needs for MARS require the
most granular sub zone data due to drawing cutset lines around a given control area. PRISM
aggregates things into one of two categories,
either a PJM RTO or World region.
2. Non-Normal weekly load shapes
The database allows for weekly load
distributions to be input.
The distribution can be from external
assessments to give the correct forecast
shape for each week in the model.
Currently this shape is used for both area 1
and area 2, with work in progress to specify
different weekly shapes for the two areas
(PJMRTO and neighboring World region).
These are the Characteristic (load) Data
Curve (CDC) files in the database which are
selected in the PRISM GUI parameters.
There is no limit for number of points used in
these distributions.
© PJM Interconnection 2011. All rights reserved
Page 57 of 136
PJM staff is working to produce the weekly shapes
used in PRISM and to improve the SAS code so
that different shapes can be used for Area1 and
Area 2.
PJM is also discussing with the GE staff the
benefits of improving the LOD-UNCY table's
granularity.
Database
File
PRISM Database
MARS Database
Comments & Comparisons
3. CLASS_AVERAGES
Values listed by categories based on type,
size, and primary fuel of a unit.
These values primarily come from the existing
PJM Fleet of units, about 1400 units. NERC
pc-GAR data is used for any categories that
do not have a sufficient number of PJM units.
These values are used when a unit does not
have 60 full months of GADS event data.
The statistical values involved in this are the
POF, EFORd, Two-State variance, EEFORd,
and EMOF.
The data source is the same as for PRISM, which
is used in the UNT-FORS, UNT-MXCP, MNTUNOP tables.
However MARS is based on the transition rate
matrix so the UNT-TRNS and UNT-CAPS tables
must be accommodated using the known data.
Directly calculating the UNT-TRNS and UNTCAPS from the GADS event data would improve
the MARS model and allow for measuring the
approximations used in determining the tables'
values based on the two-state values used in
PRISM.
MARS uses the same SAS data sources as
PRISM but any manipulation for sensitivity analysis
is done manually.
Automating this to avoid any manual intervention,
using Text Pad, would be additional effort.
This is a small number of units but can be tricky to
achieve high data quality due to the external data
(not the "normal" SAS planning data) and its
changing nature.
MARS uses the same SAS data sources as
PRISM. This data is used to automatically populate
the UNT-DATA, UNT-FORS, UNT-MXCP, and
MNT-UNOP tables from the SAS data as
described for PRISM.
It would be an improvement to directly populate the
UNT-TRNS and UNT-CAPS tables from the GADS
event data (97 Card).
Currently the UNT-CAPS data uses appropriate
estimated values.
The UNT-TRNS values are generated using the
INT-ONLY Table, documented in version 3.00.
If MARS is to use the Transition state data (UNTTRNS & UNT-CAPS), additional development
work is needed to automate this data process.
Currently the data source is the same for both
tools.
MARS as the ability to represent the full unit
characteristics based on all recorded GADS
events, using the transition state matrix, instead of
the two-state approximations.
To determine the transition state values from the
GADS event data would require additional
development, which is not trivial.
4. EXTERNAL UNITS
PRISM tracks all units that hold an agreement
of non-recallability to serve PJM load 1st.
These units are included into the PJM RTO for
study purposes.
The location of these units is an important
aspect of the modeling data so they are
placed into the correct zone when performing
sensitivity analysis.
This is one of several operation database links
that coordinates our model with agreements /
practices used in operations.
5. FUTURE_GENS_PJM
This is the forecast future units for both the
PJMRTO and the external world areas.
Information is for individual units capabilities
and when they are in-service, retired, or have
modifications (up-ratings).
The primary source of this modeling is the
PJM generation Interconnection Queues.
The values entered in this data base reflect
the commercial probability for a given study
model.
This data takes considerable time to develop
and verify.
© PJM Interconnection 2011. All rights reserved
Page 58 of 136
Database
File
PRISM Database
MARS Database
Comments & Comparisons
The units are given the class Average unit
performance values based on its category
(i.e., Fossil steam, Nuclear, Combined Cycle,
Wind).
The number of units for each neighboring
world area is based on the appropriate
reserve level per the study assumptions and
NERC Electric Supply & Demand (ES&D)
assumptions.
6. NORMAL
This data base representation uses 21 points
to appropriately model the Normal distribution.
This SAS data set is automatically referenced
as the default distribution to use in all PRISM
assessments.
The analogous item in MARS is the LOD-UNCY
table values.
However consideration of these values should be
in the context of consistent application with the
8760 deterministic hourly loads entered in the IN02
file. See discussion in Appendix C3.
The LOD-UNCY values are manually entered via a
text editor. However a SAS table is available
(LODUNCYDATA) for future automation efforts.
For a week's worth of daily peak values, PRISM
uses 105 values (5 week days X 21 points each
day), and MARS can use 70 values (7 days X 10
values each day).
o While the 70 MARS values can be unique
as each day peak is different, PRISM’s 105
values use 21 unique values –same daily
peak for each day.
The automation of this data for MARS analysis
has been identified as a valued effort in LOLE
assessments by the PJM staff.
7. OPERATING GENERATORS
This SAS table has more than 30 columns of
detailed information that allows automated
input into all possible PRISM assessment
scenarios.
This is a data base for all existing (iron in the
ground) units.
There are over 1200 PJMRTO existing units
and 4000 World units in this data base.
The existing units modeled in MARS have the
same SAS data source as PRISM.
The tables populated automatically include UNTDATA, UNT-FORS, UNT-MXCP, and MNT-UNOP
tables.
The two state unit values are typically selected for
use as the transition state matrix cannot yet be
automatically created.
The existing units modeled in MARS have the
same data source as PRISM.
However automation would need to be developed
to use the complete capabilities of MARS.
MARS Transition state matrix values for World
units would be a new development work item.
This table is a result of processing the operating
generator details table.
Most of the World unit model could come from
exchange of MARS data with neighboring ISOs.
Non Disclosure Agreements (NDA) are necessary
as data is confidential. See Operating agreement,
page 196(Section 18.17.1 paragraph B)
8. OPERATING GENERATORS DETAIL
This SAS table is the basis for the Operating
Generator SAS table.
Several small units, especially in the World
area, need combining so that the rounding of
units' rating to the nearest tens does not
significantly reduce the amount of available
capacity because of many small units (less
© PJM Interconnection 2011. All rights reserved
Page 59 of 136
This data source is typically only used for a few
sensitivities and assessment work.
The primary use of this data base source is to
create the operating generators table.
Database
File
PRISM Database
MARS Database
Comments & Comparisons
than 5 MW).
This has over 30 columns of detailed
information that allows automated input into all
possible PRISM assessment scenarios.
This is a data base for all existing (iron in the
ground) units.
There are more units in this table than the
operating generators table.
9. ZONE MAP
There are several assessment categories that
a given sub zone can be involved with.
The PJMRTO and the neighboring World
region is separated into various "cut sets" so
that a given region can be assessed and
modeled correctly.
A cut-set is a boundary line used to define a
given area (like a property line for a home).
One of the significant items of interest is
correctly representing the diversified peak
load (the sum of individual sub zones peaks
usually do not equal a regional peak).
This data base indicates child parent
relationships. These relationships include; a
location key, a zone number, level associated
with each location key and zone number, zone
name, and a character- based zone name
code.
The SAS table of the MARS Zone List is
comparable to the PRISM zone map.
The purpose is the same as PRISM’s which is to
map the correct load and capacity values together
based on a geographic "cut set" boundary.
The cut sets are predetermined by many in the
planning process.
MARS modeling requires that the correct load
parameters and capacity parameters go into
several more areas than the PRISM assessment
work.
The mapping of the combination of the load and
capacity data into various zones can be a tedious
process, especially for newly defined zones.
Zone map changes can have significant impacts
as this is the foundation for all modeling.
As experience grows in assessing a given zone,
the previous efforts assist in assessing the data
quality of the given model being developed.
OLAP tools can significantly help assessing,
analyzing and reporting modeling and providing
observations that are hidden in conventional two
dimensional efforts (spreadsheets).
The basis for all assessment work is the definition
of the cut-set boundaries and the resulting load
modeled in each defined area.
10. Emergency Operating Procedure (EOP) Data
Modeled by manually adjusting the PRISM
case and external assessment work.
There is no direct input or requirement for this
data.
© PJM Interconnection 2011. All rights reserved
Page 60 of 136
The Emergency Operating Procedures are
required for most Industry models.
This is the EOP-DATA table.
There are typically six levels modeled for the RRS
studies and interregional work with the CP-8 WG.
These values are automatically inputted into the
text file via selections in the MARS GUI.
The two SAS tables involved are the EOPDATA
and EOPNAMES tables.
The six EOP levels are: 1) Operating Reserves, 2)
Load Management, 3) 30-minute reserves, 4)
Voltage Reduction, 5) 10-minute reserves and 7)
Appeals for public curtailment.
More data and maintenance efforts are needed for
the MARS modeling.
Coordination with the Transmission Planning and
Operation departments is needed for maintaining
this data base.
Increased coordination with neighboring region
staffs will improve the data quality for external
regions in the Study model.
Database
File
PRISM Database
MARS Database
Comments & Comparisons
11. Interface Transfer Limits
Defined in Schedule 4 (sheet 29) of the
Reliability Assurance Agreement (RAA).
A single transfer pipe (interface) size value of
3,500 MW that reserves this amount, over all
PJM tie lines with our neighbors, for
requesting emergency assistance from the
neighboring regions at the discretion of the
Operation staff.
A set of values that define the transfer limits
between two areas defined in the study model.
These SAS tables are used to automatically
populate the INF-DATA, INF-TRLM tables.
There are two SAS data base tables: INFDATA
and INFTRLM.
The MARS_SEASON SAS data base table
indicates what months are included in which of the
four seasons, used in the reporting of results.
The granularity for the PRISM table is greater
(weeks) than that used for MARS (Months).
These default parameters are different than those
used in PRISM.
The intent is to ease the typical use and
assessment options specified in the GUI to
effectively be the same effort as built in the PRISM
GUI.
However MARS has a smaller set of defaults that
have warranted definition.
These defaults include: 1) For the CNV-CRIT table:
tolerance, margin state to solve at, minimum
number of replications, maximum number of
replications, random number seed value, and 2) for
the GEN-CASE tolerance.
Because PRISM has been used repeatedly for
many years, more efforts have been made to
reduce the tedious, repetitive tasks especially
those that can cause errors in performing
assessments done by PJM staff members.
Consideration of these efficiencies when moving
to perform comparable MARS assessments
should be evaluated as many years of efforts has
increased the assessment efficiency using PRISM.
More data and maintenance efforts are needed
for the MARS modeling.
Coordination with the Transmission Planning and
Operation departments is needed for maintaining
this data base.
Need to assess and verify MARS assessments
can demonstrate use of 3500 MW CBM value,
per sheet 29 of RAA.
12. SEASONS
The seasons SAS data base indicates which
weeks are included in which of the four
seasons, used in the calculating and reporting
of results.
13. PARMS - Default Parameters
There are several and various default
parameters defined to ease typical use
specified in the GUI.
This includes the zone definitions, run
controls, sub zone ratios, zone rollups, and a
geography reference file.
SAS data base tables involved in the default
parameters include: DM_Geography_XREF,
DM_LAS_ZONES,DM_PJM_AREA_MAP,
DM_ZONE_RATIOS, DM_ZONE_ROLLUPS
© PJM Interconnection 2011. All rights reserved
Page 61 of 136
Summary of Tables 1, 2, & 3 Comparisons
Table 1 Observations
Table 1 shows a comparison of PRISM and MARS for various parameters, modeling characteristics, data
requirements and other considerations. The most prominent comparisons are summarized as follows:
PRISM
An advantage of PRISM is its use of statistical input data requirements. PRISM has relatively
few statistical parameters that incorporate the full model. PRISM also uses a forecast
probabilistic load model.
PRISM’s solution time is faster – and thereby allows time for additional sensitivity cases to
more fully assess system impacts.
With regard to resources, PRISM does not require any additional PJM staff, beyond the
Resource Adequacy Planning (RAP) department, to collect and perform a full resource
adequacy assessment.
The major weakness of PRISM is that it is limited to modeling the “World” as a single outside
entity. This limitation forces assumptions that are not fully supported by known operational
practices. Assessment by the 1993 PJM Load and Capacity Working Group (LCWG) verified
that modeling the world as a single outside entity was appropriate for LOLE studies. Noting
that review of transmission constraints and load diversity, having reserves below 10% might
prove a benefit of modeling a multi-area world region.39
PRISM cannot perform any hourly LOLE assessments, and cannot determine LOLH or EUE.
MARS
The significant advantage of MARS is its ability to model many areas, perform hourly
calculations and include parameters more directly related to Operations.
MARS is well-supported and in wide use for resource Adequacy planning throughout North
America.
MARS is comparatively time-intensive with regard to data collection and maintenance of the
required inputs.
Due to its inherent structure using Monte Carlo replications to meet the Standard Error solution
criteria, MARS does require longer solution times.
MARS output offers robust user-defined results for various assessment requirements.
In some MARS parameters, certain assumptions and approximations are used to populate and
maintain tables to represent specific operational scenarios.
MARS can be used to comply with any new energy metrics and forth coming requirements, per
the NERC Planning Committee directives. MARS can report the identified new metrics, LOLH
and EUE. Historically these two metrics have not been reported within the electric Industry for
bulk power systems.
Load diversity can be explicitly modeled for each area in a multi-area system wide
assessment. Each area’s load characteristics are not influenced by another area.
39
Evaluation of PTI’s Multi-Area Reliability Program MAREL - November 1993 - developed by the PJM
Load & Capacity Working Group (L&CWG)
© PJM Interconnection 2011. All rights reserved
Page 62 of 136
Table 2 Observations
MARS has several load level, Emergency Operating Procedure level, and Interface flow summaries that are
not available in the PRISM output. Additional PRISM load related summaries are currently being developed
by the PJM staff to report out the intermediate values at the 21 points of the daily peak load distribution.
PRISM has capacity distribution summaries that are not available in the MARS output. The MARS output, by
processing the OT10 and OT7 files, can be used to develop similar capacity summaries which include the
use of the post processing tool Histogram. Care should be taken so that any new summaries are
appropriate for the solution algorithm used (Analytical-PRISM, or Monte Carlo-MARS).
PRISM does use a database schema both for its input and output results. This allows a mapping of the
relationships between these data and is defined in an Online Analytical Processing (OLAP) metadata
process that allows assessment and reporting of several complex summaries.
Both tools offer many detailed outputs to perform LOLE assessments.
Table 3 Observations
An often overlooked aspect of LOLE assessment work is the importance and significance of the mantra “It is
all about the data!” In these data measuring efforts all aspects of the data, typically limited by identifying
significant characteristics and resource limits, ensure high data quality. Having high data quality allows for
confidence, correct interpretation of reported results, appropriate decision making and high accuracy of final
reported values. The final values are used in markets where even small changes can cause stakeholders
concern due to cost fluctuations to adhere to Adequacy requirements. Having an established production
grade environment database process mitigates fluctuations, and allows for appropriate changes that are
verifiable, tested and approved by the proper staff authorities. It conforms to a culture of compliance with
Standards and process that demonstrate by auditing by a 3rd party that the best practices are used in the
underlying data systems used to perform LOLE assessments.
The MARS work is performed using the same main GUI and underlying database as PRISM. This GUI is in
the ARC application (see the access to models: ARC screen display and GUI layout section). Due to the
general philosophy to use a database schema for all aspects of the Adequacy modeling and results, the
MARS text file is consistent with the modeling used for PRISM. Any adjustments to the modeling, to ensure
100% consistency, are done using a text editor (Text Pad), adjusting the input files used for PRISM (.dat file)
and MARS (MIF file – in05).
PRISM and MARS are both parts of the main ARC process used by the PJM staff. In this process there are
many data relationships established to assist automating a consistent model between the two tools. High
data quality is paramount throughout the established Adequacy assessment process- especially in light of
providing the most benefit at the lowest cost for all PJM members.
There are more categories of input data for MARS than for PRISM. This allows MARS more flexibility and
the potential to perform assessments that PRISM cannot perform. It also requires increased coordination
among PJM department staff and requires more time for the PJM ITS support staff to maintain and update
the data.
One aspect to consider is that while the PJM staff attempts to perform most of the Adequacy work, MARS is
likely to increase RAAS member technical representatives’ responsibilities and efforts. This is seen in MARS
assessment work by neighboring ISOs and potentially several of the increased input categories, used in
MARS, might be best determined by an on-going dialog with member technical representatives.
© PJM Interconnection 2011. All rights reserved
Page 63 of 136
Resource & Cost Assessment
The seven “topical” categories listed below were evaluated to compare staff resources and estimated costs
of conducting the PJM Resource Adequacy assessments. The values shown are ”ballpark” estimates only
— and shown for comparison only. For determining staff resource time, the Full-Time Equivalent” (FTE)
was used; this reflects the time a PJM Staff person would take to complete a given task. The labor rate
used for this analysis was assumed to be $75.00 per hour.
1.
Annual Reserve Requirement Study (RRS)
The Annual Reserve Requirement Study (RRS) has historically only been performed using
PRISM, with all the existing facilities geared toward PRISM requirements. It would take
significantly more resources to perform the RRS using MARS; mainly due to start up efforts to
accommodate the different model specifics and the experience level of the PJM staff. An initial
MARS assessment, using engineering estimates and appropriate engineering judgment could
enable a preliminary MARS assessment to gain experience in both the required modeling
impacts and how best to capture and report the final results of the assessment. An initial
MARS assessment (using estimates) might give insights concerning further use of MARS for
the RRS.
2. Capacity Emergency Transfer Objective (CETO) Analysis
Due to several aspects for new modeling and operational specific parameters, it is anticipated
that using MARS for this would require significantly more resources. Both applications will
result in a final CETO result which could be used in the RTEP and RPM stakeholder process.
MARS efforts would need to have verification, several checks and scrutiny which accounts for
the significant increase in resources.
3. Short-Term MARS Investigations
The efforts outlined are from previously identified PJM staff work tasks. This identifies and
increases the priorities for these efforts so that they can be considered for possible completion
in the future. See Appendix G for further details on this topic.
4.
Winter Weekly Reserve Target Analysis
Currently PRISM is lacking an appropriate load model for weekly assessment work as required
in this analysis. Developing a PRISM load model useful in weekly assessments is not justified
by this sole analysis effort. MARS is coordinated with PRISM; see Table 3 concerning the
relationship of the models, so that the typical MARS assessment is consistent with the PRISM
model and other PRISM assessment work.
5. Ambient Derate Analysis
The existing UNT-DERT table in MARS has potential application to quantify LOLE impacts of
any events not already covered in GADS reporting. PRISM must address this issue by external
assessment work; requiring operational data not typically used in the other RRS assessment
efforts. Identifying and using the operational data is anticipated to be a significant effort.
© PJM Interconnection 2011. All rights reserved
Page 64 of 136
6. “World Region” Multiregional Analysis
PRISM cannot address more than two regional assessments. Through the coordination efforts
and established model relationships between PRISM and MARS, it is anticipated that this
MARS assessment would require a medium amount of resources.
7. Intermittent Resources
PRISM does not have the ability to model intermittent resources on an hourly basis.
Intermittent Resources include types such as hydro, wind and solar units. The MARS efforts
outlined represent initial efforts. At some point a thorough technical assessment should be
considered as these types of resources become a greater percentage of the PJM generation
fleet.
Table - Resource & Cost Summary
PRISM Resources
Task Description
FTE (hours)
DEV
PROD
MARS Resources
COST
DEV
FTE (hours)
COSTS
PROD
DEV
PROD
DEV
PROD
1
Annual Reserve
Requirement Study
1795
$$$
3547
2100
$$$$$
$$$$
2
Capacity Emergency
Transfer Objective (CETO)
Analysis
728
$$
3020
968
$$$$$
$$$
3
Short-Term MARS
Investigations
4
Winter Weekly Reserve
Target Analysis
5
Ambient Derate Analysis
6
7
700
$$
$
72
$
$$$
544
$$
“World Region” Multiregional
Analysis
624
$$
Intermittent Resources
948
$$$
FTE:
2680
112
$$$$
1160
Full-time Equivalent in hours … Labor rate used for cost estimation =$75 per hour)
st
DEV:
Development Environment – environment where all initial efforts are performed, 1 environment to
demonstrate initial successful results before moving to the Test environment and finally to the Production environment.
PROD:
Production Environment – final environment used for approved assessment work. Moving
applications into this environment requires adherence to a comprehensive and structured change management process.
Cost Key:
Very Small
Small
Medium
Large
Very Large
$
$$
$$$
$$$$
$$$$$
= < $10,000
= $10,000 - $55,000
= $55,000 - $150,000
= $150,000 - $250,000
= > $250,000
The above Task 1 comparison estimates did receive other ISO feedback, from SME(s). Because other’s
MARS estimates were in the general ball park, the Resource and Cost summary table values were deemed
reasonable. The other task estimates (2-7) were not reviewed further assuming they are reasonable too.
© PJM Interconnection 2011. All rights reserved
Page 65 of 136
1. Annual Reserve Requirement Study
PRISM
Task Description
MARS
FTE
Production / Maintenance
FTE
Development
FTE
Production / Maintenance
Modeling and Database efforts
595
800
640
Assessment analysis
Reporting
Training and development
520
320
120
750
560
447
480
320
167
Interaction with SME/Vendor/ Support staff
240
400
333
Duration
Identified Outside Resources
6.7 months
None
8.7 months
6.7 months
GE staff, Operations Staff,
Operations Staff, Transmission
Transmission Planning staff, ITS
Planning staff
staff
Outside Resource effort
0
240
80
New Data efforts
0
350
Successful completion Measure
Similar RRS report as 2009 in
time for stakeholder review and
use by PJM Board of Managers
Not defined (yet) ... consider
NYISO, ISONE, NPCC, MISO
reports as starting template.
Successful Completion Due date
Mid-Sept
Mid-Sept
Similar as those shown for
PRISM but needs clarification
and specifics as it applies to
additional MARS modeling and
assessments for input items.
80
Report as defined in initial effort
in time for stakeholder review
and use by PJM Board of
Managers
Mid-Sept
Similar as those shown for
PRISM but needs clarification
and specifics as it applies to
additional MARS modeling and
assessments for input items.
Milestones
Collection of Outside region data,
Generation Owner Review,
Approval of assumptions, draft
report to RRAWG for review
Total Identified FTE
1795
3547
2100
Cost (Assumed average rate)
$$$
$$$$$
$$$$
Comments
Depending on
Model
considerations
Depending on
Model
considerations
Notes:
FTE = Full Time Equivalent in hours. Labor rate used for cost estimation = $75 per hour .
Model considerations include assumptions for GADS event data, estimating regions EOP-DATA table values, estimating INF-TRLM table values, LOD-UNCY table
values, and design year values for in02 file.
Primarily data gathering efforts.
© PJM Interconnection 2011. All rights reserved
Page 66 of 136
2. Capacity Emergency Transfer Objective (CETO)
PRISM
MARS
FTE
Production / Maintenance
160
FTE
Development
200
FTE
Production / Maintenance
200
Assessment analysis
320
500
400
Verification checks
Reporting
Training and development
160
40
16
160
200
800
160
40
16
Task Description
Modeling and Database efforts
Interaction with SME/Vendor/ Support staff
32
500
32
1.5 months
None
9 months
Operations Staff, ITS staff
1.5 months
Operations Staff, ITS staff
Outside Resource effort
0
160
40
New data coordination efforts
0
500
80
Successful completion Measure
Full 24 LDA completed by TP
staff deadline for RTEP(TEAC
mtg) and RPM posting
Not defined(Yet). Consider
NYISO, ISONE, NPCC, MISO
reports as starting template.
Full 25 LDA completed by TP
staff deadline for RTEP(TEAC
mtg) and RPM posting
Successful Completion Due date
On-Going
On-Going
On-Going
Duration
Identified Outside Resources
Milestones
Coordination with PSSE
machine lists, incorporation of
latest load forecast data,
Updates of CIR values,
generation Queues,
uncommitted resources.
Redevelop automation for quick
consistency and repeatability: Coordination with PSSE machine
Coordination with PSSE machine lists, incorporation of latest load
lists, incorporation of latest load forecast data, Updates of CIR
forecast data, Updates of CIR
values, generation Queues,
values, generation Queues,
uncommitted resources.
uncommitted resources.
Total Identified FTE
728
3020
968
Cost (Assumed average rate)
$$
$$$$
$$$
Notes:
FTE = Full Time Equivalent in hours
One delivery year assessment, related to RPM or RTEP
Primarily data gathering efforts.
© PJM Interconnection 2011. All rights reserved
Page 67 of 136
Comments
This is for 25 LDAsadditional effort needed
to produce consistent
load model
Need evaluaion and
updates for EOP table
values
3. Short-term MARS Investigations
MARS
FTE
Production Environment
Task Description
Investigate NYISO Unified Methodology
80
Investigate ISONE comparison of Westinghouse model with MARS
60
Interpret and develop reporting uses for data in OT6,OT7,OT8,OT9, and OT11
files
Investigate MISO use of reporting IRM with MARS
Compare and analysis in02 file shapes reporting differences seen in 2002
calendar year, 2005 calendar year, 2006 calendar yr, and design year values.
Duration
Consider items related to neighboring region model and assess impacts
60
80
240
3 months
Cooperation with: NYISO staff,
ISONE staff, MISO staff, SERC
staff
LOD-UNCY values - Load uncertainty
80
EOP-DATA values - Emergency Operating Procedures levels
100
INF-TRLM values - Transmission pipe sizes
60
Specific Path related contracts
60
Severe case assumptions or sensitivity assumptions
60
Evaluation to aggregate supplied details
200
Successful Completion Due date
Mid-August
Total Identified FTE
700
Cost (Assumed average rate)
$$
Notes:
FTE = Full Time Equivalent in hours
© PJM Interconnection 2011. All rights reserved
Page 68 of 136
Comments
Known documenation and
SME
NERC LOLE WG agenda
item
Started as part of this paper
Design year needs
development efforts by PJM
staff
Assumed cooperation and
availablity from neighboring
region staffs
Some efforts performed on
this topic
Some efforts performed on
this topic
Some efforts performed on
this topic
Some information available
for northern paths
Development work needed as
this would be a new effort
4. Winter Weekly Reserve Target Analysis
MARS
PRISM
FTE
Development
FTE
Production / Maintenance
Modeling and Database efforts
800
40
Assessment analysis
Reporting
Training and development
320
80
240
32
16
0
Task Description
Interaction with SME/Vendor/ Support staff
Duration
Identified Outside Resources
Milestones
24
32
16
0
500
16
0
1.5 Weeks
1.5 weeks
None
None
240
8
Need to develop a new weekly
load model to replace TRUE.
0
0
Need to develop a new weekly
load model. Previously TRUE
was used, investigated use of
model used by ITROM
application used by load
forcasters.
0
New Data efforts
Successful Completion Due date
Need to develop a new weekly
load model to replace TRUE.
FTE
Production / Maintenance
12 months
Operations Staff, Load
Forecasting Planning staff, ITS
staff
Outside Resource effort
Successful completion Measure
Comments
500
Similar RRS reported section Similar RRS reported section
as in 2009. In time for for PC
as in 2009. In time for for PC
stakeholder review and
stakeholder review and
recommendation to OC before recommendation to OC before
Winter period.
Winter period.
Mid-Oct
Mid-Oct
Completion of RRS base case Completion of RRS base case
and sensitivity analysis,
and sensitivity analysis,
producing MARS model using producing MARS model using
ARC
ARC
Similar RRS reported section as
in 2009. In time for for PC
stakeholder review and
recommendation to OC before
Winter period.
Mid-Oct
Completion of RRS base case
and sensitivity analysis,
producing MARS model using
ARC
Total Identified FTE
2680
112
72
Cost (Assumed average rate)
$$$$
$
$
Notes:
FTE = Full Time Equivalent in hours
Model considerations include assumptions for GADS event data, estimating regions EOP-DATA table values, estimating INF-TRLM table values, LOD-UNCY table values, and
design year values for in02 file.
Primarily data gathering efforts.
TRUE = Temperature Related Universal Effect - load model
© PJM Interconnection 2011. All rights reserved
Page 69 of 136
5. Ambient Derate Analysis
PRISM
FTE
Production / Maintenance
Task Description
MARS
Comments
Need operational data
and compilation of
statistical measures
FTE
Production / Maintenance
Statistical Modeling and Database efforts
1000
Use deration matrix to assess Adequacy impacts
N/A
160
80
40
4 Months
Operations Staff
40
0
Reporting similar to previous
efforts
Mid-Jul
80
40
2 Months
Operations Staff
24
0
Establish method, evaluation
scenarios
Mid-Jul
Total Identified FTE
1160
544
Cost (Assumed average rate)
$$$
$$
Reporting
Training, development, documentation
Duration
Identified Outside Resources
Outside Resource effort
New Data efforts
Successful completion Measure
Successful Completion Due date
Milestones
Notes:
FTE = Full Time Equivalent in hours
N/A = Not Applicable
© PJM Interconnection 2011. All rights reserved
Page 70 of 136
240
Comments
Need operational data
and compilation of
statistical measures
MARS uses the UNTDERT
6. "World Region" Multiregional Analysis
MARS
FTE
Production Environment
Task Description
Create initial single area that is consistent with model used in PRISM for the
neighboring region.
Divide single area model into its components, based on Operational
considerations, and Interregional group's modeling. Verify consistentcy for
infinate Transmision limits and sharing of resources
Develop all needed modeling table values, UNT-DATA, INF-TRLM, EOP-DATA,
LOD-UNCY, RES-SHAR
Training, development, documentation
Reporting
Duration
Identified Outside Resources
Outside Resource effort
Coordinate data model with neighboring regions
Successful completion Measure
Successful Completion Due date
Milestones
120
160
160
40
40
2 Months
MISO Staff, SERC Staff,
NPCC Staff, NYISO Staff,
ISONE Staff
24
120
Establish method, evaluation
scenarios
Mid-Aug
Total Identified FTE
624
Cost (Assumed average rate)
$$
Notes:
FTE = Full Time Equivalent in hours
N/A = Not Applicable
PRISM is not capable to perform this multi-Area assessment
© PJM Interconnection 2011. All rights reserved
Page 71 of 136
Comments
7. Intermittent Resources
MARS
FTE
Production Environment
Task Description
Develop modeling method for wind units, using hourly profile(s).
Develop modeling method for Hydro units, based on GADS data.
Investigate and appropriately use previous assessment work documented in
technical assessments (EWITS) and other ISO methods.
Investigate Manual 21, Appendix B method and calculations for existing wind
units. Evaluate Adequacy impact of using a perfect performing unit at a reduced
rating in comparison of availability shown in hourly profiles for full nameplate
rating.
Training, development, documentation
Reporting
Duration
Identified Outside Resources
Outside Resource effort
Coordinate data model with neighboring regions
Successful completion Measure
Successful Completion Due date
Milestones
120
160
160
320
60
60
2 Months
MISO Staff, GE Staff, NYISO
Staff, ISONE Staff
8
120
Establish method, evaluation
scenarios
Mid-Dec
Total Identified FTE
948
Cost (Assumed average rate)
$$$
Notes:
FTE = Full Time Equivalent in hours
N/A = Not Applicable
PRISM is not capable to perform this hourly model assessment
© PJM Interconnection 2011. All rights reserved
Page 72 of 136
Comments
Frequently Asked Questions (FAQs)
In the conduct of this study, there were many questions by RAAS participants and Stakeholders. These
questions were placed into five categories as follows:
Category 1: MARS as the primary LOLE tool
Q1. What factors most concern PJM for replacing PRISM with MARS? What impacts would be seen in
time and economic resources?
PRISM is a well-defined, valid and thoroughly known product for PJM that offers its staff great
efficiencies in performing assessment work due to its ongoing development over many years. Based on
an initial assessment, PJM Staff has determined that performing MARS assessments will ultimately
yield the same (or very similar) results as PRISM. Study outputs are defined in the PJM agreements
and marketplace requirements (see the RAA, RPM Manual 18 and RTEP Manual 14B).
A detailed MARS assessment is considered essential, due to several technical issues involved in
translation of PRISM load modeling to MARS modeling. There are many details to consider for full
MARS implementation at PJM; all such related study efforts have not yet been uncovered or fully vetted
through the technical stakeholder process.
For further information, refer to the following sections: Comparative Strengths, Complementary Aspects
of PRISM and MARS, and Ongoing LOLE Assessment Work.
Q2. Why is PJM reluctant to replace PRISM with MARS?
PJM is not reluctant to use MARS as a tool, as it started to implement MARS in 2004. PJM actively
uses MARS in performing LOLE assessment work. PJM staff has been involved with Industry models
and assessments for many years; it is PJM’s position that the proper tool should fit the task at hand. In
these efforts, PJM Staff invites opportunities to review the other known Industry tools such as Gridview,
SERVM, and the Westinghouse model.
In PJM’s view, no single tool emerges with a “one-size fits all” advantage for all the LOLE assessment
desired by the PJM staff and its Stakeholders. In evaluating and recognizing the complementary aspect
of these modeling tools, PJM Staff strives to employ a consistent and yet flexible assessment strategy.
PJM Staff actively implemented MARS on the PJM production grade systems which started back in
2002 (with previous efforts to evaluate a multi-area tools launched in 1987). For the past decade, PJM
has recognized the benefit and value that a multi-area tool such as MARS brings to LOLE assessment
work.
See sections Comparative Strengths, Complementary Aspects of PRISM and MARS, and Ongoing
LOLE Assessment Work for further details.
Q3. Please recap the discussion at the December 3, 2010 RRAWG (predecessor to RAAS) meeting
concerning the MARS solution process.
Mr. Glenn Haringa, of General Electric International, Inc. (GEII), the developing company of MARS,
joined the December 3, 2010 RRAWG meeting to clarify and explain aspects of the MARS solution
process.
A.
How does MARS’ “pipe” values come into play for transmission limits? Please relate
this to FCITC values determined in load flow assessments.
© PJM Interconnection 2011. All rights reserved
Page 73 of 136
There are load flow based assessment
techniques that can help determine the pipe
size (summation of Individual Interface tie
sizes) transfer limits used by MARS. In
addition to observing the limits on the
individual ties, MARS also allows you to
define interface groups (input table INFGRPS) which allow for the modeling of
simultaneous transfer limits. The interface and
interface group transfer limits can be varied
hourly based on the availability of specified units and the value of area, pool or system loads
(input table INF-DYLM per MARS User Manual). The degree to which the interface and
interface group limits are modeled can increase total solution time.
The extracted image above shows an example of interface groupings and dynamic limits for
the Long Island Area in the New York Control Area (CA).
B.
How does MARS address or model diversity in its assessment?
The load diversity within a given regional model is determined by the timing of the loads in the
various subareas that comprise the regional model. Due to different geographical parameters
for a given hour, not all subareas typically peak at the same time, MARS inherently addresses
this by its use of an hourly chronological load shape (in02 file), specified for each modeled
subarea. The issue of diversity is addressed early in the MARS modeling process with a
separate and up-front assessment to determine the appropriate 8760 hourly loads (load
shape). The area’s month-to annual ratios (LOD-MTAR table), as determined by load
forecasting reports, can also influence these deterministic hourly load shapes.
C.
Are there any tools or techniques for choosing the 8760 hourly load values?
Hourly loads determine MARS’ load shape characteristics. The selection of the hourly loads to
use involves performing assessment work related to load forecasting combined with historical
peak demand periods. This effort includes weather and economic uncertainty (I’m not sure
what these would include). This assessment is uniquely focused on the LOLE modeling of
peak load demand, but associated with load forecasting. In this assessment are considerations
of the demand periods being both severe (high) and also high for a long duration. Different
operational concerns exist for single peak demand periods as opposed to many peak demands
over extended time periods (i.e. high daily peaks for two weeks).
The Industry groups cited in the Interregional Assessments section of this report have
practices and established methods to choose this input modeling parameter. This topic has
been discussed at length at the NPCC CP-8 WG, and due to different regional operational
issues, the techniques vary.
D.
How does MARS uses its linear program algorithm to reach a solution? Explain how it
performs reserve sharing but not load loss sharing.
The presentation slides from the December 3, 2010 RRAWG meeting included three slides (9 12) that illustrated the MARS solutions and the Standard Error (These slides are Figures 6A6C in this report). An orange line shows how the LOLE solution relates to the Standard Error.
As Mr. Haringa explained, the solution (number of replications) depends on, among other
factors, the reliability of the system being modeled. A more reliable system therefore has a
slower convergence.
The MARS solution results are unique because of the algorithm used, and repeatable due to
the way in which the random numbers that are used to simulate random events are generated.
In this system- wide solution for large regional models, the default is to share assistance
between areas on an equitable basis. This approach mitigates individual areas’ LOLE states,
without the need for a pre-determined priority. This approach prevents an area from providing
assistance to a deficient area to such an extent that it negatively impacts the LOLE state of the
assisting area. This avoids making a decision as to which areas have priority in reserve
© PJM Interconnection 2011. All rights reserved
Page 74 of 136
sharing over others in receiving assistance. However, MARS has capability of specifying a
priority list for assigning known assistance priorities among the areas that comprise the large
regional model (Table RES- PRIO).
The MARS solution does not model or simulate the ability to share loss of load events among
areas. In a consistent manner it is important to note that the PJM RTO or other balancing
authorities do not invoke load shedding to provide emergency assistance to neighboring
control areas.
E.
From an academic viewpoint, theoretically, can MARS have more than one unique
solution for a large regional model?
A given MARS model, as specified by the input data, will have one unique solution. A large
regional model has many moving parts, involving generation, load, and transmission. These
work closely together in a coordinated, consistent and robust basis. In a bulk grid system there
are several modeling techniques for the successful grid operation (i.e. increasing a unit’s
output can alleviate a transmission limit).
As several coordinated, interrelated modeling aspects are addressed, there can be different
but appropriate results due to the different handling of the modeling in the solution process. In
general, these different scenarios and approaches yield similar trends and insights’ regarding
Bulk system needs and attributes. By choosing different subareas and regional cut-sets, the
total regional model does not change. Defining different subareas or redefining attributes of the
same subareas of this total region can however give different results. The solution depends on
the input data which is dependent on the study assumptions.
The total regional LOLE is associated with the lowest common denominator or the subarea
having the highest LOLE. If a subarea with a high LOLE is combined with a subarea with a
very low LOLE, the overall regional LOLE can improve significantly, but you have effectively
eliminated the transfer limits that had previously existed between the two areas. For example
in a system wide model, there are 4 areas each with 20% IRM and a fifth area with 5% IRM. If
these areas are re-combined into 3 areas, the resulting LOLE could be much lower than the
five-area model (the fifth area drives the resulting LOLE). However, the choice for planning
models is dependent on actual operational limits and methods used by the Dispatchers in
controlling the Bulk grid. (I.e. so combining the 5 areas into 3 might be inconsistent and
inappropriate to match operation practices).
Many scenarios are performed in Industry studies that address various modeling assumptions.
Upon review by experienced Engineering staffs and technical oversight groups (such as the
RAAS ) a comprehensive interpretation of all assessments can be given for use in Planning,
Markets, and Operations.
Q4. What steps are needed to encourage adoption of MARS as the LOLE tool of choice, going forward?
PJM staffs have currently implemented and use MARS. Over the years historic PJM groups have
endorsed a multi-area modeling tool. Due to resource and cost considerations, PJM has maintained its
internal PRISM tool as the primary tool. Over time some stakeholders have come forth to promote
MARS as the primary tool. However other stakeholders express caution indicating that PRISM
addresses the current Adequacy needs. The stakeholder process should be deliberate and
comprehensive, on technical grounds –with consideration of resources. Many stakeholders believe that
PRISM is a proven tool which meets PJM Adequacy needs in a cost effective and timely manner.
It is acknowledged, on technical merits, PRISM has limitations –and that MARS is needed to address
those limitations. While all stakeholders do see merits and strengths in using MARS, the debate is if
MARS should replace PRISM fully or if the current hybrid system (using both tools) be maintained.
Appendix G outlines several items for possible future consideration. The Resource and Cost
assessment section indicates the level of commitment needed to fully implement MARS to replace
PRISM. If some of the identified efforts can be prioritized and sub divided into phases, the transition
toward a full MARS implementation could be realized with a gradual cost. This subject was not the
focus of this technical comparison: providing any conclusions or recommendations for next steps was
avoided, as directed by the RAAS.
© PJM Interconnection 2011. All rights reserved
Page 75 of 136
At the first MARS user group meeting (October 2010), GEII staff announced the development of a stateof-the art GUI, database, and output reporting for MARS. Currently MARS has a text based input and
output format that comes from its 1980’s development. Improvements of the efficiency and
effectiveness of any new MARS product; using a state of the art GUI, database schema, and output
reporting which significantly enhances the PJM staff use; might prove a good point in time for further
incorporation into PJM’s ARC application.
The interregional planning efforts involving PJM staff might also prove to be the catalyst for MARS to
evolve as the primary tool at PJM.
Category 2: MARS resource needs and computing speed
Q5. Does running MARS take more “horsepower”, including computing power and manpower, to perform
studies?
Well, yes and no. It depends on what organization is running the MARS assessment work, the
experience and skill of the staff involved in the study, and the specific assessment at hand. MARS does
need many specific discrete table entries that might require additional assessment work compared to
PRISM.
Experienced staff performing MARS LOLE assessment work can considerably mitigate any burden due
to needed extra effort. For legitimate concerns in the output tables, deeper investigation into the issue
can be time-consuming and yet, cannot be avoided.
PRISM does have an advantage in that its input parameters are statistically-derived parameters from
the base information that both applications need and use. These statistics are defined in the Industry,
like IEEE Standard 76240, and result in fewer input sets required for the analysis to begin. Strict process
adherences to Standards ensure a high level of accuracy in application of the PRISM calculations.
However these statistical quantities do require an experienced staff, versed in statistics, and their
application in order to properly conduct a Study.
Q6. Could MARS runs be processed faster (with fewer resources) than what GE or PJM has already
indicated? – (As shown in the Resource and Cost Assessment Section)
As part of the “Next Steps” process, test runs can be made to perform some basic run-time
comparisons. Work of this nature can be rather complex and labor-intensive – and it would have to be
scheduled and approved by PJM management. Resources required to do such studies can vary widely
and there are many factors that can govern completion. The numbers referenced in this report have
been reviewed by PJM Staff involved in LOLE assessment work, and by other SME experts using
MARS – and are PJM’s best estimates per that review. Other industry SMEs experienced with MARS
were also solicited for their feedback about these estimates based on their MARS experience – and that
feedback is shown in the discussion about these best estimate values.
40
IEEE Standard 762 refers to IEEE Standard definition for use in reporting electric generating unit reliability, availability and
productivity. IEEE Std-2006 (revision of IEEE STD 762-1987) - approved 9/16/2006 by IEEE-SA Standards Board.
© PJM Interconnection 2011. All rights reserved
Page 76 of 136
Q7. What important details cause MARS to have long run times?
The MARS solution is iterative, as it requires a Monte-Carlo solution, while the PRISM solution is a
direct table look up technique. It would be expected that PRISM would have a faster solution time. This
is similar to how a load flow (i.e. Newton-Raphson) technique compares to a short-circuit study (i.e.
Thevenin equivalent) technique. The question is then: Are the two tools comparable in solution times?
Many modeling and specifics of a given assessment come into play for computer resources needed to
reach a solution. The issue discussed concerning dynamic interface transfer limits indicates a MARS
increased solution time. Not every study requires this modeling. If it is desired to compare single area
solution times, it would be expected that the MARS solution would have comparable solution times as
PRISM, with both solving in a reasonable time period (i.e. in a few minutes).
Figure 3C shows the overall MARS solution steps with step 7 iterating, due to dynamic transfer limits,
causing a slow solution for a large region, many area model (Process flow step: Calculate Emergency
assistance and resulting area margins which involves the dynamic interface transfer limits.) .
Q8. What are the details of why MARS requires much more staff time than PRISM?
PRISM has several staff efficiency advantages; 1) an established database schema; 2) many
automated administrative functions for maintenance and 3) user GUI options. MARS has been
incorporated into this environment, but in a supporting role. MARS’ additional data needs (more
granular modeling) require more work for the PJM Staff. This is primarily due to the Planning,
Operations and Market division staffs having an individual focus for detailed assessments. Coordination
of these individual PJM divisional assessments is currently done on a higher level (don’t get lost in the
weeds) to ensure consistent modeling.
As stated in this report, PRISM models are not necessarily suited for operation specific interpretations.
This mitigates detailed discussions and efforts about specific modeling parameters between the various
PJM divisions. However MARS models can be suited for operations specific interpretations which would
increase coordinating efforts with potential changes to accommodate underlying data base schema
changes.
Q9. What activities are done for the hours listed in the MARS Resource and Cost Assessment section?
They seem way too large?
In looking at the first task in the Resource and Cost Assessment section, the Annual Reserve
Requirement study, there are several staff members involved who comprise various skill sets and
departments. These efforts require close coordination so that the desired Engineering technical
assessment capability be performed effectively and efficiently. Further details concerning the rows
shown: Modeling and database efforts; Outside Resource effort; Interaction with SME/Vendor/ Support
staff; are as followings. The value in parenthesis indicates how many individuals are involved.
Task Description(# of staff)
Administrative Staff (2)
Document editorial review
Posting, organization, and administration of PowerPoint, spreadsheet, word
document
Engineering Assistant(2)
Straightforward editorial review
Create Spreadsheet checks for data and assumptions
Create spreadsheet summaries of detailed results.
Create documentation items(screen Captures, How-To)
Measure and validate correct data
Database Administrator(2)
Run Extract-Transfer-Load (ETL) processes.
Develop and maintain SSIS routines.
Coordinate with RAP staff to maintain data source changes.
SAS Admin and ITS support staff(2)
Move SAS data
© PJM Interconnection 2011. All rights reserved
FTE Hours for all
staff / Annual
period
40
400
160
320
Page 77 of 136
Trouble shoot GUI/data issues
Create Change Management tests
Oversee Change Management process
Maintain SAS libraries/schemas/Tables
Coordinate ITS response to Users
Coordinate with RAP staff to maintain data source changes.
GUI / OLAP developer(1)
Develop new data processing
Maintain Data processing
Assist in moving solutions to PRD
Document and coordinate proposed changes with SME (RAP,CA,ITS)
Documentation(4)
Adhere to ITS Standards documents and format
Adhere to ITS Standards for coding
Create “How-To” documentation
Maintain existing data documentation
Testing and Change Management(5)
Development environment
Test Environment
Production Environment
Internal PJM data gathering and cleaning(4)
Load forecast
Generation Forecast
Transmission Forecast
EOP
Contracts
Data gathering for external sources: NERC, FERC, SERC, NPCC, MISO (3)
Load forecast
Generation Forecast
Transmission Forecast
EOP
Contracts
Data cleaning of external sources(2)
FERC 714 process Hourly Loads
NERC ES&D data
NYISO MARS Data
ISO-NE MARS data
NERC LTRA report
Transition to Old NERC boundaries(Pg 34-36 of 2010 RRS)
Coordinating with Operations staff(2)
Maintain data and review with Operations staff
Develop documentation of new data and review with operations staff to
establish process.
Additional Modeling of PJM control Areas(4)
Transmission planning assessments
a. Significant development effort requiring limited Engineering staff
resources.
b. Develop a MARS multi-area model that breaks the PJMRTO into is
discrete control areas which are driven by known operational
constraints.
Resource Adequacy Planning Senior Engineering Staff oversight (3)
One day of review per week over critical 3 month modeling period.
Provide examples
Formal meetings to coordinate modeling among Resource Adequacy Planning Staff,
Transmission planning department, Operation Staff, database administrators, SAS
developers, Java GUI developers, and staff involved in Change Management.(5)
Working meetings to informally gather in same room to share and draw on other’s
viewpoints while each individual task is be performed – aides in producing a consistent
product and keeping all involved on the same page.(5)
© PJM Interconnection 2011. All rights reserved
Page 78 of 136
160
80
80
30
60
60
24
320
250
120
416 – This is not
additional time, but
time allocated for
performing some of
the above tasks.
While this is not meant to be comprehensive, the above attempts to provide verification of the number of
hours shown in task 1 of the Resource and Cost assessment section. Similar details were provided by
the other ISO staffs verifying that the PJM estimates are in line with other neighboring staff efforts.
Q10. What incremental efforts are meant to be done by PJM related to items listed in the MARS Resource
and Cost Assessment section?
Although the PJM staffs are currently using MARS for several assessment topics, the full capability of
MARS within PJM’s system is not yet fully functional (i.e. it cannot replace PRISM). Items needing
improvement include:
Automated GUI parameters and options,
Addressing missing data in the model,
Maintenance of data and assessment methods,
Incorporation of other ISO’s MARS methods into an appropriate adaptation for PJM,
Measuring and calculation techniques for all the various MARS parameters,
Ensuring the input parameter package is complete and balanced.
A specific item would be to break the PJMRTO model into its internal Control Areas (six or more), from
the current four-area model. This involves Operations and Transmission Planning Staff efforts. There
are many significant non trivial efforts that would need to be done. An initial list is shown in Appendix G.
These are possible examples for what incremental efforts could be performed, with these items part of
the estimates in the Resource and Cost section.
Category 3: MARS Advantages
Q11. Does MARS represent a big advantage due to its obvious multi-area capability?
MARS not only has an advantage in multi-regional study work but can also perform hourly
assessments. MARS can model many specific operational events for close-in assessments. PRISM
should NOT be used for operational planning of bulk system requirements as its statistical methods may
or may not comply with specific actual operational experience. Typically the operational planning groups
use different techniques better suited for near term operational requirements. PRISM is a probabilistic
model well-suited for planning of future bulk grid requirements. Along with multi-area capability, the
ability to perform hourly chronological assessments and LOLH and EUE calculations will be important to
address reporting, considering the NERC-PC is requiring these items to be reported at a future date.
MARS Monte Carlo solution technique and its discrete inputs allow for evaluation of specific operational
conditions, with reported results usually indicating the number of invocations of a given Emergency
Operation Procedure (EOP) level or a confidence level of the observed assessment conditions.
Q12. Would the full implementation of MARS enhance PJM’s contribution to the interregional planning
initiatives such as the Interregional Planning Stakeholders Advisory Committee (IPSAC)?
Since 2005 PJM staff has used MARS to complement PRISM (see Q19 for more details).
PJM Staff is actively involved in performing these assessments and functions. At times, PJM staff takes
the lead in running all appropriate analysis and reviews such joint assessments. For some of these
efforts, PJM gathers all the modeling data and implements the MARS solutions – while at other times,
the other ISO staffs take project leads. The lead on any such efforts depends on available resources,
modeling and assessment expertise, given priority for a given ISO, availability of application tool that
best fits study objectives. MARS is useful for determining adequacy metrics such as LOLE, LOLH, and
EUE.
The IPSAC website includes agendas and details about these efforts. Other interregional efforts include
the NPCC CP-8 WG, and SERC’s RAWG, in which all interregional groups are currently using MARS.
http://www.pjm.com/committees-and-groups/stakeholder-meetings/stakeholder-groups/ipsac.aspx
© PJM Interconnection 2011. All rights reserved
Page 79 of 136
Q13. What is MARS’s biggest advantage?
MARS’ biggest advantage is that it can model an unlimited number of defined areas. This feature is
useful in assessing transmission system conditions and impacts on resource adequacy metrics such as
LOLE, LOLH, and EUE.
MARS does have other advantages, if: 1) the underlying hourly load data, 2) GADS events and 3)
operational data is available.
o
It can explicitly model operational events. This allows for adequacy assessments for
measuring various changes and scenarios to these events.
o
The numerous entries allow for reporting out on specific Operational define levels, in
accordance with defined Emergency Operating Procedures (EOPs).
o
MARS can capture, through its detailed input tables (See Appendix F), energy related
modeling specifics. This includes intermittent resources (Wind), demand side management
programs, and load shedding.
The typical MARS modeling practice of specifying many areas for the surrounding neighboring systems
is typically done by Interregional assessments. If transmission limits between the defined areas are
unknown, there could be significant cost and effort involved.
Category 4: Clarify MARS limitations
Q14. Why is the MARS’ output in flat text files (which have to be cut and pasted into Excel) for further
assessment efforts?
MARS has both Text and Binary final outputs. However almost all of the results, processed for reporting
LOLE results and calculations, are in the text format files. The binary file format is very useful in using
the histogram supporting application, to process the replication and capacity distributions. The PJM
Staff does not have or know of a better method to process the many MARS results in the various MARS
output tables. Using a manual text editor tool is an efficient method for single, one-off type of reporting.
However for more complex or iterative processing, this is manually intensive. There would have to be
process to define and store a database schema, for the various MARS output results. Similar to what
has been done for PRISM; this would mitigate the PJM manual efforts in processing and coordinating
results.
Potential changes to the underlying data base schema require costly and significant resources.
Q15. Why is MARS is limited to a 5 year look-ahead?
MARS does not have any time period limitations, from a modeling, calculation or processing
perspective. Results can be reported for any time period deem appropriate, by the user.
Other Control Areas report MARS results for a ten year forward period (i.e. CAISO, MAPPCOR).
However, as seen in the recent NERC PC decisions about approved metrics (by GTRPM Task Force),
operations based metrics (energy) are appropriate for either a 2 or 5 year look-ahead. The uncertainty
in forecasting values beyond that time period was deemed inappropriate.
MARS requires operational specifics to forecast the reported metrics (i.e. LOLE, LOLH, and EUE).
PRISM does not need these operational specifics, as its model uses forecast probabilistic parameters
for the individual components of load, capacity, and transmission. Hence the convolution and resulting
LOLE states are not solely contingent on any underlying operational condition or event. Note that
whatever the tool used the uncertainty of the reported forecast values increases as a function of time.
At some point the uncertainty of the reported metrics is deemed to be either not useful or causes a
misunderstanding of the reported information. MARS has been seen to be highly useful in the Industry
to report out clear, valid metrics 5 years into the future, based on the PJM Staff’s interregional study
experience to date.
© PJM Interconnection 2011. All rights reserved
Page 80 of 136
Q16. Why does MARS has many more input tables compared to PRISM?
Although both MARS and PRISM have complex, and specific input data requirements, MARS does
have 50 input table categories while PRISM has 23 input control card categories. Of the MARS 50 input
table categories approximately 22 tables, comprising 121 parameters, are typically in a large regional,
multi-area model. See Appendix F for further details.
While these MARS input details allows many variations and sensitivities to be assessed it also
necessities a detailed database schema to track and maintain any data changes, especially any
operational related parameters.
Category 5: PRISM Advantages
Q17. What is PRISM’s biggest advantage?
PRISM’s biggest advantage is that it is a quick and accurate solution, using a limited amount of
statistical input parameters (EEFORd, EFORd, Variance, and POF).
PRISM does have other advantages, if: 1) the underlying hourly load data and 2) GADS event data is
available.
o
PRISM’s use of statistical parameters mitigates any re-evaluation of the discrete
events/observations. These statistical quantities are Industry proven and have been
developed over many decades in the context of application of correct mathematical and
numerical techniques.41 42
o
The numerous entries required in MARS can lead to significant investigation and re-occurring
sensitivity analysis to quantify impacts in each study. For example, PRISM’s use of a single tie
size is an advantage in that extensive analysis was conducted in determining this value, and
this robust analysis continues per FERC requirements to report a Simultaneous Import Limit
(SIL) with the resulting Capacity Benefit Margin (CBM) value being one value agreed to in the
stakeholder process and stipulated in the PJM Agreements. It is clear and unambiguous what
CBM value should be used for the transmission pipe size. This removes any need to
investigate “what if” analysis, saving the PJM members time & money in each annual study.
PRISM’s modeling practice of placing the entire surrounding neighboring systems into one region has
been evaluated and discussed by the RAAS in-depth. A recent White paper, titled “World Modeling
Region – Technical Issues” on this subject is on the PJM web site at:
http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/res-adeq/20090515-worldmodeling-region-tech-issues.ashx .
It is recognized that several of the assumptions of this modeling can be investigated for improvement by
evaluation and assessment work using MARS’s multi-region capability.43
Q18. What specifics concerning PJM’s blended approach is used by both PRISM and MARS? Please
specify when each tool is used and the reason behind using this specific tool for the identified task
at hand.
PRISM (including the PLOTS load model) is used as the primary tool for: 1) annual RRS reporting,
CETO reporting, 2) assessment of the load model time period, and 3) most resource adequacy
sensitivity work. PRISM calculates the LOLE metric for demonstrating adherence to any defined criteria
or standard. PRISM and PLOTS are based on the existing database schema, which includes the
established data gathering efforts. Both of these tools are integrated into a GUI that increases the
efficiency and quality of PJM staff assessment efforts. There is currently no non-annual load model
(Seasonal or monthly) available as previous efforts were not carried over into the new SAS, ORACLE,
41
Power System Reliability Evaluation, - Gordon and Beach, Science Publishers, –1970 – by Roy Billinton
Probability Calculation of Generation Reserves - March 1969 - by C.J. Baldwin and published by The Westinghouse Engineer
Evaluation of PTI’s Multi-Area Reliability Program MAREL - November 1993 - developed by the PJM Load & Capacity Working
Group (L&CWG
42
43
© PJM Interconnection 2011. All rights reserved
Page 81 of 136
JAVA systems. The previous non-annual modeling was deemed to be not practical for implementation
going forward. Other modeling is seen as more appropriate (i.e.: Fuzzy logic techniques). This makes
PRISM useful for annual assessment efforts, as no seasonal or monthly focused modeling is done with
PRISM. The underlying database schema is foundational in nature. Changes to the database are
therefore resource intensive, and costly.
MARS is used for: 1) the winter weekly reserve target, 2) Wind and DR type assessment that need
hourly patterns that are energy based, and 3) Interregional coordination efforts with neighboring ISOs.
Another potential application for MARS could be to analyze the ambient impact on units and the
relationship to the LOLE. MARS is used in the winter target assessment as PRISM does not have a
non-annual probabilistic load model. MARS is used in the interregional coordination efforts as a
consistent tool avoids translations issues between different tools. MARS will be used to satisfy the
upcoming NERC reporting requirement of LOLH and EUE metrics.
For further discussion about why each tool is used, please see the section Complementary aspects of
PRISM and MARS. For further information about what is performed please see the sections Ongoing
LOLE assessment work and Database modeling relationship matrix.
© PJM Interconnection 2011. All rights reserved
Page 82 of 136
Interregional Assessments
NPCC Regional Assessment
The Northeast Power Coordinating Council (NPCC) conducted an Interregional Long Range Adequacy
44
Overview that used PJM’s 2005 RRS model as the basis for the modeling used for PJM in this MARS
assessment. This study is an example of a regional assessment to demonstrate a comparison of MARS results
with PJM Planning Division results. The graphs below reflect the results of this analysis.
The chart below is from Page 34 of this report and illustrates how LOLE increases through the Study time period.
During 2004-2006, PJM Planning Division assessments indicated reliability needs for system enhancements and
facility upgrades. Significant changes were planned as recorded in the RTEP process and the implementation of
the Reliability Pricing Model (RPM) marketplace. This chart, from MARS assessment work was consistent with
other PJM Planning Division work and gave appropriate valuations of impacts on an LOLE basis.
44
Northeast Power Coordinating Council “Interregional Long Range Adequacy Overview” (November 28, 2006), conducted by the
NPCC CP-8 Working Group. The full report can be obtained on the NPCC website
© PJM Interconnection 2011. All rights reserved
Page 83 of 136
IPSAC Regional Assessment
PJM participates in interregional planning activities with all its interconnected Planning Coordinator
neighbors and all Planning Coordinators throughout the Eastern Interconnection pursuant to various
interregional agreements. Two of these agreements specify the formation of an Inter-regional Planning
Stakeholder Advisory Committee (IPSAC) for the purpose of allowing for review of and input to
coordinated system planning activities by all stakeholder groups.
o
The Joint Operating Agreement between PJM and the MISO
o
The Northeast ISO/RTO Planning Coordination Protocol among PJM, the NYISO and the ISONE.
PJM activities related to these IPSAC groups and the related interregional agreements can be followed
at the web pages for the NYISO, MISO, PJM and others. Related meetings and information that are
organized and run by PJM will be shown as PJM meetings in the space below. Notice and information
of IPSAC meetings organized and run by others will be disseminated via email to the PJM IPSAC, PC
and TEAC “exploder” lists. Registration information for meetings run by others will be included with
those emails.
IPSAC past meeting materials are posted at: http://www.pjm.com/committees-and-groups/stakeholdermeetings/stakeholder-groups/ipsac-ny-ne.aspx ( see 12-18-2009 Agenda item “Interregional
Planning”, and 06-30-2009 Agenda item “NYISO / PJM focused Study” ).
IPSAC meeting materials related to the Midwest ISO are posted at: http://www.pjm.com/committeesand-groups/stakeholder-meetings/stakeholder-groups/ipsac-midwest.aspx .
PJM staff has performed MARS assessments which involved up to a 28-area model of the Northeast
(PJM, NYISO and ISO-NE) in these efforts. Some figures and summaries concerning MARS methods
and solutions in this Report are extracted from these detailed Industry modeling efforts.
SERC Reliability Assessment Working Group
The PJM staff participates in the SERC Reliability Corporation’s45 , Reliability Assessment Working
Group (RAWG). This does include review of MARS modeling and assessments performed for the
SERC region and its sub-regions. This modeling includes the neighboring- external systems. The
PJMRTO model represents portions of the SERC sub regions and a portion of the SERC neighboring
external region.
Participation in the SERC RAWG requires a signed NDA and adherence to confidentially rules as
defined in the SERC Reliability Corporation’s stakeholder process. PJM Staff has access to the secure
RAWG portion of the SERC web site due to compliance with these rules.
The public SERC web site is at: http://www.serc1.org/Application/HomePageView.aspx
45
SERC is one of nine regional electric reliability councils under North American Electric Reliability Corporation (NERC)
authority. SERC was formed in 2005, as the successor to the Southeast Electric Reliability Council (also known as SERC).
© PJM Interconnection 2011. All rights reserved
Page 84 of 136
Glossary
Also see PJM Manual 35, Definitions and Acronyms at:
http://www.pjm.com/~/media/documents/manuals/m35.ashx
Adequacy – the ability of a bulk electric system to supply the aggregate electric demand and energy
requirements of the consumers at all times, taking into account scheduled and unscheduled outages of
system components. One part of the Reliability term.
Applications for Reliability Calculations (ARC) – ARC is the principle computer environment that allows for all
LOLE Adequacy assessment work to be performed. There are five parts of this application: 1) Week Peak
Frequency (WKPKFQ); 2) PRISM; 3) MARS; 4) Administration (Security and database); 5) PIW view (PJM
Information Warehouse).
Bulk Power System (BPS) – The Bulk Power System (BPS) refers to all generating facilities, bulk power
reactive facilities, and high voltage transmission, substation and switching facilities. The BPS also
includes the underlying lower voltage facilities that affect the capability and reliability of the generating and
high voltage facilities in the PJM Control Area. As defined by the Regional Reliability Organization, the
BPS is the electrical generation resources, transmission lines, interconnections with neighboring systems,
and associated equipment, generally operated at voltages of 100 kV or higher. Radial transmission
facilities serving only load with one transmission source are generally not included in this definition.
Capacity – The amount of electric power (in megawatts) that can be delivered to both: 1) firm energy to load
located electrically within the PJM Interconnection and 2) firm energy to the border of the PJM Control
Area for receipt by others. Installed capacity (ICAP) and unforced capacity (UCAP) are related measures
of this quantity.
Capacity Benefit Margin (CBM) – the CBM (in megawatts) represents the amount of import capability that is
reserved for the emergency import of power to help meet LSE load demands during peak conditions and
is excluded from all other firm uses. This is defined on page 104 of the Reliability Assurance Agreement
(RAA) , posted October 14, 2010.
Capacity Emergency Transfer Limit (CETL) – CETL represents the sub-area’s actual import capability as
determined from power flow studies. The sub-area criterion is satisfied if its CETL is equal to or exceeds
its CETO (see below). PJM’s CETO/CETL analysis is typically part of the PJM’s deliverability
demonstration; see PJM Manual 20, and PJM Manual 14B, attachment C for details.
Capacity Emergency Transfer Objective (CETO) – CETO represents the sub-area’s import capability objective
required to satisfy the RFC’s resource adequacy requirement LOLE. CETO is an independent evaluation
that is done in a coordinated and consistent manner with the annual RRS.
Central Limit Theorem (CLT) – In probability theory, the CLT states conditions under which the mean of a
sufficiently large number of independent random variables, each with finite mean and variance, will be
approximately normally distributed. In more general probability theory, a CLT is any of a set of weakconvergence theories. The formal mathematical theorem states:
1) If you have a simple random sample of “N” observations from a population with mean µ and
a standard deviation σ. 2) And if “N” is large, then 3) the sample average is approximately
normally distributed with a mean µ and standard deviation σ / N ½.
Two important practical implications of the Central Limit Theorem are: a) even if the sample values are not
normally distributed, the sample average is approximately normally distributed, and b) because the
sample average is normally distributed, you can use the Empirical Rule to summarize the distribution of
sample averages.46
Complementary – Something that perfects, completes, or adds to something47. Both PRISM and MARS add its
unique features and capabilities to each other to enhance (add to) a given individual application of each
tool.
46
SAS System for Elementary Statistical Analysis, Second Edition – 1997 – by Sandra D. Schlotzhauer and Ramon C. Littell,
PHD, published by the SAS Institute Inc, Cary NC, USA, pages 170-171.
47
Webster’s Dictionary and Thesaurus, Deluxe Edition, Nichols Publishing Group, Copyright 2001.
© PJM Interconnection 2011. All rights reserved
Page 85 of 136
Control Area (CA) – Electric power system or combination of electric power systems bounded by
interconnection metering and telemetry. A common generation control scheme is applied in order to:
o
o
o
o
o
Match the power output of the generators within the electric power system(s) plus the energy
purchased from entities outside the electric power system(s), with the load within the electric
power system(s);
Maintain scheduled interchange with other Control Areas, within the limits of Good Utility
Practice;
Maintain the frequency of the electric power system(s) within reasonable limits in accordance
with Good Utility Practice and the criteria of the applicable regional reliability council of NERC;
Maintain power flows on Transmission Facilities within appropriate limits to preserve reliability;
and
Provide sufficient generating Capacity to maintain Operating Reserves in accordance with
Good Utility Practice.
Convolution – the term “convolution” is used to depict the relationship between two input distributions of load
and capacity and the resulting loss of load distribution. Engineering problems approach this term from
considering the system as a set of weighting coefficients. The convolution method is called the recursive
method. The load model curve is thereby “convolved” (joined) with the generation system model for
computing the LOLE.
CP-8 Working Group – An NPCC planning group that is focused on LOLE studies for NPCC regional
assessments, review and compliance to NPCC planning protocols and Standards (such as demonstrating
an LOLE of 0.1 days/Year) . This group consists of representatives from the five control areas within
NPCC with some participation from neighboring regions’ staff. Participation requires a Non-Disclosure
Agreement as confidential information is required in the group efforts.
CURTAIL – An analysis tool based on legacy techniques that evaluate the load management’s reliability benefit
in terms of LOLE. The number of load management interruptions and the amount of load management are
key inputs in determining the benefit for each load management interruption. Typical assessment using
this tool focus on the reliability value’s saturation (measured in terms of LOLE) of load management and
the maximum amount of load management that cannot degrade the reliability value when 10 interruptions
are invoked.
Cut-Set – A line or boundary that defines a geographical demarcation to distinguish, typically, a Locational
Deliverability Area (LDA). All electrical facilities within a cut-set boundary are modeled within this zone.
Delivery Year (DY) – The Delivery Year (DY) is the twelve-month period beginning on June 1 and extending
through May 31 of the following year. As changing conditions may warrant, the PJM PC may recommend
other Delivery Year periods to the PJM Board of Managers. In prior studies, the DY was formerly referred
to as the “Planning Period”.
Deliverability – Deliverability is a test of the physical capability of the transmission network for transfer capability
to deliver generation capacity from generation facilities to wherever it is needed to ensure, only, that the
transmission system is adequate for delivery of energy to load under prescribed conditions. The testing
procedure includes two components: (1) Generation Deliverability; and (2) Load Deliverability. (See PJM
Manual 14B)
Demand Resource (DR) – A resource with the capability to provide a reduction in demand. DRs are a
component of PJM’s Load Management (LM) program. The DR is bid into the RPM Base Residual
Auction (BRA). See Load Management (LM) and ILR.
Demand Resource (DR) Factor – Associated with the Forecast Pool Requirement (FPR) factor and specified in
the Reliability Assurance Agreement (RAA). This factor is annually approved by the PJM Board of
Managers. The factor is multiplied by the amount of demand resources to yield the resulting value used in
PJM Markets.
Demand – The rate at which electrical energy is delivered to or by a system or part of a system, generally
expressed in kilowatts or megawatts, at a given instant or averaged over any designated interval of time.
Demand is equal to load when integrated over a given period of time. See Load.
Deterministic System – A deterministic system has a single result or set of results for a given set of input
parameters. A deterministic algorithm is an algorithm which behaves predictably. Deterministic
mathematical functions always produce the same output given a certain input. (Compare to Probabilistic
System.)
© PJM Interconnection 2011. All rights reserved
Page 86 of 136
Diversity – Diversity is the difference of the sum of the individual maximum demands of the various subdivisions
of a system, or part of a system, to the total connected load on the system, or part of the system, under
consideration. The two regions modeled in the RRS are the PJM RTO and the surrounding “World”
region. If the model has peak demand periods occurring at the same time, for both regions (PJM RTO
and World), there is little or no diversity. The peak demand period values are determined as the Expected
Weekly Maximum (EWM). A measure of diversity can be the amount of MWs that account for the
difference between a Transmission Owner zone’s forecasted peak load at the time of its own peak and the
coincident peak load of PJM at the time of PJM peak.
Effective Equivalent Demand Forced Outage Rate (EEFORd) – is used for reliability and reserve margin
calculations. For each generating unit, this outage rate is the sum of the EFORd plus ¼ of the equivalent
maintenance outage factor. See PJM Manual 22.
Electricity Reliability Organization (ERO) – The generic name used in the U.S. Energy Policy Act of 2005 to
refer to the independent entity that would be given the authority to develop and enforce mandatory
reliability standards for the North American bulk power system. NERC was designated as this “electricity
reliability organization” by FERC on July 20, 2006. “ERO” refers to NERC’s role, but “ERO” is not an
official name.
Empirical Rule – If data is from a normal distribution, the Empirical rule gives a quick and easy way to
summarize the data. This rule states: 1) 68% of the values are within one standard deviation of the mean.
2) 95% of the values are within two standard deviations of the mean 3) more than 99% of the values are
within three standard deviations of the mean. For a normal distribution about 68% of the values occur
48
between µ-σ and µ+σ, corresponding to the Empirical Rule.
Equivalent Demand Forced Outage Rate (EFORd) – the portion of time that a generating unit is in demand,
but is unavailable due to a forced outage. See PJM Manual 22.
Equivalent Maintenance Outage Factor (EMOF) – for each generating unit modeled, the portion of time a unit
is unavailable due to maintenance outages. See PJM Manual 22.
Expected Unserved Energy (EUE) – is the expected number of MWh of load that will not be served in a given
year. EUE indicates the capability of the Transmission system to continuously serve all loads at all
delivery points while satisfying all planning criteria. Information required for computing EUE includes: a)
frequency of each contingency (outage/year), b) duration of each contingency (hr/outage) and c) unserved
MW load for each contingency.
Expected Weekly Maximum (EWM) – the weekly peak load corresponding to the 50/50 load forecast, typically
based on a sample of 5 weekday peaks. The EWM parameter is used in the PJM PRISM program. Also
see PJM Manual 20 pages 19-23.
Forecast Error Factor (FEF) – FEF is a value that can be entered in the PRISM program per Delivery Year to
indicate the percent increase of uncertainty within the forecasted peak loads. As the planning horizon is
lengthened, the FEF generally increases 0.5% per year.
Forced Outage Rate (FOR) – FOR is a statistical measurement as a percentage of unavailability for generating
units and recorded in the GADS. FOR indicates the likelihood a unit is unavailable due to forced outage
events over the total time considered. It is important to note that there is no attempt to separate out
forced outage events when there is no demand for the unit to operate.
Forecast Peak Load – the expected peak demand (Load) representing an hourly integrated total in megawatts,
measured over a given time interval (typically a day, month, season, or delivery year). This expected
demand is a median demand value indicating there is a 50% probability actual demand will be above or
below the expected peak.
Forecast Pool Requirement (FPR) – the FPR represents the total UCAP requirement (percent) for the PJM
Control Area — determined by taking 100% plus the percentage of IRM for the PJM Control Area. The
percentage FPR is required pursuant to the Reliability Assurance Agreement (RAA) and approved by the
Reliability Committee pursuant to Schedule 4 of the RAA.
GEBGE – GEBGE is a resource adequacy calculation program, used to calculate daily LOLE that was jointly
developed in the 1960s/1970s by staff at General Electric International, Inc. (GEII) and Baltimore Gas and
Electric (BGE). The GEBGE program has since been largely superseded and replaced by PJM’s PRISM
in the conduct and evaluation of IRM studies at PJM. (See PRISM.)
48
SAS System for Elementary Statistical Analysis, Second Edition – 1997 – by Sandra D. Schlotzhauer and Ramon C. Littell,
PHD, published by the SAS Institute Inc, Cary NC, USA, page 136
© PJM Interconnection 2011. All rights reserved
Page 87 of 136
General Electric International, Inc. (GEII) – GEII is a global infrastructure, finance and media company taking
on the world’s toughest challenges. From everyday light bulbs to fuel cell technology, to cleaner, more
efficient jet engines, GE has continually shaped our world with groundbreaking innovations for over 130
years. See the GEII web site for further details.
Generator Availability Data System (GADS) – As defined in the NERC Generating Availability Data System
(GADS) (http://www.nerc.com/page.php?cid=4|43 ), PJM fully supports GADS using the eGADS web
based application. All generating facilities taking part in the PJM markets are required to submit unit
statistical performance and reliability data to determine the value of the facility as an unforced capacity
(UCAP) resource, by using PJM’s eGADS application found as one of the eSUITE applications available
on the PJM web site (https://esuite.pjm.com/mui/ ).
Generator Forced / Unplanned Outage – An immediate reduction in output, capacity, or complete removal from
service of a generating unit by reason of an emergency or threatened emergency, unanticipated failure, or
other cause beyond the control of the owner or operator of the facility. A reduction in output or removal
from service of a generating unit in response to changes in or to affect market conditions does not
constitute a Generator Forced Outage.
Generator Maintenance Outage – The scheduled removal from service, in whole or in part, of a generating unit
in order to perform necessary repairs on specific components of the facility approved by the PJM Office of
Interconnection .
Generator Planned Outage – A generator planned outage is the scheduled removal from service, in whole or in
part, of a generating unit for inspection, maintenance or repair – with the approval of the PJM.
Graphical User Interface (GUI) – a GUI is a Java-based web environment that allows the user to point and click
selections to create a desire model, assessment options, and analysis parameters. Supported and
maintained by the PJM Information Technology Services (ITS) staff.
Institute of Electrical and Electronic Engineers Inc. (lEEE) – the IEEE is a non-profit professional association
dedicated to advancing technological innovation related to electricity. IEEE is chartered to advance the
theory and practice of Electrical, Electronics, Communications and Computer engineering, as well as
computer science, the allied branches of engineering and the related arts and sciences. IEEE serves as a
major publisher of scientific journals and a conference organizer. It is also a leading developer of industrial
standards (having developed more than 900 active industry standards) throughout a broad range of
disciplines. IEEE has more than 395,000 members in more than 160 countries, 45% outside the United
States.
Installed Capacity (ICAP) – commonly refers to “iron in the ground” – or rated capacity of a generation unit (in
megawatts) prior to derating or other performance adjustments.
In02 file – One of the 3 necessary MARS input files that have the 8760 hourly loads used in the calendar year
assessment. This is a discrete distribution of the load model shape used in the Monte Carlo simulations.
A separate set of 8760 values is needed for each area in the model. This is a text file, using the historic
EEI format for loads (80 columns, 12 hours (am) on 1st line, 12 hours (pm) on 2nd line.
In05 file – One of the 3 necessary MARS input files typically referred to as the Master Input File (MIF). This is a
text file that has all the MARS tables specified for the assessment.
ln17 file – One of the 3 necessary MARS input files that can have hourly shape information, such as for wind
units and Demand Side Management(DSM) profiles. This file is a text file but can be blank if no hourly
shapes are part of analysis.
Import Capability –Import Capability (in megawatts) is a single value that represents the simultaneous imports
into PJM that can occur during peak PJM system conditions. The capabilities of all transmission facilities
that interconnect the PJM Control Area to its neighboring regions are evaluated to determine this single
value.
Installed Reserve Margin (IRM) – IRM represents the amount of aggregate generating unit capability above the
forecasted peak load that is required to meet a given resource adequacy level. IRM is expressed as a
percentage, and is calculated by dividing the total installed capacity (ICAP) by the net peak load (after
accounting for qualified forecast Load Management). The IRM is used to determine the Forecast Pool
Requirement (FPR). The PJM IRM is the level of installed reserves needed to meet the ReliabilityFirst
Corporation criteria for an LOLE of one day, on average, every 10 years
© PJM Interconnection 2011. All rights reserved
Page 88 of 136
Independent Electricity System Operator (IESO) – is the nonprofit Regional Transmission Organization (RTO)
responsible for managing Ontario's bulk electricity power system and operating the wholesale market. The
IESO is one of the five main control areas within the NPCC. (http://www.ieso.ca/ ).
Independent System Operator (ISO) – An Independent System Operator (ISO) is an organization formed at the
direction or recommendation of the Federal Energy Regulatory Commission (FERC). In the areas where
an ISO is established, it coordinates, controls, and monitors the operation of the electrical power system,
usually within a single state or province.
Independent System Operator of New England (ISO-NE) – is a regional transmission organization (RTO) and
not-for-profit corporation responsible for reliably operating the bulk electric power generation, transmission
system and wholesale electricity markets in New England. Created in 1997 and with headquarters in
Holyoke, MA, the ISO-NE control extends throughout New England including Maine, New Hampshire,
Vermont, Rhode Island, Massachusetts and Connecticut. (www.iso-ne.com)
Locational Deliverability Areas (LDAs) – are zones that comprise the PJM RTO as defined in the RAA
schedule 10.1 and can be an individual zone, a combination of two or more zones, or a portion of a zone.
There are currently 25 LDAs within the PJM footprint.
Load - Integrated hourly electrical demand, measured as generation net of interchange. Loads generally can be
reported and verified to the tenth of a megawatt (0.1 MW) for this report.
Load Management (LM) - Load Management, previously referred to as Active Load Management (ALM),
applies to interruptible customers whose load can be interrupted at the request of PJM. Such a request is
considered an emergency action and is implemented prior to a voltage reduction. This includes both
Demand Resources (DR) and Interruptible Load for Reliability (ILR).
Load-Serving Entity (LSE) – an LSE is a utility company, distribution company or power marketer that provides
the distribution, customer, and energy services for natural gas and electricity. An LSE may also be
referred to as a Utility Distribution Company (UDC).
Loss of Load Expectation (LOLE) - Generation system Adequacy is determined as LOLE and is expressed as
days (occurrences) per year. This is a measure of how often, on average, the available capacity is
expected to fall short of the restricted demand. LOLE is a statistical measure of the frequency of firm
load loss and does not quantify the magnitude or duration of firm load loss. The use of LOLE to assess
Generation Adequacy is an internationally accepted practice. The term expectation is not in the ordinary
sense but adheres to a mathematical definition (See Reference 4).
Loss of Load Hours (LOLH) – represents the hours of unserved energy for a given delivery year.
Multi-Area Reliability Simulation (MARS) – developed by General Electric International, Inc., the MARS (or
GE-MARS) model is a probabilistic analysis program that uses sequential Monte Carlo simulation to
analyze the resource adequacy for multiple areas. MARS is used by ISOs, RTOs, and other
organizations to conduct multi-area reliability simulations.
Metadata – metadata is essentially a data specification – the purpose being to define the many relationships
among various and different database sources that are related yet distinct and separate. This term and
specification technique is used in several state-of-the-art database analysis tools to quickly assess and
report complexities and quantify relationships that otherwise were not practical due to resource limitations
and knowledge transfer among the many database schemas. The Microsoft product SSIS (SQL Server
Integration Services) and SSRS (SQL Server Reporting Services) are examples of available OLAP tools
within the Information Technology Services (ITS) industry. Metadata is used to join together the various
load parameters with the generation parameters, on a time and geographical basis.
Midwest Independent System Operator (MISO) – is a not-for-profit corporation responsible for reliably
operating the bulk electric power generation, transmission system and wholesale electricity markets in
the Midwest. Created in 1996 and with headquarters in both Carmel, IN and St. Paul MN, MISO’s control
extends throughout through 13 states and one Canadian province, including all or parts of: Illinois,
Indiana, Iowa, Michigan, Minnesota, Missouri, Montana, Nebraska, North Dakota, Ohio, Pennsylvania,
South Dakota, Wisconsin and Manitoba (Canada). (www.midwestiso.org)
Monte Carlo Simulation Methods – as mathematical methods for assessing risk probabilities, Monte Carlo
methods involve computational algorithms that rely on repeated random sampling to compute their results.
Monte Carlo methods are often used in simulating physical and mathematical systems where it is
unfeasible or impossible to compute an exact result with a deterministic algorithm. Monte Carlo methods
are especially useful in studying systems with a large number of variables, coupled degrees of freedom
and significant uncertainty in inputs, such as risk assessment.
© PJM Interconnection 2011. All rights reserved
Page 89 of 136
Developed at the time during development of the World War II-based Manhattan Project, Monte Carlo
simulations have been successfully applied in space exploration and oil exploration, actual observations of
failures, cost overruns and schedule overruns are routinely better predicted by the simulations than by
human intuition or alternative "soft" methods. The alternative to Monte Carlo methods is to solve the
problems using the mathematics of probability. This can be performed using numerical methods using
established techniques involving cumulative probability arrays.
New York Independent System Operator (NYISO) – is a not-for-profit corporation responsible for reliably
operating the bulk electric power generation transmission system and wholesale energy markets in New
York State. The NYISO was created from the former New York Power Pool (NYPP) in 1999 and has its
headquarters in Albany, NY. (www.nyiso.com)
New York State Reliability Council (NYSRC) – is a nonprofit, sub-regional electric reliability organization
(ERO) within the NPCC. Working in conjunction with the NYISO, the NYSRC’s mission is to promote and
preserve the reliability of electric service on the New York Control Area (NYCA) by developing,
maintaining and updating reliability rules which shall be complied with by the New York Independent
System Operator (NYISO). (www.nysrc.org)
North American Electric Reliability Corporation (NERC) – is a nonprofit corporation based in Princeton, NJ
(Plans to move to Atlanta GA). NERC was formed in 2006 as the successor to the North American
Electric Reliability Council (also known as NERC). NERC supports the electric utility industry in
promoting reliability and adequacy of the bulk power transmission in the electric utility systems of North
America. NERC's mission states that it is to "ensure that the bulk power system in North America is
reliable." NERC oversees eight regional reliability entities and encompasses all of the interconnected
power systems of the contiguous United States, Canada and a portion of Baja California in Mexico.
(www.nerc.com)
Northeast Power Coordinating Council (NPCC) – is a regional electric reliability organization within NERC that
is responsible for ensuring the adequacy, reliability, and security of the bulk electric supply systems of the
Northeast region comprising parts or all of: New York, Maine, Vermont, New Hampshire, Connecticut,
Rhode Island, Massachusetts, and the Canadian provinces of Ontario, Quebec, Nova Scotia, New
Brunswick, and Prince Edward Island. The five control areas that comprise the NPCC region include:
NYISO, ISO-NE, Ontario (IESO), Hydro Quebec, and the Maritimes. (http://www.npcc.org/ )
Online Analytical Processing (OLAP) – OLAP (/o-lap/), is an approach to quickly answer multi-dimensional
analytical queries. OLAP is part of the broader category of business intelligence, which also
encompasses relational reporting and data mining. The typical applications of OLAP are in business
reporting and business process management (BPM). OLAP was derived from the traditional database
term OLTP (Online Transaction Processing). Databases configured for OLAP use a multidimensional
data model, allowing for complex analytical and ad-hoc queries with a rapid execution time – borrowing
aspects of navigational databases and hierarchical databases that are faster than relational databases.
The output of an OLAP query is typically displayed in a matrix (or pivot) format. The dimensions form the
rows and columns of the matrix; the measures form the values.
Peak Load – The Peak Load is the maximum hourly load over a given time interval, typically a day, month,
season, or delivery year. (Refer to Forecast Peak Load.)
Peak Load Ordered Time Series (PLOTS) – The PLOTS load model is the result of the Week Peak Frequency
(WKPKFQ) application. This is one of the load model’s input parameters. This is discussed in the 2010
Reserve Requirement Study Report, load forecasting and WKPKFQ parameters section of Part II –
Modeling and analysis.
Peak Season – Peak Season is defined to be those weeks containing the 24th through 36th Wednesdays of the
calendar year. Each such week begins on a Monday and ends on the following Sunday, except for the
week containing the 36th Wednesday, which ends on the following Friday. Please note that the load
forecast report(s) used for studies define peak season as June, July and August.
Pipe Size – Determined by load flow based assessment techniques of individual transmission facilities to assess
a single value representative of the composite set of transmission facilities. The pipe size is a summation
of Individual Interface tie sizes to represent transfer limits between areas used by LOLE modeling
methods.
PJM Interconnection Inc. – PJM is a regional transmission organization (RTO) responsible for reliably
operating the bulk electric power generation, transmission system and wholesale energy markets
throughout all or parts 13 states and the District of Columbia, including: Delaware, Illinois, Indiana,
© PJM Interconnection 2011. All rights reserved
Page 90 of 136
Kentucky, Maryland, Michigan, New Jersey, North Carolina, Ohio, Pennsylvania, Tennessee, Virginia,
West Virginia and the District of Columbia.
Planned Outage Factor – A factor measured in weeks per year that indicates the forecast amount, for each unit
to capture the following outages as recorded in PJM’s eGADS application: A unit’s outage of
predetermined length that is scheduled well in advance of its occurrence. The outage must be included in
a regularly issued maintenance schedule at least one month prior to the starting date of the outage; i.e.,
the outage must appear in two consecutive issues of the PJM Unit Maintenance Schedule prior to starting
the outage. See PJM Manual 22 for further details.
Planning Committee – The PJM Planning Committee (PC) is established under the Operating Agreement
(PDF) and has the responsibility to review and recommend system planning strategies and
policies as well as planning and engineering designs for the PJM bulk power supply system to assure
the continued ability of the member companies to operate reliably and economically in a competitive
market environment. Additionally, the PC makes recommendations regarding generating capacity reserve
requirement and demand-side valuation factors. The RAAS reports to this Committee.
Probabilistic Reliability Index Study Model (PRISM) – PRISM is PJM’s primary planning reliability program.
Developed by PJM Staff, PRISM replaced GEBGE which was a FORTRAN language program. The
models are based on statistical measures for both the load model and the generating unit model. This is
a computer application developed by PJM that is a practical application of probability theory and is used
in the planning process to evaluate the generation adequacy of the bulk electric power system.
Probabilistic System – A probabilistic system will have results that vary, due to observable certainty described
by its system distribution parameters. Probabilistic systems (also called stochastic model, process, or
system) are often solved with Monte-Carlo methods – where a computer program uses a pseudo random
number generator to provide values of the attributes in the system that can vary. The program provides an
assessment of the uncertainty of results. Typically, a large number of runs (trials or iterations) are made.
Summary statistics may include the value that occur most frequently (mode), the mean value, and low and
high range, for instance the 10% and 90% percentile. The standard deviation and histogram of results
may also be part of the summary information. There is no single standard presentation as this will depend
on the application. (Compare to Deterministic System.)
Probability Density Function (PDF) – is P (x) of a continuous distribution is defined as the derivative of the
(cumulative) distribution function D (x). The probability density function (PDF) of a continuous distribution
is defined as the derivative of the (cumulative) distribution function, with a full mathematical definition of
the PDF shown at the following link (Wolfram MathWorld – the web’s most extensive mathematics
resource): http://mathworld.wolfram.com/ProbabilityDensityFunction.html.
Probable Load – A given amount of load (typically in megawatts) and the associated probability for that load
value is the probable load. Probable Load is the relationship and association between a given load value
and the likelihood that this load amount will occur.
Resource Adequacy Planning (RAP) – the PJM RAP Department is responsible for the conduct of the annual
Forecast Pool Requirement (FPR) development process and associated Unforced Capacity (UCAP)
Obligation process. The development of the FPR begins with the Reserve Requirement Development
Process utilizing Load Forecasts as well as other resource reports. Encompassing both Planning and
Actual data, RAP considers the impact of load, generation, demand resources and transmission.
Reliability – in a bulk power electric system, is the degree to which the performance of the elements of that
system results in power being delivered to consumers within accepted standards and in the amount
desired. The degree of reliability may be measured by the frequency, duration, and magnitude of adverse
effects on consumer service. Bulk Power electric reliability cab be addressed be considering two basic
and functional aspects of the bulk power system – adequacy and security.
ReliabilityFirst Corporation (RFC) – is one of nine regional electric reliability councils under NERC authority.
RFC began operations in 2006 as the successor to three other reliability organizations: The Mid-Atlantic
Area Council (MAAC), the East Central Area Coordination Agreement (ECAR) and the Mid-American
Interconnected Network (MAIN). Based in Akron, OH, RFC lies within the Eastern Interconnection and
covers territory stretching from the Eastern United States to the lower Great Lakes, covering all of the
states of Pennsylvania, New Jersey, Delaware, Maryland, West Virginia, Ohio, Indiana, and portions of
the states of Wisconsin, Michigan, Illinois, Kentucky and Virginia. (www.rfirst.org)
Reliability Index (RI) – is a value that is used to assess the bulk electric power system’s future occurrence for a
loss-of-load event. A RI value of 10 indicates that there will be, on average, a loss of load event every ten
years. A given value of reliability index is the reciprocal of the LOLE.
© PJM Interconnection 2011. All rights reserved
Page 91 of 136
Reliability Pricing Model (RPM) – RPM is PJM’s capacity-market model. Implemented in 2007, RPM is based
on making capacity commitments three years ahead, is designed to create long-term price signals to
attract needed investments in reliability in the PJM region. The long-term RPM approach, in contrast to
PJM’s previous short-term capacity market, includes incentives that are designed to stimulate investment
both in maintaining existing generation and in encouraging the development of new sources of capacity –
resources that include not just generating plants, but demand response and transmission facilities.
The RPM model works in conjunction with PJM’s Regional Transmission Expansion Planning (RTEP)
process to ensure the reliability of the PJM region for future years. The RPM includes the continued use
of self-supply and bilateral contracts by load-serving entities (LSEs) to meet their capacity obligations. The
capacity auctions under the RPM obtain the remaining capacity that is needed after market participants
have committed the resources they will supply themselves or provide through contracts.
Regional Transmission Expansion Plan (RTEP) – PJM’s RTEP identifies transmission system additions and
improvements needed throughout the PJM footprint. Studies are conducted to test the transmission
system against mandatory NERC national standards as well as PJM regional standards. RTEP looks
ahead 15 years into the future to identify transmission overloads, voltage limitations and other reliability
standards violations.
RTEP results are used to assist PJM’ Planning Dept. in developing transmission plans to resolve
violations that could otherwise lead to overloads and black-outs. These plans are examined for their
feasibility, impact and costs and are discussed throughout the development process with PJM
stakeholders. When reliability criteria violations are detected, PJM then develops solutions to mitigate
those violations. RTEP recommendations are submitted periodically to PJM’s independent Board of
Managers to resolve identified reliability criteria violations. Once mitigation plans are approved, they
become part of PJM’s overall RTEP.
Sub-Regional RTEP Committees (SR RTEPs) have been formed for the Mid-Atlantic, Southern and
Western sub-areas to provide review and input of sub-regional RTEP projects and provide
recommendations to the TEAC.
Regional Transmission Organization (RTO) – Each entity (a) that owns, leases or otherwise has a possessory
interest in facilities used for the transmission of electric energy in interstate commerce, (b) that provides
Transmission that is a party to the PJM Transmission Owners Agreement and PJM Operating Agreement.
Reserve Requirement Study (RRS) – Performed annually, the objective of PJM’s RRS is to derive a single
calculated percentage of IRM that represents the amount above peak load that must be maintained to
meet the RFC adequacy criteria. The RFC adequacy criteria are based on a probabilistic requirement of
experiencing a loss-of-load event, on average, once every ten years. (The RRS is also referred to as the
R-Study.)
Reserve Requirement Assumptions Working Group (RRAWG) – this former group previously reported to the
PJM PC. Beginning in January 2011 this RRAWG group was renamed the Resource Assumptions
Analysis Subcommittee (RAAS).
Resource Adequacy Analysis Subcommittee (RAAS) - reporting to the PC, the RAAS assists PJM staff in
performing the annual Reserve Requirement Study (RRS), setting LOLE study assumptions, performing
CETO assessments and maintains the reliability analysis documentation. This subcommittee name
replaced the Reserve Requirement assumptions Subcommittee (formed in January 2011 replacing the
RRAWG) name.
Restricted Peak Load - For the given forecast period, the restricted peak load equals the unrestricted peak load
minus anticipated load management.
SAS – SAS Institute offers several software products and analytical solutions driven by a proprietary 4th
generation computer language. It is owned by SAS Institute Inc. , providing software solutions since
1976. SAS Inc. is the largest independent vendor in the business intelligence market, based in Cary,
North Carolina.
Security – refers to the ability of the bulk electric system to withstand sudden disturbances such as electric short
circuits or unanticipated loss of system components or switching operations. One part of the Reliability
term.
© PJM Interconnection 2011. All rights reserved
Page 92 of 136
SERC Reliability Corporation (SERC) - SERC is one of nine regional electric reliability councils under North
American Electric Reliability Corporation (NERC) authority. SERC was formed in 2005, as the successor
to the Southeast Electric Reliability Council (also known as SERC) and is based in Charlotte, NC. The
SERC region serves all or parts of Alabama, Arkansas, Florida, Georgia, Illinois, Iowa, Kentucky,
Louisiana, Missouri, North Carolina, Oklahoma, South Carolina, Tennessee, Texas and Virginia.
(www.serc1.org )
Simultaneous Import Limit (SIL) – The SIL is the amount (megawatts) that can be imported into the PJM RTO
over all transmission tie-line facilities during capacity emergency conditions. The SIL determined for FERC
compliance filings, such as Order No. 697, can use different assumptions than used for forecasting future
planning requirements, and has specific, detailed study assumptions specifications. However the SIL is
one of several considerations used in review of the Capacity Benefit Margin (CBM).
Summer Net Dependable (SND) – is the rating for a given generation unit is used in the summer period. All
processes use the SND rating as the basis for evaluating a unit. See PJM Manual 21 for further details.
Transmission Expansion Advisory Committee (TEAC) – the TEAC provides advice and recommendations to
aid in the development of the RTEP and SR RTEPs.
Unforced Capacity (UCAP) – Installed capacity rated at summer conditions that are not on average
experiencing a forced outage or forced derating, calculated for each Capacity Resource on the 12-month
period from October to September without regard to the ownership of or the contractual rights to the
capacity of the unit.
Unrestricted Peak Load – this represents the metered load plus estimated impacts of Load Management.
Variance - A measure of the variability of a unit's partial forced outages which is used in reserve margin
calculations. This term is also called the two-state variance, a result of the PRISM model only having two
states either full on or full off. See PJM manual 22 for further details.
Weak Peak Frequency (WKPKFQ) – A computer application that creates the Peak Load Ordered Time Series
(PLOTS) load model; its input data is hourly load data. This application is part of the ARC application.
Weather Normalized Loads - The weather-normalized loads are estimated seasonal peak assuming median
peak day weather conditions. The weather-normalized loads are also referred to as 50 / 50 loads.
Winter Weekly Reserve Target – part of the annual RRS process. PJM Staff develops the Winter Weekly
Reserve Target, covering the 13-week winter season (December through February.). This parameter is
used for operation’s planned outage scheduling of the PJM generation fleet. PJM uses the MARS
program to evaluate this parameter.
Zone / Control Zone - An area within the PJM Control Area, as set forth in PJM’s Open Access Transmission
Tariff (OATT) and the Reliability Assurance Agreement (RAA). Schedule 10 and 15 of the RAA provide
information concerning the distinct zones that comprise the PJM Control Area.
© PJM Interconnection 2011. All rights reserved
Page 93 of 136
References
1.
Multi-Area Generation Reliability – Phase I Testing Report - October 22, 1991 - developed by the New
York Power Pool (NYPP) Reliability Evaluation Task Force for the ESEERCO Project EP87-28
2.
Generating Reserve Capacity Determined by the Probability Method - March 25, 1947 - by Giuseppe
Calabrese, presented at the AIEE Midwest meeting of November 3, 1947
3.
Probability Methods Applied to Generating Capacity Problems of a Combined Hydro and Steam
System - August 14, 1947 - by E.S. Loane and C.W. Watchorn; presented at the AIEE Midwest meeting of
November 3, 1947
4.
Reliability Evaluation of Power Systems – 1984 – by Roy Billinton and Ronald Allan, published by the
Plenum Press, New York
5.
NPCC Regional Reliability Reference Directory #1 “Design and Operation of the Bulk Power System”
(December 1, 2009) - section 5.2, page 9.
6.
MAPP Loss of Load Expectation (LOLE) Study, MAPPCOR and MAPP Composite System Reliability
Working Group (December 30, 2009);
http://www.mapp.org/ReturnBinary.aspx?Params=584e5b5f405c567900000002cb
7.
Mid-Atlantic Area Council (MAAC) Document A-1, MAAC Reliability Principles and Standards (March 30,
1990).
8.
MAIN Guide #6, Generation Reliability Study 20045-2014 (September 27, 2005).
9.
Standard BAL-502-RFC-02 (December 4, 2008) - ReliabilityFirst Corporation (RFC) region planning
resource Adequacy Analysis, Assessment and Documentation,: http://www.nerc.com/files/BAL-502-RFC02.pdf .
10. Midwest Reliability Organization (MRO) Standard RES-501-MRO-01, planned Resource Adequacy
Assessment (December 29, 2007);
http://www.midwestreliability.org/04_standards/approved_standards/mro_standards/RES-501MRO-01_Final_20071229_Clean.pdf .
11. PJM Generation Adequacy Analysis: Technical Methods - October 2003 – developed by PJM Staff;
available at: http://www.pjm.com/planning/resource-adequacy-planning/~/media/planning/resadeq/20040621-white-paper-sections12.ashx.
12. Probability Calculation of Generation Reserves - March 1969 - by C.J. Baldwin and published by The
Westinghouse Engineer, This paper is copyright protected but can be purchased online at Infotrieve; article
information accession number 00434361 (800-422-4633; www.infotrieve.com).
13. Evaluation of PTI’s Multi-Area Reliability Program MAREL - November 1993 - developed by the PJM
Load & Capacity Working Group (L&CWG)
14. Power System Reliability Evaluation, Mathematical Expectation - page 12, Gordon and Beach, Science
Publishers, –1970 – by Roy Billinton and Reinvent Legacy Software with SAS, the Web, and OLAP Reporting, Appendix
B –2007 – available at: http://www.qlx.com/whitepapers/25-2008.pdf .
15. Re-invent a Legacy System with SAS, the Web and OLAP reporting, November 13, 2007, 2007
Northeast SAS user group conference – Applications Big and Small, Paper Number 10.
http://www.lexjansen.com/nesug/ , Appendix : Download the code (2369 KB)
16. Schedule 15 of Reliability Assurance Agreement (RAA) (link) and PJM Manual 14B, pages 49-50 (Link),
December 02, 2010.
17. Power System Reliability Evaluation, Mathematical Expectation - page 12, Gordon and Beach, Science
Publishers, –1970 – by Roy Billinton and Reinvent Legacy Software with SAS, the Web, and OLAP
Reporting, Appendix B –2007 – available at : http://www.qlx.com/whitepapers/25-2008.pdf .
18. The Scientist and Engineer's Guide to Digital Signal Processing , By Steven W. Smith, Ph.D, copyright
© 1997-2006 by California Technical Publishing – http://www.dspguide.com/pdfbook.htm
© PJM Interconnection 2011. All rights reserved
Page 94 of 136
19. Course notes, module PE.PASU19.5 on Generation adequacy evaluation, Convolution techniques,
item U19.7.3 - page 43, by Dr. James McCalley’s. http://www.ee.iastate.edu/~jdm/ee653/ee653schedule.htm
20. Course notes: Electrical Power System Reliability, part3 Discrete Convolution Method - page 30,
copyright 1995, by Dr. Chanan Singh, http://www.ece.tamu.edu/People/bios/singh/coursenotes/part3.pdf
21. Northeast Power Coordinating Council Tie Benefits Methodology, by Glenn Haringa and Philip Fedora,
November 5-6, 2008, Best LOLE Practices meeting held at California ISO offices, Agenda item 8. See slide
12, last bullet.
22. New York Control Area Installed Capacity Requirements for the Period May 2010 through April 2011 Technical Study Report - December 4, 2009 – developed by the New York State Reliability Council, LLC
(NYSRC) Installed Capacity Subcommittee (ICS)
23. New York State Reliability Council LLC NYSRC Policy, No 5-3, procedure for establishing New York
Area Installed Capacity Requirements, November 16, 2009)
24. Westinghouse Capacity Model and GE MARS, by Fei Zeng ISO New England, April 27-28, 2010, North
America Electric Reliability Corporation (NERC) Loss-Of-Load-Expectation (LOLE) Working Group, Portland
OR. Agenda item 7d – http://www.nerc.com/filez/lolewg.html .
25. California ISO Planning Reserve Margin – 2010 – 2020, May 21, 2010 Final Report to California
Independent System Operator for Planning Reserve Margin (PRM) Study - 2010 – 2020.
http://www.caiso.com/279d/279ded0337f20.pdf
26. IEEE Standard 762-2006: IEEE Standard definition for use in reporting electric generating unit
reliability, availability and productivity. IEEE Std-2006 (revision of IEEE Std 762-1987) - approved
9/16/2006 by IEEE-SA Standards Board. To obtain a copy: http://standards.ieee.org/findstds/standard/7622006.html )
27. Northeast Power Coordinating Council Interregional Long Range Adequacy Overview - (November 28,
2006), conducted by the NPCC CP-8 Working Group. http://www.npcc.org/docSearch.aspx .
28. SAS System for Elementary Statistical Analysis, Second Edition – 1997 – by Sandra D. Schlotzhauer
and Ramon C. Littell, PHD, published by the SAS Institute Inc, Cary NC, USA
29. 2009 PJM Reserve Requirement Study – 11-year Planning Horizon: June 1st 2009 to May 31st 2020 –
October 13, 2009 - Analysis performed by PJM Staff with review by the PJM Reserve Requirement
Assumptions Working Group (RRAWG)
30. NYISO load model processing for selection of LOLE load models –New York State Reliability Council
(NYSRC), Installed Capacity Subcommittee (ICS) Meeting, Albany, NY, June 1, 2010. Footnoted: Draft –
For discussion only.
31. Reliability Assessment of Electrical Power Systems using Monte Carlo Methods – 1994 – by Roy
Billinton and Wenyuan Li, published by the Plenum Press, New York
32. PJM Manual 22, Generator Resource Performance indices, revision 15, June 1, 2007 - :
http://www.pjm.com/~/media/documents/manuals/m22.ashx
33. PJM Manual 21, Rules and Procedures for Determination of Generating, revision 9, May 1, 2010 :
http://www.pjm.com/~/media/documents/manuals/m21.ashx
34. PJM Manual 20, PJM Resource Adequacy Analysis, revision 3, June 1, 2007 :
http://www.pjm.com/~/media/documents/manuals/m20.ashx
35. A Monte Carlo Simulation Model for Multi-Area Generation Reliability Evaluation, by A. A. Chowdhury,
L. Bertling, B. P. Glover, G.E. Haringa, June 11-15, 2006, International conference on Probabilistic Methods
Applied to Power Systems, Stockholm Sweden.
36. Reliability Evaluation of Engineering Systems: Concepts and Techniques – 1984 – by Roy Billinton
and Ronald Allan, published by the Plenum Press, New York
37. Reliability Pricing Model (RPM) Auction User Information, 2013/2014 Scenario Analysis Results.
Simulation # 4 of 22. These simulations are available on the PJM Web Site.
© PJM Interconnection 2011. All rights reserved
Page 95 of 136
38. Measurement Practices for Reliability and Power Quality, a toolkit of reliability measurement
practices, June 2004, by J.D. Kueck and B.J. Kirby, Oak Ridge National Laboratory, P.N. Overholt, US DOE, L.C.
Markel, Sentech, Inc. Report link. DOE Research report abstract.
39. PJM Load forecast report, January 2010. Available on PJM web site at this location.
40. 2010 PJM Reserve Requirement Study – 11-year Planning Horizon: June 1st 2010 to May 31st 2021 –
September 30, 2010 - Analysis performed by PJM Staff with review by the PJM Reserve Requirement
Assumptions Working Group (RRAWG). Report posted here.
41. Federal Energy Regulatory Commission, 18 CFR part 40, Docket No. RM10-10-000 – Planning
Resource Adequacy Assessment Reliability Standard, issued October 21, 2010. Notice of Proposed
Rulemaking (NOPR).
42. Reliability Assurance Agreement among load serving entities in the PJM Region , Rate Schedule FERC
No. 44. Agreement posted here.
© PJM Interconnection 2011. All rights reserved
Page 96 of 136
APPENDICES
o
Appendix A – “MARS Model vs. PRISM Model Overview” - PJM Staff presentation for the
April 15, 2008 RRAWG meeting
o
Appendix B - Calculations to translate from PLOTS load models used in PRISM to the LODUNCY table values used in MARS
o
Appendix C -- Load Modeling Comparison Issues
Appendix C1 – Load Distribution
Appendix C2 – Baldwin Paper: Choice of Sigma
Appendix C3 – ISO-NE’s Comparison of Westinghouse Capacity Model and GEMARS
o
Appendix D – MARS Solution Techniques
o
Appendix E – Transparency of process and steps used to perform LOLE calculations
o
Appendix F – Detailed Input Parameter Requirements
o
Appendix G – Possible future investigation activities
© PJM Interconnection 2011. All rights reserved
Page 97 of 136
Appendix A – MARS Model vs. PRISM Model Overview PJM Staff presentation
“MARS Model vs. PRISM Model Overview”
PJM Staff presentation for the April 15, 2008 RRAWG meeting
The following PJM Staff presentation was delivered at the 15 April 2008 RRAWG meeting, per requests to show how MARS and PRISM are being used and how they compare.
This presentation was developed to capture the efforts of several PJM staff, but was not meant to provide full and comprehensive documentation for how to coordinate
consistent modeling between the different tools. The PJM staff recognizes that thorough documentation needs development; however this was identified as a lower priority task.
The important points of this presentation information include: 1) PJM staff have been using MARS and PRISM in concert since 2004; 2) PRISM and MARS are parts of the
overall production environment Applications for Reliability Calculations (ARC) ; 3) While MARS is driven using the ARC GUI, MARS has many more manual text entries ; 4) The
key component for consistent models is the load model entries in four MARS tables. ; 5) There are more MARS categories of input requirements and more data than PRISM. ;
6) MARS provides capabilities that PRISM does not and they are used in a complementary fashion. ; 7) Comprehensive documentation will require additional efforts.
© PJM Interconnection 2011. All rights reserved
Page 98 of 136
© PJM Interconnection 2011. All rights reserved
Page 99 of 136
© PJM Interconnection 2011. All rights reserved
Page 100 of 136
© PJM Interconnection 2011. All rights reserved
Page 101 of 136
© PJM Interconnection 2011. All rights reserved
Page 102 of 136
Appendix B – Translation from PLOTS to LOD-UNCY table
Calculations to translate from PLOTS load models used in
PRISM to the LOD-UNCY table values used in MARS
The Appendix B spreadsheet maps the steps in the calculation process to translate the PRISM load model to the
MARS load model so that modeling can be consistent (See Appendix C3, slides 12-15).
© PJM Interconnection 2011. All rights reserved
Page 103 of 136
The values are processed from left to right – and then from top to bottom. The teal-highlighted values represent a
distribution for the mean while the light blue-highlighted values represent the final EWM distribution values.
1.
Run Week Peak Frequency (WKPKFQ) to create the Peak Load Ordered Time Series (PLOTS) Load
Model
This is a probabilistic load model typically based on 7-10 years of history and the current load
forecast growth rates. This creates the mean and standard deviation for each week in the model (152). This is shown in the Mean seasonal Trend and the Standard Deviation columns (yellow
highlighted area). These are the starting values for the translation process.
2.
Translate the ARC weeks into calendar months.
The ARC weeks correspond to a delivery year time period. A delivery year starts around the 1st of
June and ends the following May. The values are gathered into the appropriate calendar month, as
shown in the in the columns before the dark bold line. I.e. ARC week 1 is calendar week 20, which is
in mid-May.
3.
Calculate Mean Values (teal-highlighted area)
The calculated values of variance, which is the square of the standard deviation, and the variance
times the mean help to determine a monthly weighted variance and standard deviation(standard
deviation is the squared root of the variance). These weighted Standard deviations for each month
are used to determine the 7 LOD-UNCY values at the given number of STD away from the mean,
assuming a Standard normal distribution. These values are shown in the teal-highlighted region.
4.
Final EOP-UNCY Table values using EWM
The monthly weighted Expected Weighted Maximum (EWM) is determined from the monthly
weighted standard deviation values. [ EWM = 1* (1+1.16295* Wtd. STD Dev) ] .
o
o
o
o
o
The 7 EOP-UNCY values for each month are the mean values in the teal-highlighted area
divided by the monthly EWM values (calculated from Wtd. STD Dev values shown on that
row for that month).
These EOP-UNCY table values now take on the distribution using the EWM methods
deployed in the PLOTS and PRISM load model.
A recognized limitation is that one can only subset the LOD-UNCY into monthly aggregates.
The ability to specify weekly or daily LOD-UNCY values is not available yet. A request to GE
staff has been submitted to be considered for future MARS improvements.
The final LOD-UNCY values are shown in the gray-highlighted area.
It is worth noting that the ISO-NE staff presented recent material that indicates a similar process is involved to join
their Westinghouse model with the MARS model (See Appendix C3).
The ISO-NE Westinghouse modeling is analogous to what is used in GEBGE (which was the basis for PRISM).
They cite the use of the MARS LOD-UNCY table is a significant step in ensuring that the load models are
consistent and measure the same load model attributes/distributions.
A primary interest is the tails of the load model distributions between the two applications, which was discussed at
length at several RRAWG meetings during the 2009 RRAWG assessment of load model issues. The tails of these
distributions is where most of the LOLE occurs hence attention to the details of the distributions at these values is
very significant and important in the LOLE assessment efforts. (Example RRAWG presentations:
1. http://www.pjm.com/~/media/committees-groups/working-groups/rrawg/20090910/20090910item-03a-reporting-on-50-50-solution-loads.ashx
2. http://www.pjm.com/~/media/committees-groups/working-groups/rrawg/20090925/20090925item-03a-consistence-between-load-forecast-rrs-peak-load.ashx
3. http://www.pjm.com/~/media/committees-groups/working-groups/rrawg/20090731/20090731item-03a-truncating-normal-distribution.ashx
4. http://www.pjm.com/~/media/committees-groups/working-groups/rrawg/20090701/20090701item-03a-and-03b-load-model-selection.ashx
© PJM Interconnection 2011. All rights reserved
Page 104 of 136
Appendix C – Load Modeling Comparison Issues
Load Modeling parameters to use in the LOD-UNCY table
The LOD-UNCY values are associated with the deterministic inputted 8760 hourly loads that define the shape
used in the MARS daily peak solution method, using a Monte Carlo method.
Topics considered in this Appendix:
C1 - Load distribution granularity and impacts concerning greater granularity.
C2 - Choice of Sigma: Load model truncation discussion in the CJ Baldwin paper.
C3 - Use of LOD-UNCY table for consistent modeling between Westinghouse method and MARS.
This appendix provides further details around a central topic when comparing the PRISM and MARS modeling
and solution results.
Appendix C1 – Load distribution granularity
Load Distribution
Load distribution granularity and impacts concerning greater granularity
Assessment and discussion of the use of 21 points in the PRISM load distribution and the possible need for greater
49
granularity allowed for review of the Westinghouse Engineer paper, March 1969, by CJ Baldwin which is the basis
of the load modeling techniques used in PRISM. The PJM staff also independently performed an evaluation of
impacts around using various points for the daily peak distribution curve.
An issue to consider in this matter is that typically the Industry models that use MARS, apply a 7-point
distribution in the MARS LOD-UNCY table. This table has a maximum value of 10 points. Considering the
weekly distribution, with MARS assessing all seven days of the week, MARS considers 49 points each week (7
points in each of 7 days per week) while the use of the 21-point approach in PRISM considers 105 points per
week (21 points in each of 5 days per week).
There is a direct impact, almost one for one, in the MARS solution time by adding more points in the
distribution.
Assessment of number of points in daily peak distribution
PJM Staff summarized Adequacy analysis and spreadsheet analysis that assessed using up to 321 points in
the load distribution.
Figure C1 shows the change in LOLE, for the peak week, as a function of the number of points in the
load distribution.
The spreadsheet work indicates that the choice of 21 points for the distribution allows for statistically
valid results while addressing the need for solution time.
The results of the 21 point distribution cannot be found to be statistically more or less accurate than
the results seen for up to 321 points.
An issue in this type of assessment work has been typically called “Lumpiness”. This term relates that
the model does well in general aggregate assessment terms, but drilling further into the small values
being calculated may result in unexpected, somewhat quirky results. This might explain the
reason for what is seen when evaluating the 9-point distribution.
In addition, an assessment to evaluate the 41-point distribution was made. The distribution was
truncated at the zero point, and doubled the numbers of points up to 4.2 standard deviations. That
sensitivity case gave an IRM result 0.08% different than the 21 point result, partially explained due to
a different solved RI.
49
C.J. Baldwin, “Probability Calculation of Generation Reserves” (March 1969), published by The Westinghouse
Engineer, This paper is copyright protected but can be purchased online at Infotrieve; article information accession
number 00434361 (800-422-4633; www.infotrieve.com).
© PJM Interconnection 2011. All rights reserved
Page 105 of 136
Figure C1 – LOLE vs. number of points in Load model distribution
The conclusions drawn from this portion of assessment work were:
LOLE computed using different representations is function of the installed reserves
Representations using less than 21 points can result in either lower or higher LOLE’s for a given
reserve- Not suitable for estimating risk
LOLE computed stabilizes with representations of 21 points and beyond.
Increase in number of points beyond 21 has diminishing effects on LOLE.
Choice of representation should also consider computation time
© PJM Interconnection 2011. All rights reserved
Page 106 of 136
Appendix C2 – Baldwin Paper
Baldwin Paper: Choice of Sigma
Choice of Sigma: Load model truncation discussion in the CJ Baldwin paper.
The following PJM Staff presentation was delivered to the 19 August 2009 RRAWG meeting. This
presentation was developed to capture the essence and content of the aforementioned C.J. Baldwin paper
“Probability Calculation of Generation Reserves” (March 1969) paper written by CJ Baldwin, dated March
19699. This paper serves as the basis for the load modeling algorithms used in GEBGE and PRISM.
Per the CJ Baldwin paper the Years/Day Reliability Index (RI) saturates just above 4.0 standard deviations.
The choice of modeling out to 4.2 standard deviations in the PRISM model is justified by this paper’s Table VI,
shown below.
The discussion in the paper states “….A number of engineers (but rarely mathematicians) will argue that utility
loads are in fact not normal because load can never exceed connected load, which is certainly not infinite.
They argue that the load distribution should be truncated at connected load. The argument is valid, but the
important question is the effect of truncation on risk, which is illustrated in Table VI. ……..“
Analysis for the 2008 RRS single area model was performed to see if this type of relationship can be empirically
shown for the PJM model. The results of that analysis, adjusting the 21-point distribution to change the sigma,
shows that 4 standard deviations, or more, is a good modeling parameter choice based on the specifics of the PJM
system model.
© PJM Interconnection 2011. All rights reserved
Page 107 of 136
© PJM Interconnection 2011. All rights reserved
Page 108 of 136
© PJM Interconnection 2011. All rights reserved
Page 109 of 136
Appendix C3 – ISO-NE’s Comparison of Westinghouse Model and MARS
ISO-NE’s Comparison of Westinghouse Model and MARS
LOD-UNCY table for consistent modeling between Westinghouse method and MARS, version 3.00 .
A complete set of the following slides were presented at the 4/27/2010 NERC LOLE WG meeting and are available at: http://www.nerc.com/filez/lolewg.html .
The Westinghouse model is similar to PRISM’s single area model and calculations.
© PJM Interconnection 2011. All rights reserved
Page 110 of 136
© PJM Interconnection 2011. All rights reserved
Page 111 of 136
© PJM Interconnection 2011. All rights reserved
Page 112 of 136
© PJM Interconnection 2011. All rights reserved
Page 113 of 136
Several observations can be drawn from this presentation and subsequent work by PJM staff:
The essence of the comparison between Westinghouse and MARS is the same as the comparison
between PRISM and MARS.
Westinghouse and PRISM use the same analytical (convolution) solution techniques, with PRISM
having the ability to represent two areas while Westinghouse typically is used for a single area.
The PRISM model does not use the 3 moment (“skew-ness”) in its load model.
The capacity distributions between MARS and the analytical models, both Westinghouse and PRISM
can be shown as identical (see slide 9). PJM staff is developing these MARS distribution graphs using
Industry models and the Histogram post processing tool (HST-DATA).
The primary issue for consistency between the analytical models and MARS is the load model
distribution, followed by the transfer pipe limits.
Attention to the high load segment of the load distribution, which has a low probability of occurrence, is
imperative for proper LOLE modeling and assessments.
For single area models, similar LOLE results can be achieved when equivalent input assumptions are
used between the two modeling methods (analytical vs. Monte Carlo).
Equivalent input assumptions necessitates the use of the LOD-UNCY table:
rd
1.
2.
3.
Minimize least-square error between two distributions.
Identify more weight on the high load portion of the load distribution.
LOD-UNCY values will change as the associated hourly load shape values change.
© PJM Interconnection 2011. All rights reserved
Page 114 of 136
Appendix D – MARS public solution techniques
Two public information sources are given that describe the MARS solution methods used in the LOLE calculations
and reporting.
The first public information source, concerning MARS solution calculations and methods, is from the
NPCC CP-8 WG 2004 Summer assessment, Appendix C. This information is available on the NPCC web site
at: http://www.npcc.org/documents/reports/Seasonal.aspx.
NPCC SUMMER 2004
MULTI-AREA PROBABILISTIC RELIABILITY ASSESSMENT
NPCC CP-8 Working Group – May 2004 39 Final Report
APPENDIX C
Multi-Area Reliability Simulation Program Description
General Electric International, Inc.’s Multi-Area Reliability Simulation (MARS) program allows assessment of the
reliability of a generation system comprised of any number of interconnected areas.
Modeling Technique
A sequential Monte Carlo simulation forms the basis for MARS. The Monte Carlo method allows for many different
types of generation and demand-side options.
In the sequential Monte Carlo simulation, chronological system histories are developed by combining randomly
generated operating histories of the generating units with the inter-area transfer limits and the hourly chronological
loads. Consequently, the system can be modeled in great detail with accurate recognition of random events, such
as equipment failures, as well as deterministic rules and policies that govern system operation.
Reliability Indices
The following reliability indices are available on both an isolated (zero ties between areas) and interconnected
(using the input tie ratings between areas) basis:
.
Daily LOLE (days/year)
Hourly LOLE (hours/year)
LOEE (MWh/year)
Frequency of outage (outages/year)
Duration of outage (hours/outage)
Need for initiating Operating Procedures (days/year or days/period)
The CP-8 Working Group used both the daily LOLE and EOP indices for this analysis.
The use of Monte Carlo simulation allows for the calculation of probability distributions, in addition to expected
values, for all of the reliability indices. These values can be calculated both with and without load forecast
uncertainty.
The MARS program probabilistically models uncertainty in forecast load and generator unit availability. The
program calculates expected values of Loss of Load Expectation (LOLE) and can estimate each Area's expected
exposure to their Emergency Operating Procedures. Scenario analysis is used to study the impacts of extreme
weather conditions, variations in expected unit in-service dates, overruns in planned scheduled maintenance, or
transmission limitations.
Resource Allocation among Areas
The first step in calculating the reliability indices is to compute the area margins on an isolated basis, for each hour.
This is done by subtracting from the total available capacity in the area for the hour the load demand for the hour. If
an area has a positive or zero margin, then it has sufficient capacity to meet its load. If the area margin is negative,
the load exceeds the capacity available to serve it, and the area is in a loss-of-load situation.
If there are any areas that have a negative margin after the isolated area margins have been adjusted for
curtailable contracts, the program will attempt to satisfy those deficiencies with capacity from areas that have
© PJM Interconnection 2011. All rights reserved
Page 115 of 136
positive margins. Two methods are available for determining how the reserves from areas with excess capacity are
allocated among the areas that are deficient.
In the first approach, the user specifies the order in which an area with excess resources provides
assistance to areas that are deficient.
The second method shares the available excess reserves among the deficient areas in proportion to the
size of their shortfalls. The user can also specify that areas within a pool will have priority over outside
areas. In this case, an area must assist all deficient areas within the same pool, regardless of the order of
areas in the priority list, before assisting areas outside of the pool. Pool-sharing agreements can also be
modeled in which pools provide assistance to other pools according to a specified order.
Generation
MARS has the capability to model the following different types of resources:
. Thermal
. Energy-limited
. Cogeneration
. Energy-storage
. Demand-side management
An energy-limited unit can be modeled stochastically as a thermal unit with an energy probability distribution (Type
1 energy-limited unit), or deterministically as a load modifier (Type 2 energy-limited unit). Cogeneration units are
modeled as thermal units with an associated hourly load demand. Energy-storage and demand-side management
impacts are modeled as load modifiers.
For each unit modeled, the installation and retirement dates and planned maintenance requirements are specified.
Other data such as maximum rating, available capacity states, state transition rates, and net modification of the
hourly loads are input depending on the unit type.
The planned outages for all types of units in MARS can be specified by the user or automatically scheduled by the
program on a weekly basis. The program schedules planned maintenance to levelize reserves either on an area,
pool, or system basis.
MARS also has the option of reading a maintenance schedule developed by a previous run and modifying it as
specified by the user through any of the maintenance input data. This schedule can then be saved for use by
subsequent runs.
Thermal Units
In addition to the data described previously, thermal units (including Type 1 energy-limited units and
cogeneration) require data describing the available capacity states in which the unit can operate. This is
input by specifying the maximum rating of each unit and the rating of each capacity state as a per unit of
the unit's maximum rating. A maximum of eleven capacity states are allowed for each unit, representing
decreasing amounts of available capacity as governed by the outages of various unit components.
Because MARS is based on a sequential Monte Carlo simulation, it uses state transition rates, rather than
state probabilities, to describe the random forced outages of the thermal units. State probabilities give the
probability of a unit being in a given capacity state at any particular time, and can be used if you assume
that the unit's capacity state for a given hour is independent of its state at any other hour. Sequential
Monte Carlo simulation recognizes the fact that a unit's capacity state in a given hour is dependent on its
state in previous hours and influences its state in future hours. It thus requires the additional information
that is contained in the transition rate data.
For each unit, a transition rate matrix is input that shows the transition rates to go from each capacity state
to each other capacity state. The transition rate from state A to state B is defined as the number of
transitions from A to B per unit of time in state A:
Number of Transitions from A to B
TR (A to B) = _____________________________
Total Time in State A
© PJM Interconnection 2011. All rights reserved
Page 116 of 136
If detailed transition rate data for the units is not available, MARS can approximate the transition rates
from the partial forced outage rates and an assumed number of transitions between pairs of capacity
states. Transition rates calculated in this manner will give accurate results for LOLE and LOEE, but it is
important to remember that the assumed number of transitions between states will have an impact on the
time-correlated indices such as frequency and duration.
Energy-Limited Units
Type 1 energy-limited units are modeled as thermal units whose capacity is limited on a random basis for
reasons other than the forced outages on the unit. This unit type can be used to model a thermal unit
whose operation may be restricted due to the unavailability of fuel, or a hydro unit with limited water
availability. It can also be used to model technologies such as wind or solar; the capacity may be available
but the energy output is limited by weather conditions.
Type 2 energy-limited units are modeled as deterministic load modifiers. They are typically used to model
conventional hydro units for which the available water is assumed to be known with little or no uncertainty.
This type can also be used to model certain types of contracts. A Type 2 energy-limited unit is described
by specifying a maximum rating, a minimum rating, and a monthly available energy. This data can be
changed on a monthly basis. The unit is scheduled on a monthly basis with the unit's minimum rating
dispatched for all of the hours in the month.
The remaining capacity and energy can be scheduled in one of two ways. In the first method, it is
scheduled deterministically so as to reduce the peak loads as much as possible. In the second approach,
the peak-shaving portion of the unit is scheduled only in those hours in which the available thermal
capacity is not sufficient to meet the load; if there is sufficient thermal capacity, the energy of the Type 2
energy-limited units will be saved for use in some future hour when it is needed.
Cogeneration
MARS models cogeneration as a thermal unit with an associated load demand. The difference between
the unit's available capacity and its load requirements represents the amount of capacity that the unit can
contribute to the system. The load demand is input by specifying the hourly loads for a typical week (168
hourly loads for Monday through Sunday). This load profile can be changed on a monthly basis. Two types
of cogeneration are modeled in the program, the difference being whether or not the system provides
back-up generation when the unit is unable to meet its native load demand.
Energy-Storage and DSM
Energy-storage units and demand-side management impacts are both modeled as deterministic load
modifiers. For each such unit, the user specifies a net hourly load modification for a typical week which is
subtracted from the hourly loads for the unit's area.
Transmission System
The transmission system between interconnected areas is modeled through transfer limits on the
interfaces between pairs of areas. The transfer limits are specified for each direction of the interface and
can be changed on a monthly basis. Random forced outages on the interfaces are modeled in the same
manner as the outages on thermal units, through the use of state transition rates.
Contracts
Contracts are used to model scheduled interchanges of capacity between areas in the system. These
interchanges are separate from those that are scheduled by the program as one area with excess capacity
in a given hour provides emergency assistance to a deficient area. Each contract can be identified as
either firm or curtailable. Firm contracts will be scheduled regardless of whether or not the sending area
has sufficient resources on an isolated basis, but they can be curtailed because of interface transfer limits.
Curtailable contracts will be scheduled only to the extent that the sending Area has the necessary
resources on its own or can obtain them as emergency assistance from other areas.
© PJM Interconnection 2011. All rights reserved
Page 117 of 136
The second public information source, concerning MARS solution calculations and methods, is available on the NERC web site at:
http://www.nerc.com/filez/lolewg.html . (Agenda Item 7D)
General Electric International, Inc. (GEII) Presentation Slides
© PJM Interconnection 2011. All rights reserved
Page 118 of 136
© PJM Interconnection 2011. All rights reserved
Page 119 of 136
© PJM Interconnection 2011. All rights reserved
Page 120 of 136
© PJM Interconnection 2011. All rights reserved
Page 121 of 136
Appendix E – Transparency of process to perform LOLE calculations
PRISM
Detailed output of almost all aspects of the input processing, calculation steps, solution parameters and reported results can be observed by using the DEBUG
option for a given PRISM run. All DEBUG information is stored based on the PRISM run number. In the final reporting the PRISM run number is shown as it
allows a quick reference for any follow-up detailed inquires.
The location of the SAS data tables on the PJM Local Area Network is :
\\sas12vwp\sasAdHoc$\sasDepts\CapPlan\\sasAdHoc\sasDepts\CapPlan\PRISM_Debug_Folder
The following summary lists the information available, on an area and delivery year basis. These SAS data table summaries have been used to perform detailed
comparisons with the GEBGE calculations.
List of information in the DEBUG directories, for a given PRISM run number:
o
EWM Calculations - 52 Weekly values for: Original GrowX (historic growth rate), gsdn (mean), EWM, growx (final growth rate using load
forecast).
o
PRISM Parms: Indicates general solution parameters, name of user, solution run time, source location of 1) input file, 2) files containing
capacity base units, adders, changes, maintenance schedules, 3) yearly constants, 4) probability table 5) Capacity and load factors 6)
load models, 7) tie size and 8) solved data results.
o
PRISM Time: Start and end times for various steps in the solution process.
o
Period Probability: Similar to the output shown in Table 2 for the period probability category.
o
Summary Table: Similar to the output shown in Table 2 for the summary table category
o
Syspar (unit model aggregate parameters): Underlying data used to create the output in Table 2 for the Syspar category including
distribution values.
o
Weekly Solution: Values associated with the output shown in Table 2 for the period probability category.
o
Yearly Solution: The automatic solution process loads and resulting RI to iterate to the solution tolerance for the given desired RI (Typically
10 years - per occurrence)
o
Weekly EWM: Information similar to the Expected Weekly Maximum (EWM) calculations.
o
Control Card Data: The term “Control Card” is based on the historic card readers used to input data. This term was carried over to PRISM.
This is a direct list of the individual input data showing what was used for all input values. There are over twenty individual control cards
that are used for all aspects of the model.
© PJM Interconnection 2011. All rights reserved
Page 122 of 136
o
Outage Schedule: Individual unit schedule for planned outages, indicating the starting week and duration of the outage. This schedule is
typically automatically determined by the PRISM algorithms to maintain a constant level of available reserves over the delivery year
period.
o
Probable Peaks: 52 Weekly probable peak (mean) and system parameters (SYSPAR - System Unit Average) variable values.
o
Unit Descriptors: Individual list of Units in the model indicating Area number (1 or 2), Name, Unit number, ID, zone number and zone
descriptor.
There are 3 sub directories for each directory year:
o
Capacity Model: Data sets include added generation, base generation, capacity changes, weekly mlnst (MWs out on PO) variable, scheduled
planned maintenance, Base unit list (out area), All units(Base plus future units)
o
Input Data: added generation, area data, base generation, capacity changes, load model, schedule planned outages, yearly peak loads.
o
Load Model: plots load model variables, weekly load probable peak and capacity, yearly peak load.
The following files are for each Delivery year of a PRISM Run:
o
Area adder units
o
Area capacity changes
o
Area capacity model
o
Area capacity summary
o
Area capacity factors
o
Area cumulative probability table
o
Area load model
o
Area load model factors: weekly load model mean and standard deviation values.
o
Area manual maintenance
o
Area outage schedule
o
Area unit descriptors
o
Area unit list
o
Area unit data set
© PJM Interconnection 2011. All rights reserved
Page 123 of 136
o
Area Unit values by week
o
Area Updated model
o
Area Weekly EWM: Orig_growx, gsdn, EWM (MPP), growx.
o
Area Weekly capacity factors
o
Area data
o
Capacity changes intermediate steps, Temp, Temp2, Temp 3a, and Temp3b.
o
Core calculation intermediate steps: There are 3 places in the calculation process that are available for review. Core1 shows the load model
processing. Core2 shows automatic solution processing. Core5 has a format and data similar to the period probability table discussed in Table
2.
o
Unit availability state in each of 52 periods(Cumprob1)
o
Intermittent steps for capacity processing: Added generation, Adder units, capacity changes, capacity model.
o
Final run flag that indicates best solution iteration in automatic solution: main_finalprob, main_finalprobability, Main_finalrunflag
o
Several capacity model intermediate steps: unitds_1, updated_mode, schedule1, schedule2, unitdescriptors, units_area1, unitvaluesbyweek,
weeklycapadders, weeklycapearlyadders, weeklycapearlychanges, weeklycapmodel, weeklycapsortchanges, weeklycapsortmodel,
weeklycapstartmw.
o
Solution primary load, secondary load, and LOLE during automatic solution process. Weeklyoutptds, yearlyoutputds.
o
Load model intermediate steps involved with load distribution (zprob): weekly sigma value and associated probability (typically 21 points for
each week for each area). Weekly number of points in distribution. (Zprob, zprobtable, zsummary).
It was noted by the RRAWG recently and confirmed by the PJM staff that output of the internal calculated 21-point steps is not currently saved or available after
the final solution is determined. However the PJM staff has identified the issues needed to be addressed for this reporting and have established a plan to
complete the processing to output these intermediate steps. Due to the modular nature of the code and its ability to automatically find a solution, these 21-point
internal calculation values were not originally output due to the large file size.
MARS
OT 10 File
In the “Years to Study and annual Output” Options (pages 3-14 of the February 2009 MARS user manual) the Hourly output is listed in a column format shown in
Table 44. The Clock column starts at Hour 1, for which there are 8760 hours for a single replication. The clock does not get reset, so the value for the 2nd
replication starts at 8761.
© PJM Interconnection 2011. All rights reserved
Page 124 of 136
The load level is associated with the values given in the LOD-UNCY table, typically there are seven values (1 = largest Load level, 4 = 50/50 load level).
The Margin is from the EOP Table; typically there are six EOP levels used: 1 = Operating reserves, 2 = Load Management, 3 = 30-minute reserves, 4 = Voltage
Reduction, 5 =10-Minute reserves, and 6 = Appeals).
The isolated /Interconnected column (ISO/INT) indicates a 1 = Isolated and 2 = Interconnected ISO, and the area name columns show the amount of reserves
that were deficient causing an LOLE event, for that EOP level and LOD-UNCY load amount.
By using these values and the associated input probabilities, each replication calculates the resulting LOLE for the various LOD-UNCY distribution points and
EOP levels defined in the input data. (This output file can be very large (gigabytes).)
To measure and determine the values as shown in the OT9, values could be processed for each replication. The possible outcomes for a study that requires, for
example, 2000 replications, to solve to an acceptable Standard Error, assuming that 500 hours have a non zero risk(a negative value for load) could be 2000
(replications) x 500 (contributing hours) x 7(LFU values) x 6 (EOP levels) = 42 million values to evaluate.
It is important to note that the process would need to place the risk into the correct EOP bucket and that a check needs to be made to make sure that a given
hour is the daily peak, to match the reporting in the MARS Standard output OT files when the solution convergence index is option 3, for LOLE in Days/Year
(reference CNV-CRIT-00 Table). This process needs to be developed and once successful, automated, so it can be part of the GUI used to process any user
chosen model’s area.
Table 44 – MARS Hourly Output
CLOCK
LOAD LEVEL
MARGIN
ISO / INT
Area Margin
4193.0
4505.0
4505.0
4528.0
4529.0
4529.0
5369.0
5369.0
5369.0
5441.0
13265.0
13289.0
13289.0
13289.0
13310.0
13311.0
13312.0
13313.0
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
1
1
1
0
0
1
2
1
2
0
1
1
1
2
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
-1690.08
-3804.65
-477.65
-652.67
-4675.25
-746.25
-4836.28
-907.28
-1509.28
-2635.19
-613.57
-5173.03
-1244.03
-1846.03
-1449.30
-5661.37
-6784.05
-10463.62
© PJM Interconnection 2011. All rights reserved
Page 125 of 136
Additional details are available in the OT8, OT7, and OT 11 files, as described in Table 2 in the Period Probability category.
Comparison comments.
The determination of the capacity distributions in each of the MARS replications is not a simple and direct effort. The capacity outage distribution summary
might take more time for MARS while the load distribution summary in PRISM takes more time.
There are many, many calculation step values available in both tools. Both tools offer a process to improve the transparency of the processing steps by
requesting improvements to the staff that maintains and owns the intellectual property rights of the tool.
© PJM Interconnection 2011. All rights reserved
Page 126 of 136
Appendix F – Details of Input Parameter Requirements
The following shows the input parameters available for both PRISM and MARS. The specifics given should
allow the reader to discern the level of effort needed to make an analysis run, to observe the differences in
the event driven MARS inputs vs. statistical quantities for PRISM, and to capture the breadth and depth of
what is available to allow the tools to provide assessments for a large number of various and diverse
assessments needs. Both tools need hourly loads and individual unit GADS event data to perform an
assessment.
PRISM
The PRISM file format is automatically generated via a JAVA based GUI. The format uses a Control Card
(CC) category which is a holdover from historic formats used in the 1960s.
CC1 – Report Title
Cols 1-5
Cols 6-61
Control Code (01)
Report Title – One title line per area
CC2 – Report Remarks
Cols 1-5
Cols 6-85
Control Code (02)
Report Remarks
CC3 – Case Identification
Cols 1-5
Cols 6-13
Control Code (03)
Case ID
CC7 – Constant Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-16
Cols 17-26
Cols 27-31
Cols 32-36
Cols 37-41
Cols 42-44
Cols 45-46
Cols 47-56
Cols 57-66
Cols 67-69
Control Code (07)
Area Number
Study Year
Simulated Maintenance (Load?)
Probability Table Limit
Target Solved Reliability Index (RI)
Tolerance
Per Unit Reserve Factor
Days per Week
Calendar Week Equal to Period 1
First Order Statistic
Forecast Error Factor
Step Size
CC8 – Season Data
Cols 1-5
Cols 6-13
Cols 14-21
Cols 22-29
Cols 30-37
Control Code (08)
Spring
Summer
Fall
Winter
CC9 – Case Data
Cols 1-5
Cols 6-10
Cols 11-15
Control Code (09)
Study Start Year
Study End Year
© PJM Interconnection 2011. All rights reserved
Page 127 of 136
Cols 16-17
Cols 18-22
Col 25
Cols 26-27
Cols 28-59
Cols 60-61
Cols 62-63
Cols 64-65
Cols 66-75
Cols 76
Cols 77
Cols 68-107
Cols 108-147
Number of Areas
LAS Year
Modified Two-state (Y/N)
Normal Flag (Y/N)
Non-normal or Normal Curve data set name
CETO/CETL Flag (Y/N)
CETO/CETL Adders Flag (Y/N)
EFORd Flag (Y/N)
Class Average Years
PSSE Flag (Y/N)
OpGen Type (R=Rollup, D=Detail)
PSSE File Name
PSSE Adder File
CC10 – Distribution Data
Cols 1-5
Cols 6-15
Cols 16-25
Cols 26-27
Control Code (10)
Per Unit Sigma
Probability of Occurrence
Week Number
CC12 – Load Levels, Tie Sizes, and Margin Levels
Cols 1-5
Cols 6-10
Col 11
Cols 12-17
Cols 18-23
Cols 24-29
Cols 30-35
Cols 36-41
Cols 42-47
Cols 48-53
Cols 54-59
Cols 60-65
Cols 66-71
Cols 72-77
Control Code (12)
Year
cc12 Type (1-Load Levels, 2-Tie Sizes, 3-Margin Levels)
Value 1
Value 2
Value 3
Value 4
Value 5
Value 6
Value 7
Value 8
Value 9
Value 10
Value 11
CC16 – Capacity Factor Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-19
Cols 20-27
Cols 28-35
Cols 36-43
Cols 44-46
Cols 47-49
Control Code (16)
Area Number
Year
Spring Factor
Summer Factor
Fall Factor
Winter Factor
Start No Maintenance Period
Stop No Maintenance Period
CC17 – Load Model Data
Cols 1-5
Cols 6-7
Cols 8-15
Cols 16-23
Cols 24-25
Cols 26-37
Cols 38-43
Cols 44-93
Control Code (17)
Area Number
Mean or Trend Value
Standard Deviation
Week
Title
Load Model Type (Plots or Trend)
(Text used in recalling saved data)
© PJM Interconnection 2011. All rights reserved
Page 128 of 136
Cols 94-95
Cols 96-104
Season (1, 2, 3, 4)
Study Period (Ex. 1997-2001)
CC18 – Load Factor Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-13
Cols 14-21
Cols 22-29
Cols 30-37
Cols 38-45
Cols 46-53
Cols 54-61
Cols 62-69
Cols 70-77
Cols 78-85
Cols 86-93
Cols 94-101
Cols 102-109
Cols 110-117
Cols 118-125
Cols 126-133
Cols 134-141
Cols 142-149
Cols 150-157
Cols 158-165
Cols 166-173
Cols 174-181
Cols 182-189
Cols 190-197
Cols 198-205
Control Code (18)
Area Number
Year
Number of Factors (4 or 12)
Spring Factor or FCST Load Month 1
Summer Factor or FCST Load Month 2
Fall Factor or FCST Load Month 3
Winter Factor or FCST Load Month 4
FCST Load Month 5
FCST Load Month 6
FCST Load Month 7
FCST Load Month 8
FCST Load Month 9
FCST Load Month 10
FCST Load Month 11
FCST Load Month12
ALM Month1
ALM Month2
ALM Month3
ALM Month4
ALM Month5
ALM Month6
ALM Month7
ALM Month8
ALM Month9
ALM Month10
ALM Month11
ALM Month12
CC20 – Base Unit Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-33
Cols 34-38
Cols 39-43
Cols 44-49
Cols 50-55
Cols 56-59
Cols 60-61
Cols 62-63
Cols 64-65
Cols 66-67
Cols 68-69
Cols 70-71
Cols 72-73
Cols 74-75
Cols 76-77
Cols 78-79
Cols 80-82
Cols 83-122
Cols 123-186
Cols 187-198
Cols 199-203
Cols 204-211
Control Code (20)
Area Number
Unit Number
Unit Name
Summer Capability
Winter Capability
Variance
EEFORd
Maintenance Start Year
1st Maintenance Cycle
2nd Maintenance Cycle
3rd Maintenance Cycle
4th Maintenance Cycle
5th Maintenance Cycle
6th Maintenance Cycle
7th Maintenance Cycle
8th Maintenance Cycle
9th Maintenance Cycle
10th Maintenance Cycle
Zone Number
Zone Name
Owner Index
GEBGE Name
Unique Generator Name
MARS Name
© PJM Interconnection 2011. All rights reserved
Page 129 of 136
Cols 212-217
Cols 218-220
Cols 221-230
Cols 231-240
Cols 241-246
EFORd Value
Gen Class Key
Load Model Code
SAS Loc Key (From DM_TBLS.Geography_Xref)
XEFORd
CC20A – Case Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-51
Cols 52-61
Cols 62-64
Control Code (20A)
Area Number
SAS Loc Key (From DM_TBLS.Geography_Xref)
Zone Name
Load Model Code
Zone Number
CC21 – Adder Units
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-33
Cols 34-37
Cols 38-39
Cols 40-44
Cols 45-50
Cols 51-56
Cols 57-58
Cols 59-60
Cols 61-62
Cols 63-64
Cols 65-66
Cols 67-68
Cols 69-70
Cols 71-72
Cols 73-74
Cols 75-76
Cols 77-81
Cols 82-84
Cols 85-124
Cols 125-188
Cols 189-200
Cols 201-212
Cols 213-220
Cols 221-226
Cols 227-229
Cols 230-239
Cols 240-244
Cols 245-250
Control Code (21)
Area Number
Unit Number
Unit Name
Install Year
Install Period
Summer Capacity
Variance
EEFORd
1st Maintenance Cycle
2nd Maintenance Cycle
3rd Maintenance Cycle
4th Maintenance Cycle
5th Maintenance Cycle
6th Maintenance Cycle
th
7 Maintenance Cycle
8th Maintenance Cycle
9th Maintenance Cycle
10th Maintenance Cycle
Winter Capacity
Zone Number
Zone Name
Owner Index
Load Model Code
GEBGE Name
MARS Name
EFORd
Gen Class Key
SAS Loc Key
Unique Gen Key
XEFORd
CC22 – Capacity Changes (There can be multiple records per unit per year)
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-33
Cols 34-35
Cols 36-40
Cols 41-42
Cols 43-46
Cols 47-48
Cols 49-53
Control Code (22)
Area Number
Unit Number
Unit Name
Period Number
New Capacity
Change Month
Change Year
Unused placeholder
Winter Capability
© PJM Interconnection 2011. All rights reserved
Page 130 of 136
Cols 54-56
Cols 57-96
Cols 97-108
Cols 109-113
Zone Number
Zone Name
GEBGE Name
Unique Gen Key
CC23 – Manual Maintenance
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-33
Cols 34-37
Cols 38-39
Cols 40-42
Cols 43-44
Cols 45-46
Cols 47-48
Cols 49-50
Control Code (23)
Area Number
Unit Number
Unit Name
Change Year
Number of Periods
Zone Number
Period 1
Duration 1
Period 2
Duration 2
The periods and durations continue in pairs for the number of periods specified in columns 38-39.
Zone Descriptor is added to the end of the string starting one column after the periods and durations finish
and is 40 characters long. GEBGE Name is added after Zone Name for 12 characters. Unique Gen Key is
added after GEBGE for 5 characters.
CC24 – Tie Size
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-21
Control Code (24)
Area Number
Year
Tie Size
CC25 – Curtail Data
Cols 1-5
Cols 6-9
Cols 10-19
Cols 20-29
Cols 30-45
Cols 46-50
Col 51
Control Code (25)
Year
ALM
Reserve Value
Generator Name
Large Generator Threshold
ALM Run Type
CC26 – Solve Data
Cols 1-5
Cols 6-7
Cols 8-11
Cols 12-19
Cols 20-21
Cols 22-23
Cols 24-25
Cols 26-27
Cols 28-29
Control Code (26)
Area Number
Year
Load
Automatic (1 is automatic)
Print Seasonal LOLE
Print Planned Outages
Print Loads
Print Stored
CC31 – Generators table for PSSE
Cols 1-5
Cols 6-12
Cols 13-15
Cols 16-24
Control Code (31)
PSSE Bus Number
PSSE Machine Id
PSSE Bus Name
© PJM Interconnection 2011. All rights reserved
Page 131 of 136
Cols 25-54
Cols 55-59
Cols 60-64
Cols 65-66
Cols 67-68
Cols 69-90
Cols 91-100
Cols 101
Cols 102-111
Cols 112-115
Cols 116-119
Cols 120-149
Cols 150-153
Cols 154-155
PSSE Generator Name
PSSE Max MW
Operating Generator Capacity
PSSE Bus Count
Operating Generator Count
Plant Name
Generator Code EIA
Activity Code (1 = Adder, 2 = Change record, 3 = Retire)
Commercial Date
Generator Key Match
Generator Key
Non-unique Generator Name
ARC Year
ARC Week
CC32 – PSSE No match table for PSSE
Cols 1-5
Cols 6-12
Cols 13-15
Cols 16-24
Cols 25-29
Cols 30-36
Cols 37-46
Cols 47-49
Control Code (32)
PSSE Bus Number
PSSE Bus ID
PSSE Bus Name
PSSE Max MW
PSSE Area Name
Load Model Code
cc32 Type
CC33 – PJM Gens table (or all Operating Generators) for PSSE
Cols 1-5
Cols 6-27
Cols 28-37
Cols 38-70
Cols 71-73
Cols 74-78
Cols 79-83
Cols 84-89
Cols 90-95
Cols 96-97
Cols 98-103
Cols 104-106
Cols 107-170
Cols 171-182
Cols 183-190
Cols 191
Cols 192-201
Cols 202-205
Cols 206-207
Cols 208-214
Cols 215-220
Control Code (33)
Operating Generators Plant Name
Generator Code EIA
Zone Name
Sub-zone Number
Summer Capability
Winter Capability
EFORd
EEFORd
POF
Variance
Gen Class Key
Owner
GEBGE Name
GEMARS Name
Activity Code (1 = Adder, 2 = Change record, 3 = Retire)
Commercial Date
ARC Year
ARC Week
PSSE Bus ID
XEFORd
CC9999 – Separator
Cols 1-5
Control Code (9999)
© PJM Interconnection 2011. All rights reserved
Page 132 of 136
MARS
Table Category
Name
*
# Inputs
Description
General Study Data
GEN-CASE
GEN-POOL
GEN-AREA
GEN-ARGP
GEN-UNTY
GEN-PRNT
GEN-TIME
CNV-CRIT
GEN-OPTN
MNT-AOPT
INT-ONLY
11
2
6
3
2
4
9
13
5
6
12
Case Identification and General Options
Pool Identification
Area Identification
Area Group Identification
Unit Summary Type Identification
General Print Information
Years to Study and Annual Output Options
Simulation Convergence Criteria
Program Options
Annual Output Options
Infrequently Used Programs and Output Options
6
4
4
Annual Area Load Data
Annual Month-to-Annual Ratios
Load Forecast Uncertainty
7
General Unit Data
2
2
4
4
3
4
Capacity States
Thermal Unit Maximum Capacity
Thermal Unit Capacity Derations
Transition Rate Data
Forced Outage Rates
Number of State Transitions
6
3
4
2
7
2
2
2
2
2
Unit Maintenance Data Options
Thermal Unit Fixed Daily Maintenance
Non-Thermal Unit Fixed Daily Maintenance
ES and DSM Capacities for Maintenance Scheduling
Maintenance Scheduling Options
Cycles for Per Unit Planned Outage Rates
Cycles for Weeks of Planned Maintenance
Annual Peaks for Maintenance Load Model
Weekly Ratios for Maintenance Load Model
Capacity Adjustments for Maintenance Scheduling
Load Data
LOD-DATA
LOD-MTAR
LOD-UNCY
General Unit Data
UNT-DATA
Thermal Unit Data
UNT-CAPS
UNT-MXCP
UNT-DERT
UNT-TRNS
UNT-FORS
NUM-TRNS
Maintenance Data
MNT-UNOP
MNT-FIXD
MNT-FDMD
MNT-MDCD
MNT-OPTN
MNT-PLOR
MNT-WEEK
MNT-ANPK
MNT-RATI
MNT-FIXC
Type 1 Energy-Limited Unit
ELU-DIST
3
Unit Outage Distribution
Non-Thermal Units and Load Modifiers
MOD-ELMW
MOD-ELDD
5
3
Ratings for Type 2 Energy-Limited Unites
Loads for Scheduling Type 2 Energy-Limited Units
© PJM Interconnection 2011. All rights reserved
Page 133 of 136
Table Category
Name
MOD-CGMW
MOD-MDMW
MOD-SHAP
MOD-PENE
MOD-PRIO
# Inputs*
Description
2
2
3
2
1
Cogeneration Unit Hourly Load Demand
ES and DSM Net Hourly Load Modification
DSM Hourly Shapes
DSM Hourly Shape Penetration Factors
Priority Order for Scheduling Non-Thermal Units and Load Modifiers
7
Contract Data
4
4
3
5
11
Definition of Area Interfaces
Definition of Interface Groups
Identification of Closed Interface Groups
Interface Transfer Limits
Dynamic Interface Transfer Limits
1
4
Priority Order for Allocating Resources among Areas
Pool Reserve Sharing Data
Contracts
FCT-DATA
Area Interfaces
INF-DATA
INF-GRPS
INF-CLSD
INF-TRLM
INF-DYLM
Resource Allocation
RES-PRIO
RES-POOL
Emergency Operating Procedures (EOPs)
EOP-DATA
EOP-DLAY
11
2
Margin State Data for Modeling Emergency Operating Procedures
Data for Delayed Implementation of Emergency Operating Procedures
Histogram Post-Processor Data
HST-DATA
Total Parameters
7
Indices to be processed by HISTO
225*
*
The number of inputs indicated is an average estimate, for typical modeling. However they can vary due to a given
assessment model requirements. The full MARS User Manual is the best source for determination of input requirements.
The MARS User Manual is available to: 1) Licensed MARS users 2) parties that have executed a signed Use and nonDisclosure Agreement (NDA) with GEII.
In total, MARS has 50 input tables that allow approximately 225 parameters to be entered. The commonly
used tables are highlighted in yellow, which total 22 Tables and 121 parameters. There are a few additional
files necessary for analysis to start including the in02 file that stores the 8760 hourly loads, the in17 file that
has a resource hourly energy profile (MOD-SHAP), a control file, and a batch file.
In the proprietary MARS user manual the input parameter requirements are detailed over approximately 70
pages of text and figures. This manual is an excellent and detailed reference, extremely useful in
manipulating the various options that MARS allows the user to choose.
Taking advantage of MARS’ strengths requires using other Table inputs such as: a) the MOD-SHAP table to
specific wind unit profiles, b) MOD-ELMW and MOD-ELDD to investigate hourly pattern impacts, c) INFDYLM to adjust imports over several paths and d) HST-DATA to investigate and report underlying model
calculations and details.
Of the 50 MARS tables, there are several tables that are populated automatically via the ARC Java-based
GUI.
© PJM Interconnection 2011. All rights reserved
Page 134 of 136
Input parameter comparison comments.
Due to the length of time that the PJM Staff has used PRISM, there are many more automated facilities and
tools for PRISM data inputs than there are for MARS inputs. Many of the MARS tables require manual
intervention to establish correct entries. Both PRISM and MARS store the final input data file in a text-based
format which does allow for quick access to make changes.
MARS has more categories of input parameters; however both PRISM and MARS have load and capacity
inputs that enable many different assessments. MARS typically needs Emergency Operational Procedure
(EOP) data, reserve sharing, contract data, transmission pipe data, and load forecast uncertainty values for
each area modeled.
The detailed input requirements for PRISM are shown above. The MARS detailed input requirements are
covered in 70 pages of the MARS User Manual. PRISM has much detailed input requirements available in
its GUI and with approximately the same level of detailed as given in the MARS User Manual. There are 50
categories for MARS Tables and 23 categories for PRISM’s Control Cards (CC).
Both PRISM and MARS are part of the Application for Reliability Calculations (ARC) in the PJM Production
environment. The data is fed into the input tables via the database schemas and established process
options of the GUI.
© PJM Interconnection 2011. All rights reserved
Page 135 of 136
Appendix G – Items for possible future investigation
There are a number of future investigation activities that can supplement and augment this PRISM-MARS
comparison report. These items not only enhance the existing complementary tasks between both
programs but can also increase stakeholder understanding of each program.
The following potential future PRISM-MARS study activities are listed in priority order with deference to
resources and time needed to complete each task:
1.
While currently dedicated to using PRISM for resource adequacy studies, PJM also uses several
key MARS assessment modules to supplement this work (discussed in the “complementary”
section of this report.) For the benefit of PJM stakeholders, PJM Staff should make greater efforts
to clarify and document how MARS is already used in conjunction with PRISM for LOLE
assessments.
2.
With ISO-NE Staff, PJM Staff should more fully explore the use of PRISM’s probabilistic load
modeling (PLOTS) in assisting the development of the MARS Load Forecast Uncertainty table
(LOD-UNCY).
3.
MARS can assist the evaluation of high ambient conditions on generation resources and
performance by using its UNT-DERT table. (This is related to the item shown in the 2010 RRS
report, shown in section RRAWG activities, 4th bullet [page 48].)
4.
Interpret and develop analytical processes for reporting MARS output data (such as OT6, OT7,
OT8, OT9 and OT11 output files); including processes for MARS post-processing tools such as the
Histogram application.
5.
Compare and analyze the MARS hourly load shape file (in02 file) for reporting differences seen
with calendar year 2002, 2005 and 2006 – years identified in the industry as possible hourly load
shapes. Investigate a process for developing a “design year”.
6.
Investigate the NYSRC’s “Unified Methodology” procedure that coordinates and links the New York
Control Area (NYCA) IRM requirements with the Locational Capacity Requirements (LCRs)
developed by the NYISO. (LCRs are developed separately for the NYCA Zones J (New York City)
and K (Long Island) and are differentiated from the NYCA Rest of State (ROS).)
7.
Investigate the NYISO and MISO’s use of reporting IRM with MARS.
8.
Consider items in the PJM neighboring region’s MARS model and assessment methods. If needed
enter into a Non Disclosure agreement (NDA), so that any investigation is comprehensive and
measurable. These MARS modeling methods should initially be focused on the development of the
following MARS tables and coordinated within the NPCC CP-8 WG, MISO LOLE WG, IPSAC, and
the SERC Corporation RAWG. A long term goal of these efforts could be the development of a
process to transfer the following MARS data into the PJM database schemas.
o
o
o
o
o
o
o
LOD-UNCY values - Load uncertainty
In02 hourly load file values
EOP-DATA values - Emergency Operating Procedures levels
INF-TRLM values - Transmission pipe sizes
Specific Path related contracts
Severe case assumptions or sensitivity assumptions
Evaluation to aggregate supplied details to match the PJM data model needs.
© PJM Interconnection 2011. All rights reserved
Page 136 of 136
End: Comparison of PRISM and MARS
February 17, 2011
DOCs #599848
PJM Resource Adequacy Analysis Subcommittee (RAAS)
© PJM Interconnection 2011. All rights reserved
Last Page
