Resource ID and Recorder Extract
User Guide:
Resource ID and Recorder Extract
Version 2.02
07/28/2010
Resource ID and Recorder Extract – User Guide
ERCOT Public
Table of Contents
1.
Overview .................................................................................................................................................. 3
1.1.
Background....................................................................................................................................... 3
1.2.
Document Purpose ........................................................................................................................... 3
1.3.
Applicable Documents, Standards, and Policies ............................................................................... 3
1.3.1.
2.
Summary.................................................................................................................................................. 4
2.1.
4.
5.
Overview of the Resource ID (RID) and Non Relational Recorder (REC) Extracts........................... 4
2.1.1.
General Extract Information .................................................................................................... 4
2.1.2.
Extract Recipients ................................................................................................................... 4
2.1.3.
Receiving Data ........................................................................................................................ 4
2.1.4.
Data File Naming Conventions ................................................................................................ 4
2.2.
3.
Service Level Agreement ........................................................................................................ 3
About ERCOT Data Extracts ............................................................................................................ 5
2.2.1.
Data Definition Language Files ............................................................................................... 5
2.2.2.
Creating the Database Structure ............................................................................................. 6
2.2.3.
Applying Changes to the Database Structure ......................................................................... 7
2.2.4.
Loading Scheduled Extract Data ............................................................................................. 7
2.2.5.
Handling Exceptions................................................................................................................ 9
Content .................................................................................................................................................... 9
3.1.
Content Description .......................................................................................................................... 9
3.2.
Timing ............................................................................................................................................. 10
3.3.
Security Requirements.................................................................................................................... 10
Design .................................................................................................................................................... 10
4.1.
Format of the Extract ...................................................................................................................... 10
4.2.
DDL................................................................................................................................................. 10
4.3.
Table Types .................................................................................................................................... 10
4.3.1.
Dimensional Tables ............................................................................................................... 10
4.3.2.
Transactional Tables ............................................................................................................. 11
4.3.3.
RID Tables ............................................................................................................................ 11
4.3.4.
REC Tables ........................................................................................................................... 11
Delivery .................................................................................................................................................. 12
5.1.
General Delivery Information .......................................................................................................... 12
5.2.
API/Web Services ........................................................................................................................... 12
5.3.
Scheduling an extract ..................................................................................................................... 12
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
i
Resource ID and Recorder Extract – User Guide
5.4.
6.
ERCOT Public
Scheduling an extract ..................................................................................................................... 12
Appendices ............................................................................................................................................ 13
6.1.
Appendix A – Table Order for Daily Loading................................................................................... 13
6.2.
Appendix B – RID Extract Relationships ......................................................................................... 14
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
ii
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
1. Overview
1.1.
Background
This extract was developed due to a desire of Market Participants to have access to EPS (ERCOT
Polled Settlement) meter data. During development of the extract, the scope was expanded to include
TDSP read Resource ID data and also was expanded to provide ERCOT the capability of sending IDR
data to TDSPs that need the data for planning and operations purposes but otherwise would not have
access to the data (Recorder Extract Data).
1.2.
Document Purpose
This document describes the data contained within the Resource ID (RID) and Non Relational Recorder
(REC) extracts, audience for which the information is available, access and delivery information, an
explanation on how the report is designed and the tables contained.
The document is intended to provide a business understanding of the data contained and how this data
can be applied to the Market Participant. In addition, this document can also be used as a technical
reference for technical readers proficient with relational database programming and
administration and basic knowledge of relational modeling as it describes the environment
and strategies for data loading. The examples contained in this document use Oracle
architecture since ERCOT and many market participants already use this database. The
concepts, however, can be applied to any relational database system with adjustments .
When translating the information within this document to your own systems, please be aware that
modifications may be needed in order to accommodate your unique environment and thorough testing
is strongly advised.
Please note that this document is not intended as a complete guide for scheduled data extracts.
Supplemental information regarding individual reports and extracts will be communicated to the market
from ERCOT on an as needed basis.
1.3.
Applicable Documents, Standards, and Policies
The following Nodal Protocol(s) apply to Resource ID and Recorder Extracts:


Nodal Protocol Section for Recorder Extract (REC) 11.1.10 and 11.1.11
Nodal Protocol Section for Resource ID Extract (RID) 12.4.4.2.3.1 (5) , 11.1.10 (25) and 11.1.11 (3)
1.3.1.
Service Level Agreement
The Data Extract Working Group (DEWG) which falls under the Commercial Operations
Subcommittee (COPS) is responsible for helping to maintain and manage the Data Extract and
Reports SLA. This report currently falls under the Extract and Reporting SLA. Please refer to
the SLA section on ERCOT.com for the most up to date information on this report.
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
3
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
2. Summary
This section is broken into two distinct segments. The first section covers general information related to the
Resource ID and Recorder extract specifically while the second section covers general information pertaining to
ERCOT data extracts.
While data extracts are not intended to provide a single solution to resolve all Market Participant needs, they are
meant to provide Market Participants with the data sets used by ERCOT to manage and settle the energy
market.
2.1.
Overview of the Resource ID (RID) and Non Relational Recorder (REC) Extracts
The REC/RID Extracts are daily extracts containing load and/or generation data. The data sets contain
information pertinent to the recorder or resource ID read by ERCOT or the TDSP and SCADA data.
2.1.1.
General Extract Information
The REC extract provides data to Market Participants with proprietary and legal entitlement to
the data but are not actual owners of the meter point or Resource ID.
The RID extract includes all necessary reference table data and transactional changes
pertaining to resource data including Resource ID data, ERCOT Polled Settlement metering
information, generation unit telemetry, TDSP read generation, and relationship information.
2.1.2.
Extract Recipients
This extract is intended for Qualified Scheduling Entities (QSEs), Resource Entities (PGCs),
Transmission Distribution Service Providers (TDSPs) and Meter Reading Entities (MREs).
The RID extract goes to all Market Participants with ownership to the data as defined by
access to the data via the ERCOT system and the REC extract goes to Market Participants
with a legal right to this data but who are ‘non-owners’ of the data.
2.1.3.
Receiving Data
ERCOT will post the Resource ID and Recorder Extract Data to the ERCOT Portal every day.
All file naming conventions and structures will remain the same. See ‘Data File Naming
Conventions’ below for details.
Market Participants must have access to the ERCOT TML in order to retrieve their data. If you
do not have the appropriate access, please contact your User Security Administrator (USA) or
your ERCOT Account Manager.
2.1.4.
Data File Naming Conventions
The Resource ID and Recorder extracts that are posted to the TML Report Explorer Folder
named “Resource ID Extract” and “Recorder Extract” will have the following naming
convention:
CSV ZIP file:

rpt.000RPTID.000000000000DUNS.yyyymmdd.hh24miss.given_filename.zip
Where:
o
Filename = RID_Daily or RID Initial or REC_Daily
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
4
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
o
DUNS = DUNs number places padded with leading 0’s to 16 characters
o
RPTID = Report type ID for the report extract posted. The report type id for Recorder
Extract is 232 and the report type id for Resource ID Extract is 231.
The files that come inside the Resource ID and Recorder extract zip files from the ERCOT
portal will have the following naming conventions:

The naming convention of the counts files stored within the .zip file is:
[16-digit DUNS].[EXTRACT_NAME].COUNTS.csv
(DUNS number places are padded with leading 0’s)

The naming convention of the public files stored within the .zip file is:
[TABLENAME]-DD-MM-YY.csv

The naming convention of private files is:
[16-digit DUNS]-[TABLENAME]-DD-MON-YY.csv
(DUNS number places are padded with leading 0’s)
2.2.
About ERCOT Data Extracts
ERCOT data extracts provide a framework that allows market participants to retrieve ERCOT market
data for analysis. This framework is comprised of two elements: DDL and Data Extract distributions.
Data Definition Language (DDL)
ERCOT provides the structures for Market Participants to load ERCOT data into their own environment
in the form of data definition language (DDL). This DDL provides the metadata data for the data type of
each field, the table primary and foreign key constraints, and a brief description of the data that is to be
loaded into each column.
Data Extract Distributions
ERCOT utilizes a standard comma-separated value file format (CSV) for extract data delivery to ensure
portability across most platforms and architectures. These CSV’s are distributed to the market through
the Texas Market Link (TML) website packaged in ZIP files.
While data extracts are not intended to provide a single solution to resolve all market participant needs,
they are meant to provide Market Participants with the data sets used by ERCOT to manage retail and
wholesale operations and to settle wholesale capacity and energy markets.
2.2.1.
Data Definition Language Files
The data delivered to market participants comes from archived ERCOT Lodestar database
data. There is a specific methodology which should be followed for importing data. As
described in About ERCOT Data Extracts, ERCOT makes available a set of metadata data
files that contain data definition language (DDL) in Oracle format to create relational tables and
constraints (primary and foreign keys). ERCOT makes available a set of metadata data files
that contain data definition language (DDL) to create relational tables and constraints (primary
and foreign keys) that can store the data being extracted and delivered to market participants.
In addition, the DDL also contains database comments to define the expected use of each
table and field. While ERCOT provides DDL scripts in Oracle format, there are several CASE
tools on the market that can reverse-engineer this DDL file and create new DDL scripts for a
broad range of database products. A database administrator should also have the ability to
alter the DDL to represent the intended database structures for their particular environment.
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
5
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
The ERCOT provided DDL scripts (posted to the ERCOT Portal in the “Public” folder – “Extract
Data Definitions” subfolder) can be executed against an Oracle database instance. The same
DDL script can be executed more than once against a database without harming structures
that are already in place. Error messages will occur on DDL execution when the structure or
constraint is already in place. These messages do not harm the database structures. These
messages would include: “table already exists”, “table cannot have more than one primary key”
and “foreign key already exists in table.” See the “Creating the Database Structure” section
below for more details.
When there is a change in the requirements of the extract, ERCOT will generate and post a
new set of DDL scripts, reflecting the new table structure. When this occurs, ERCOT will send
out a market notification and produce both a complete DDL and an incremental DDL. If a
market participant has previously created the extract tables in their own database, they should
run the incremental DDL to apply the appropriate updates. If a market participant is new to the
extract process, they should run the complete DDL. Upon execution of the appropriate DDL
file, the extract database schema will be updated to accommodate the extract data in its new
format. Although running the complete DDL on your database will not harm your data
structures, failure to run an Incremental DDL change on existing databases could leave the
database without the required updates. This could cause data loading errors going forward.
The column comments provided within the DDL are to aid the user with the business
definitions of field values.
Please note that the DDL does not contain statements which define the physical storage
parameters of the individual tables. Storage values will vary greatly by market participant. The
DDL also does not contain performance-based indexes. If you have performance issues with
your queries, we suggest that you consult with your DBA.
2.2.2.
Creating the Database Structure
When a market participant is setting up a database for an extract for the first time, it is
important to determine if your company will benefit more from a single schema/database
containing all data retrieved from ERCOT with scheduled extracts or if it is best to generate
independent, private schemas/databases for each ERCOT extract. This is not an issue for you
if the Resource ID or Recorder Extract or both are the only ERCOT extract(s) that your
company uses.
If you decide to create a unified schema, keep in mind that one table can be defined in more
than one DDL file. Therefore, running all DDL scripts in a single schema will generate errors
indicating previous existence of foreign keys, primary keys and tables. ERCOT recommends
the use of a separate schema or database instance for this extract in order to minimize
confusion.
ERCOT recommends the creation of two database structures: a staging area and a work area.
The staging area should contain only table definitions (no primary or foreign keys) that will be
used for staging the data rows being imported. These staging tables would hold data
temporarily and will allow for better processing and error tracking. All staging tables MUST be
truncated to an empty state after each extract load or prior to the next extract load. The work
area will have the tables, primary keys and foreign keys as defined in the DDL.
This is a simplified example for the daily extract loading process using a staging area:
1. Download data extract Zip file from the ERCOT Portal
2. Extract .csv files from Zip file
3. Load all extracted CSV files into staging area
4. For each staging table (in the order found in Appendix A) iterate through each row:
a. Insert row - if there’s a primary key violation, use INSERT/ELSE UPDATE logic retaining the
appropriate record with the greatest add time (i.e. PRIMARY KEY and PIT_START) in your
database
b. Remove row from staging area
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
6
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
In order to implement this process, the market participant will need programmatic support.
There are several options for development and implementation: SQL*Loader, PL/SQL, PERL,
Visual Basic, etc. See “Loading Scheduled Extract Data” for more information about loading
data into DDL structures.
2.2.3.
Applying Changes to the Database Structure
The data extract files are based on a database model expressed by the DDL scripts. Every
time there is a change in the underlying data structures, a new DDL script will be released by
ERCOT. As mentioned previously, ERCOT produces a complete DDL and an incremental DDL
every time a change is necessary.
Following is a list of possible changes to the database and courses of action. This is a general
guide, not an all-inclusive list.

New Table
Create new tables in your database based on your DDL (and staging area, if you have
one) and import the data from the extract. Transactional table data will begin appearing
on the day the new DDL is scheduled to be implemented. Dimensional data tables
(e.g., QSE) will receive a complete load of the records on the go-live date relevant to
the market participant. Subsequent data extracts will contain any new or changed
records in the ERCOT system for the new table.

Table Removed
Drop the table from your system. ERCOT will provide detailed instructions, as well as a
new DDL, for these types of database changes.

Column Removed
In Oracle, it is possible to issue an “alter table” command with a “drop column” action.
For other databases, perform the appropriate action to achieve the desired result (this
may include the creation of a temporary table followed by the re-creation of the table). If
the column is part of the primary key, there will be foreign keys on other tables affected
by the change. The constraints must be dropped before making the changes (on all
affected tables) and recreated afterwards.

Added Column
In Oracle, a column can be added by issuing an “alter table” command with an “add”
option. In most cases the column can be added at the appropriate time and with proper
adjustments, the load process will proceed seamlessly. If the new column has been
added to the primary key of a table, all child tables will be changed as well. Constraints
must be dropped before adding the column and recreated afterwards. If the column is to
be included in the primary key there may be special instructions on how to initialize the
values for the column (i.e. no nulls).
2.2.4.
Loading Scheduled Extract Data
Once the ZIP file is retrieved from the market participant folder in the ERCOT Portal, it should
be expanded into a directory and inspected for completeness. Each individual CSV inside the
ZIP file contains data for a single table. The table name and processing date are part of the file
name. For tables that are transactional in nature, the market participant DUNS number will
also appear in the name of the CSV.
The file format is a standard comma-separated values file. It can be opened using Excel if
there is a desire to view the contents on an ad hoc basis. It is important to note that text fields
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
7
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
are enclosed in quotation marks (“). The tool used for importing the data (such as Oracle’s
SQL*Loader) should be set up to expect the quotation marks in order to load the data
correctly. A comma inside a text field is a valid value so it is necessary to delimit text fields in
this manner.
ERCOT recommends using the date embedded in the name of the .csv file for each table to
determine load order if you are processing more than one day of extracts at any given time.
Example: PL/SQL procedure to load table from “staging” area to “work” area
Following is an example of a SQL*Loader process to load the QSE table. First, create a
working directory and place the CSV file in that directory. Create a SQL*Loader control file in
that directory and call it QSE.CTL. For example:
LOAD DATA
INTO TABLE RESOURCEID
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
TRAILING NULLCOLS
(QSECODE VARCHAR2(64),
QSENAME VARCHAR2(64),
STARTTIME
DATE "mm/dd/yyyy hh24:mi:ss",
STOPTIME
DATE "mm/dd/yyyy hh24:mi:ss",
ADDTIME
DATE "mm/dd/yyyy hh24:mi:ss",
DUNSNUMBER VARCHAR2(64),
UIDACCOUNT NUMBER(10),
PIT_START
DATE "mm/dd/yyyy hh24:mi:ss",
PIT_STOP
DATE "mm/dd/yyyy hh24:mi:ss",
OV_ID
INTEGER EXTERNAL)
Please note that the control file lists all columns found in the table definition in the DDL file in
the same order. This is very important because SQL*Loader will use those names and order to
place data in the correct columns. After creating the control file, run the SQL*Loader utility
passing the CSV file name (which will change from day to day as the processing date
changes) as a parameter:
sqlldr userid=dbuser/dbpassword file=RESOURCEID-03-MAR-03.csv
control=RESOURCEID.csv
Example: PL/SQL procedure to load table from “staging” area to “work” area
ERCOT recommends the use of staging tables in the process of loading data. Staging tables
are temporary tables that have the exact same structure as their production counterparts but
none of the restrictions (no primary keys or foreign keys). The staging area allows you to load
data into the database tables in any order you want and then process this data routing valid
rows to the actual production tables. The procedure below, coded in PL/SQL (language
supported by the Oracle database), gives an example of how the transport of data from the
staging table into the work table could be implemented:
CREATE OR REPLACE PROCEDURE LOAD_RESOURCEID IS
BEGIN
FOR R IN (SELECT * FROM STAGE_RESOURCEID) LOOP
BEGIN
INSERT INTO RESOURCEID (UIDRESOURCEID,
RESOURCEID,
STARTTIME,
STOPTIME,
ADDTIME)
VALUES (R.UIDRESOURCEID,
R.RESOURCEID,
R.STARTTIME,
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
8
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
R.STOPTIME,
R.ADDTIME);
EXCEPTION
# INSERT FAILED. TRY UPDATE <- comment
WHEN DUP_VAL_ON_INDEX THEN
UPDATE RESOURCEID
SET UIDRESOURCEID = UIDRESOURCEID,
RESOURCEID = RESOURCEID,
STARTTIME = STARTTIME,
STOPTIME = STOPTIME,
ADDTIME = ADDTIME
WHERE UIDRESOURCEID = R. UIDRESOURCEID;
END;
END LOOP;
END;
2.2.5.
Handling Exceptions
Foreign Key Error
This means that a table’s row is being loaded before its parent record is loaded causing a
foreign key error. To solve this problem, it is necessary to load the CSV’s in the correct order.
The loading of the RID/REC DDLs do not add any Foreign Key constraints, so the error will not
be produced loading the extracts into these structures. This error would only be generated if
referential integrity is enforced through additional Foreign Key constraints
Duplicate Primary Key
If a circumstance occurs that causes a duplicate, the row with the greater PIT_START should
be retained, unless a history of all transactions is being kept within the database. The record
with the latest PIT_START will be the most recent version of the record. Anytime a duplicate
row is identified and there is no difference in the PIT_START or PIT_STOP columns, then one
row should be deleted, as these would be redundant.
3. Content
3.1. Content Description
The below tables are included in the extract:














MRE – This table stores the Meter Reading Entities found in the ERCOT Market
PGC – This table stores the Power Generating Companies found in the ERCOT Market
QSE – This table stores the Qualified Scheduling Entities found in the ERCOT Market
REP – This table stores the participating Retail Electric Providers
TDSP – This table stores the participating Transmission/Distribution Service Providers
PGCSERVICEHIST – This table documents the historical relationships between PGCs and their
associated QSEs.
GENERATORSITEHIST – This table contains the history of the Generator Site Table. The
Generator site is the facility that contains one or more generators.
REPSERVICEHIST – This table identifies the assignment of retail electric providers to scheduling
entities over time.
RECORDER – This table lists all the physical devices (i.e. meters) that are set up to receive
interval data.
NOIEHIST – This table stores characteristics associated with a particular NOIE.
NOIETIEHIST – This table stores characteristics associated with a NOIE metering point.
PGCOWNERSHIP – This table specifies the specific percentage amount each owner has in a
multi-owned generator site. It also defines the virtual meter point used to store that amount of
generation from the one generator.
PGCMASTEROWNER – This table identifies which PGC is the master owner of a multi-owned
gensite and identifies any other owners of a gensite.
CHANNEL – This table is used to determine if the energy data is generation (channel 1) or load
channel (4).
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
9
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________




RESOURCEID – This table describes meter points read by ERCOT, SCADA feeds, or load points
tied to a generation unit that is TDSP read and their settlement characteristics.
LSCHANNELCUTHEADER – This table contains summary information for interval data cuts
LSCHANNELCUTDATA – Table contains the interval information for interval data cuts
LSCHANELCUTSTATUS – Table contains the parsed data information for interval data cuts.
3.2. Timing
The Resource ID and Recorder Extract is a daily extract that posts as a single operating day per extract
from 00:00:00 - 23:59:59 for a given operating date. There may be multiple extracts that post per day
but each one is specific to an operating date and settlement run. The extracts are available Monday
thru Sunday
3.3. Security Requirements
The Resource ID and Recorder extract is a certified classification of data available via the TML and API.
In order to access the extract, a Digital Certificate with specific permissions is required. A Digital
Certificate must be obtained from your entity’s User Security Administrator. If you are unsure who your
company’s USA is, please contact you Account Manager or contact the ERCOT helpdesk for addition
information.
4. Design
4.1. Format of the Extract
The extract is available as a CSV file type. See 2.1.4 ‘Data File Naming Conventions’ above for file
naming conventions for both zip and table level file name.
4.2. DDL
The DDL associated with this report is available on the Texas Market Link Report Explorer in the public
folder in a subfolder titled: Extract Data Definitions. The DDL name for this report is Resource
ID/Recorder Extract. The most current DDL for all reports and extracts will be available in this location.
Please note that a Digital Certificate is required to access this area.
4.3. Table Types
4.3.1.
Dimensional Tables
Dimensional table data is provided to all Market Participants. The dimensional data tables are
as follows:





MRE
PGC
QSE
REP
TDSP
All market participants will receive a Resource ID Extract for data changes to Dimensional
Data tables.
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
10
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
4.3.2.
Transactional Tables
Transactional table data is protected data. The transactional data tables are as follows:













4.3.3.
CHANNEL
GENERATORSITEHIST
LSCHANNELCUTDATA
LSCHANNELCUTHEADER
LSCHANELCUTSTATUS
NOIEHIST
NOIETIEHIST
PGCMASTEROWNER
PGCOWNERSHIP
PGCSERVICEHIST
RECORDER
REPSERVICEHIST
RESOURCEID
RID Tables
Resource ID Level data is only sent to the appropriate market participant data owners based
on the relationships in the PGCSERVICEHIST and REPSERVICEHIST tables. The data here
generally pertains to both load and generation data. The tables that contain Resource ID
Level data are as follows:

















CHANNEL
GENERATORSITEHIST
LSCHANNELCUTDATA
LSCHANNELCUTHEADER
LSCHANNELCUTHEADERSTATUS
MRE
NOIEHIST
NOIETIEHIST
PGC
PGCMASTEROWNER
PGCOWNERSHIP
PGCSERVICEHIST
QSE
RECORDER
REP
REPSERVICEHIST
TDSP
Market Participants will only receive Resource ID Level tables in their extracts when there are
related Resource ID Level data changes within the extract time window.
4.3.4.
REC Tables
Recorder Level data is only sent to the appropriate market participant based on the data
sharing agreements made between the Market Participants. These tables commonly contain
load data information. The tables that contain Recorder Level data are as follows:






CHANNEL
LSCHANNELCUTDATA
LSCHANNELCUTHEADER
LSCHANNELCUTHEADERSTATUS
RECORDER
RESOURCEID
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
11
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
Market Participants will only receive Recorder Level tables in their extracts when there are
related Recorder Level data changes within the extract time window.
5. Delivery
(Where the extract resides, posting times, market agreements, etc)
5.1.
General Delivery Information
This extract is available on the TML as well as the API. The extract is available in CSV format and is
posted as a zip file. Accordingly, the DDL has been published for this extract. The extract is posted on
a daily basis by noon according to protocol .
As stated above, there will only be one trade day worth of data per operating day. As REC/RID are
proprietary extracts, there will be only 1 zip file posted for each run.
Extracts delivered to the market have historically been delivered in .zip packages with .csv file
components. If no data change has occurred for a particular table included in the Resource ID and
Recorder extracts, the .csv file is not included in the .zip package. At no time is an empty .csv file sent
to the market. In addition, if no data change has occurred for all the .csv files within a particular .zip file,
a .zip file will not be sent to the market.
5.2. API/Web Services
To programmatically download the information from the API, the user needs to use the reports “report
type” ID in order to “Get report” or “Search” for the report on the API. The Resource ID (RID), report
type ID = 231 and Recorder extract (REC), report ID = 232.
5.3. Scheduling an extract
To receive the REC Extract, the Market Participant must go through a legal documentation process with
ERCOT and the owner of the meter point in order to obtain authorization to the data. The data provided
upon such agreement is the same load data set as provided in the RID Extract.
5.4. Scheduling an extract
To schedule an extract, Market Participants must access TML using their digital certificate. Once logged
into TML, Market Participants will need to access the Archive tab in the navigation menu to schedule an
extract.
From the Archive tab, Select ‘Schedule/Unschedule an Extract’ from the ‘Schedule an Extract section
part ways down the left hand menu tab. Once you have selected ‘Schedule/Unschedule an Extract’, an
index of available extracts based on your entity role will appear.
Each link will lead to a list of extracts available to the Market Participant To schedule, simply check the
box next to the extract and click the Update button. If the extract is already scheduled, the word
“Scheduled” will appear in place of a check box:
To unschedule an already scheduled extract, click the Unschedule an Extract button on the
bottom of the page:
For additional help on any of the items in the screen, click on the blue “?” button in the top
right corner.
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
12
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
6.
Appendices
6.1. Appendix A – Table Order for Daily Loading
RID Extract
Load set one (may run in parallel)
• MRE
• PGC
• QSE
• REP
• TDSP
Load set two (may run in parallel)
• PGCSERVICEHIST
• GENERATORSITEHIST
• REPSERVICEHIST
• RECORDER
Load set three (may run in parallel)
• NOIEHIST
• NOIETIEHIST
• PGCOWNERSHIP
• PGCMASTEROWNER
• CHANNEL
• RESOURCEID
Load set four (in order)
• LSCHANNELCUTHEADER
• LSCHANNELCUTDATA
• LSCHANELCUTSTATUS
REC Extract
(In order)
• RECORDER
• CHANNEL
• RESOURCEID
• LSCHANNELCUTHEADER
• LSCHANNELCUTDATA
• LSCHANELCUTSTATUS
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
13
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
6.2. Appendix B – RID Extract Relationships
6.2.1.
GENSITE Driven RID Relationships
The GENSITE Driven RID relationship is one of 2 RID datasets (the other being NOIE Driven RID) that
will be included in the RID extract. The GENSITE Driven RID relationship consists of EPS, SCADA,
and TDSP read data. This data is relevant for TDSP, MRE, RE, and QSE entities. It is not relevant for
LSE’s.
DATA OWNERSHIP MODEL:
GENERATORSITEHIST
TDSP
RESOURCEID
MRE only
PGC
PGCSERVICE
HIST
PGCMASTER
OWNER
QSE
GENERATOR
SITEHIST
RESOURCE
ID
PGC
PGCSERVICE
HIST
PGCOWNERSHIP
GENERATOR
SITEHIST
QSE
RESOURCE
ID
*Resource id then joins to LSCHANNELCUTHEADER, LSCHANNELCUTDATA, and LSCHANNELCUTSTATUS
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
14
Resource ID and Recorder Extract- User Guide
ERCOT Public
__________________________________________________________________________________________________________________________________________
6.2.2.
Usage Example:
The above model will be used as a guideline for the distribution of extracts. For example, when
distributing the Resource ID data from this model to a QSE, Resource ID records may qualify for
distribution using 2 separate criteria:

Net Change:
A QSE will receive Resource ID records that have changed since the last extract delivery based
on the relationships shown below.

Ownership:
Ownership may also be cause for data distribution, for example, a PGC can begin using a
different QSE. Upon this ownership update in the PGCSERVICEHIST table, the QSE should
receive all RESOURCEID records that are related to the newly associated PGC.
Delivery or data for this scenario would be based on the relationships as seen below:
RESOURCEID Distribution
QSE
6.2.3.
RESOURCEID
(Net Change)
PGCMASTEROWNER
PGCSERVICEHIST
(Ownership)
PGC
NOIE Driven RID Relationships
The NOIE driven RID relationship is one of 2 RID datasets (the other being GENSITE Driven RID) that
will be included in the RID extract. The NOIE driven RID relationship consists of EPS data only. This
data is relevant for TDSP, LSE, and QSE entities. It is not relevant for RE’s. In addition, because the
associated MRE is always ERCOT, MRE is not relevant for the NOIE driven RID relationship.
DATA OWNERSHIP MODEL:
NOIETIEHIST
TDSP
REP
RESOURCEID
NOIEHIST
REPSERVICEHIST
QSE
*Resource id then joins to LSCHANNELCUTHEADER, LSCHANNELCUTDATA, and LSCHANNELCUTSTATUS
** There are 2 possible join paths between NOIETIEHIST and NOIEHIST: 1) NOIETIEHIST.NOIECODE = NOIEHIST.NOIECODE
or 2) NOIETIEHIST.ADJNOIECODE = NOIEHIST.NOIECODE
© 2006 Electric Reliability Council of Texas, Inc. All rights reserved.
15