Virginia Star Quality Initiative
Classroom-Based Rater Guidelines
These guidelines are designed to be a resource for training, observation and reporting processes for the
Virginia Star Quality Initiative (VSQI) for classroom based programs/centers. The Virginia Department of Social
Services (VDSS) through the Office of Early Childhood Development (OECD) and the Virginia Early Childhood
Foundation (VECF) co-sponsor the VSQI and are referred to as the “Hub”. Any questions regarding the content
of these guidelines should be addressed to OECD/VDSS and VECF/VSQI.
TRAINING
Raters must complete specified training and field work on:
Early Childhood Environment Rating Scale Revised Edition (ECERS-R)
Infant Toddler Environment Rating Scale Revised Edition (ITERS-R)
Pre-K and Toddler Classroom Assessment Scoring System (CLASS)
Report Writing
Consistency:
Consistent, reliable use of the observation tools is essential. Approved raters recognize the same behaviors
and assign similar scores across programs. All raters must reach consistency of
85% on the ERS and reliability of 80% on the CLASS. Inter-rater reliability must be maintained.
Refer to the “Inter-rater Reliability Process” section of this document for more details. Raters
not reaching consistency during training will be responsible for obtaining training to consistency
at their own expense and submitting the appropriate training documentation to the “Hub” prior
to being approved to conduct a VSQI rating observation.
PRE-K CLASS OBSERVER CERTIFICATION
After CLASS training, those who pass the CLASS reliability test are certified for one year to use the CLASS tool to observe
and score classrooms. All CLASS observers must recertify annually, the cost is the responsibility of the rater.
Certification Process:
Within two weeks of completing the new rater training, raters are enrolled in the on-line certification process.
The subscription is good for 1 month from the date of the notice.
 Once enrolled, raters will receive an e-mail providing login in information to the certification site.
 Raters should take time to study the CLASS manual prior to taking the certification test.
 Raters should expect to spend approximately three hours completing the test, which includes
observing and coding classroom videos.
 Raters have 1 month to complete the certification process.
 Raters not completing the certification process in the allotted time will be responsible for paying the
fee for a new subscription.
 Raters not passing the certification process will be responsible for paying any additional fees related to
obtaining certification.
Rater Guidelines
1|Page
After completion of the certification:
Print the certificate of completion provided at the end of the training.
Mail, fax or email the certificate and the reliability scores received during certification to: Sandra Wilberger,
VSQI Training Coordinator, VCU T/TAC 10 E. Franklin St. Ste. 200 Richmond, VA 21219, Fax 804-828-7495,
slwilber@vcu.edu
SCHEDULING
Establish a three-week window for the observation:
The local coordinator or their designee will establish a three-week period during which the observation will
occur including any days within that three-week window when the observation cannot occur. There may be
three “unavailable” days within the three-week period. The local coordinator will inform the rater of their
assignments and timeframes. If a rater is unable to complete the observation during the timeframe given, they
should contact their local coordinator as soon as possible. Raters are responsible for selecting the actual date
of the observation and relaying that information to the local coordinator. No specific dates for the observation
should be given to the program.
Contact the director to confirm the 3-week window:
At least one week prior to the beginning of the 3-week window, the rater should call the director to confirm
the weeks of possible observation, unavailable dates, and the time the rater will be arriving. The rater should
confirm the ages of the children enrolled and inform the director that they will need a copy of the daily
schedule for each classroom observed on the day of the observation.
When a program has multiple classrooms, they are randomly selected for observation when the rater arrives.
All classrooms with children ranging in age from 16 months to five years may be observed. Teachers should be
notified that their class may be selected. The rater should confirm the ages of the children in these classrooms
upon arrival to what tool(s) will be needed.
To prepare for the visit, raters should:
 Verify the address, directions and phone number for the program
 Record the program name and director’s name
 Review ERS (ECERS-R and ITERS-R) and CLASS (Pre-K and Toddler) and additional notes for clarification
 Select blank score sheets for each instrument
 Take all observation tools, pencils, measuring tape, clipboard, and a probe
 Include hand washing procedures, meal guidelines, and playground information to use
with the ERS
 Leave personal items in the trunk of their car
OBSERVATION GUIDELINES
Random selection of classroom(s) to be observed: One out of every three classrooms should be
observed, and at least one classroom for each relevant age group must be observed. If there is more than one
classroom for a relevant age group, each classroom should be assigned a number. Ask the director to pick a
number for the age group and that will be the classroom observed on that day. (Be mindful that this may
2|Page
provide some anxiety for the director.) When multiple classrooms need to be evaluated, the Star Quality
Rating process will take more than one day.
Complete the following matrix to determine how many classroom observations to conduct:
Relevant Age Groups
# of classrooms where the #
of
classroom
majority (51% or more) of observations to conduct
the children are in this age
range
36 months to 5 years
(ECERS-R/ Pre-K CLASS)
30 months to 36 months
(ECERS-R/ Toddler CLASS)
16 months to 30 months
(ITERS-R/ Toddler CLASS)
When there is a substitute: If the person left in charge of the classroom is the assigned classroom
assistant teacher or a “classroom floater”, then continue with the observation. You will note this information
in the “Program Information” section of your summary report. If a substitute has been called to fill-in, and
they are not at the program on a regular basis, then you may ask to select a different classroom to observe. If
there are no additional classrooms, you will need to arrange to complete the observation on a different day to
get a “typical” day in the life of the classroom.
Daily schedule:
Ask the director to give you a copy of the selected classroom’s daily schedule. This schedule will be for you to
use during the observation.
Conflict of Interest and Confidentiality:
A conflict of interest is defined as any relationship between a Star Quality Rater and a program, which could
interfere or be perceived to interfere with the rater’s ability to exercise objectivity in the rating process. Raters
should recuse themselves from an assignment due to conflict of interest if they:
 are/were employed by the program within the last 10 years;
 serve(d) as a mentor, consultant, or evaluator in the program;
 have a close relative who is/was employed by the program or is/was a mentor,
consultant, or evaluator;
 are or have served on the program’s Board of Directors, or have a relative in that
position;
 have a monetary or personal interest in the outcome of the program’s Star rating;
 have a significant personal relationship with individuals involved in the program; or
have any relationship that might compromise the objectivity of the ratings process .
 Confidentiality is holding in trust any confidential information or documents received through the
observation. All information received will be used for the purposes of this observation and will not be
disclosed to any third party without approval from the participating provider.
 The local coordinator should be notified immediately of any potential conflict of interest.
3|Page

Conflict of Interest Waivers and Confidentiality Forms should be completed prior to the observation
and submitted to the local coordinator.
Multiple classrooms to be observed in the same program:
All classroom observations must take place within the three-week window for the observation.
Programs with multiple classrooms to be observed within the three-week window should be observed by the
same rater whenever possible. This will allow summary reports to have one writing voice.
Securing materials:
Raters should be aware of all their materials at all times. It is very easy to leave an instrument on a bookcase
or shelf while completing other components of the observation. Raters should be sure score sheets are always
out of the teacher’s view and that they have all materials, especially the score sheets, when leaving the
program.
Arrival time:
Raters should attempt to arrive no later than 8:30, although it may be necessary to arrive earlier depending
upon the hours of the program. Allow enough time to conduct a complete observation.
Dress code:
Raters should dress professionally and comfortably, and wear comfortable shoes as they could be standing for
long periods of time. Raters may be sitting on the floor and bending over to count materials. Raters should
always remember that they are representing the Virginia Star Quality Initiative.
Cell phone and other electronic devices:
Raters should leave cell phones (including Bluetooth headsets) and PDAs in trunk of the car. This way they will
not be distracted by a call, text or email. If a rater is on the phone or texting, they cannot be actively
observing.
Meeting the director:
Upon arrival at the program, raters should introduce themselves to the director, and let the director know
how long they anticipate being at the program and that they will be in the classroom and following the
children for the entire observation. They should ask the director if she will be available after the observation is
complete to answer some questions. They should also ask the director what bathroom to use and if there are
any specific policies, they need to know about.
Eating and drinking while at the program:
Raters should refrain from eating any food served at the program. Some programs have strict restrictions
concerning food brought into the program. Observations take several hours. During that time the rater may
need to eat a snack and continue with the task.
Explaining the process to the classroom teacher:
Raters should introduce themselves to the teacher(s) in the classroom, and help the teacher(s) feel at ease by
explaining that they will be writing rapidly during the time they are in the classroom which is roughly 4-5
hours. Raters should explain that they are not able to join in activities and that they will try to be as
unobtrusive as possible. Raters should explain that they cannot be counted in the ratio or be left alone with
4|Page
the children at any time. Raters should explain that once the observation is completed they will need to ask
the teacher(s) a few questions about things they are unable to see during the observation. Raters should let
the teacher(s) know that they will be looking for materials and counting items so it may be necessary to open
doors and cabinets the children access. Raters should inquire if there is a specific chair that can be used during
the observation to sit down.
Talking to the teacher during the observation:
Raters should be as quiet as possible and refrain from asking the teacher questions until the teacher interview
at the end of the observation.
What if I observe neglect or child abuse during an observation?
Mandated reporting laws require that professionals who deal with children report suspicious circumstances
suggestive of child maltreatment to child protective services. In addition to requiring professionals to report,
the laws generally outline penalties for failure to report. Mandated reporters should not wait for a child to
spontaneously tell them about the abuse. It is not the reporter’s task to substantiate the abuse, only to report
reasonable suspicion. If a rater suspects any type of abuse, the rater should end the observation immediately,
go to the director, and inform the local coordinator that the rating observation has been terminated due to
the suspicion. The hotline number should then be called. The Child Protective Services’ Hotline number for
Virginia is 1-800-552-7096.
If children approach:
Children often want to know who is visiting in their classroom. Raters should briefly tell children their name
and that they are “writing down all the fun activities that you are doing in your classroom”. Raters should
refrain from participating in classroom activities, assisting children in the bathroom and serving meals. At no
time should the rater be left alone with the children.
Raters may not be counted in the staff to child ratio.
Positioning in the classroom:
The CLASS instrument requires the rater to hear and observe language and communication between children
and staff and children and their peers. Therefore raters will need to move from time to time to be close to the
“action”. Raters should try not to distract the teacher when doing so. When the children go outside or to
another area of the building, raters should follow them.
Opening cabinets:
Materials that are in plain view and accessible to children constitute the materials to be counted as materials
located in the classroom. When the teacher is interviewed, she may indicate that materials are in locked
cabinets for rotation. Raters should ask the teacher or director to see the materials.
Facial expressions and body language:
Raters should remain objective and focused during the observation, and be careful about negative body
language as well as facial expressions.
5|Page
COMPLETING THE OBSERVATION
The 4 cycles of CLASS:
Only 1 cycle of the CLASS observation should be scored during routine care (i.e. meals,
transitions/bathroom breaks, etc.); the other 3 cycles should be scored during whole group, small group, and
individual choice time or center times. The CLASS observation may not be scored during outdoor free play.
Interviewing the classroom teacher:
At the end of your observation, the rater will need to ask the Lead Teacher questions about the items that
were not observed. Raters should refer to the appropriate ERS (ITERS-R or ECERS-R).
The page with the items “notes for clarification” includes suggested questions for indicators that are
frequently not observed. Raters should ask non-leading questions. For example: “Are the children permitted
to use the computer? If yes, for how long?”
The rater should not interrupt the teacher during class time with questions. Raters should wait until the end of
the observation to ask questions. A good time is when the children are down for their naps.
Interviewing the director for the Parents and Staff subscale:
Several of the indicators found in the ERS subscale Parents and Staff can be observed during the
observation. Raters should familiarize themselves with these indicators in order to keep the
interview process with the director to a minimum. All other Parents and Staff indicators will be scored
through interviewing the director or the staff in charge for the day. Raters should refer to the appropriate ERS
(ITERS-R or ECERS-R). The page with the items “notes for clarification” also includes suggested questions for
these indicators.
Thanking the teacher(s) and/or the director/provider:
Raters should thank the teacher(s) for allowing them to come into the classroom. Raters should thank the
director/provider after the interview session for their participation in the VSQI. Raters should not offer any
information on the outcome of the visit. Raters should explain that the director will receive a summary report
of the observation as soon as the information has been reviewed.
Leave with all your materials:
Raters should check to make certain they have all materials with them before leaving the program. Any
materials left behind opens up the possibility of the program reviewing score sheets.
Reviewing the score sheets and determining final scores:
After the observation, raters need to review their ECERS-R/ITERS-R score sheets. Raters should use their All
about the ECERS-R or All about the ITERS-R book to determine final scores. Raters should check to
verify the correct score is circled based on the indicators selected. Final scores should be entered on the VSQI
website.
6|Page
INSTRUCTIONS FOR ENTERING SCORES ON VSQI WEBSITE
Rating Forms
Once the application has been processed at the “Hub” and a rater is assigned, an email will be sent to the
rater notifying them that they have been assigned a rating and now have access to enter scores.
Once a rater is logged in, they will view a list of the programs that they are currently assigned to rate. Select
“Pending” under “Review Rating Forms” to complete forms for a specific program.
Rater will be prompted to:
 Complete the Waiver of Conflict of Interest and Confidentiality Agreement
 Click Save.
Step 1:
 Click Add Ratings under the name of the classroom you observed
 Select the Category and Form from drop down
 Verify Teacher Name or enter new Teacher information, if necessary
 Complete the Rating Form for each tool that was used.
 Click SaveUnder Add Rating, complete Standard 3 form for each classroom that has been rated
 Information that was submitted by the Facility will display at the top of the Standard 3 form
 Enter Rater Observation information, then Save
 For each classroom that is observed, complete an ERS form, CLASS form, and Standard 3 form. You will
not be able to submit the forms to the hub until all three forms have been completed.
Step 2-Site Evaluation:
 Select Add Site Evaluation
 Complete the form based on your entire experience with the program. You do not need to fill out a
different evaluation for each day that you observed the program.
 Click Save
Step 3-Summary Score:
 Select Review Summary Score & Submit Rating
 Review the data displayed on the QRIS Star Points Calculator
 Select Submit
The rater will no longer be able to view the program’s rating information. If the rater can still see information
for this program, the scores have not been submitted properly and the rater will need to submit again. This
information will be submitted to the “Hub”, and the local coordinator will be notified when a Star Rating has
been assigned.
Actual score sheets should never be shared with participating programs. In some cases a local coalition may
request access to raw scores for the purpose of reporting progress to a funder. Any requests for access to
score sheets or raw scores should be coordinated through the Hub. Raters should keep all score sheets on file
for five years or until the next rating occurs (whichever occurs sooner) and be prepared to submit them to the
Hub upon request. If a rater selects not to continue to participate in the VSQI, the rater should notify the Hub
immediately and submit all score sheets to the Hub.
7|Page
THE SUMMARY REPORT
Raters should write the summary report as soon as possible once an observation visit is completed and prior
to conducting another observation at another location. Raters should submit written summary within 10 days
of the last observation date to their local coordinator.
Reports should not include personal opinions. Technical writing should be clear and concise. “I observed”
should not be used throughout the report as this assists the rater in using inappropriate or biased comments.
The summary report is a record of what was observed which is very different from how the rater felt about
the observation. Raters should indicate in the report when information was obtained during an interview.
The names of the classroom teachers should not be used (except in the Program Information section); “Lead
Teacher” and “Assistant Teacher” are appropriate. Refer to the children as children, not students, when
writing the summary report. Children’s names should not be used in the summary report. Be detailed but not
redundant. If multiple classrooms are observed, refer to specific classrooms as “Classroom 1” or “Classroom
2”.
Proofread the report for content, grammar, spelling and syntax. It is recommended that raters familiarize
themselves with common grammatical errors. George Mason University Department of English provides a
short writing guide for correcting common mistakes:
(http://classweb.gmu.edu/WAC/EnglishGuide/Critical/grammar.html).
Remember this is a document that will be reviewed by many people. Be specific and give examples when
needed to paint a detailed picture of the observation.
Raters should not complete additional program observations until they have finished writing the summary
report for the previous program observed. This will enable raters to write a clear report without confusing
observations from multiple programs.
Submitting the report:
After writing the summary report and reviewing it for content, grammar, spelling and syntax, raters should
electronically submit it to their local coordinator within 10 days of the observation.
Discussing the report with the Star Mentor:
The rater, director, and local coordinator will receive an official copy of the summary report from the Hub. The
local coordinator will share the summary report with the mentor assigned to the program. The mentor may
contact the rater to schedule a meeting to review the content of the report. This meeting can be either in
person or via phone. The purpose of this meeting is to discuss any questions in regards to the summary report
that will assist in the development of the program’s Quality Improvement Plan (QIP).
INTER-RATER RELIABLITY PROCESS
Raters must maintain inter-rater reliability. Inter-rater reliability is defined as two Raters scoring within one
point for at least 85% of the ERS observation, and for 80% of the CLASS observation. The inter-rater reliability
“buddy visit” will take place every seven (7) centers and/or family child care homes or every six (6) months,
whichever comes first. If the rater hasn’t scheduled the required buddy visit at this time, they will be marked
8|Page
as Inactive until they complete the process, and will not be able to rate any programs. Raters (Candidates) of
the classroom-based programs must maintain reliability in CLASS, Toddler CLASS, ECERS and ITERS. Raters of
family child care homes must maintain reliability in FCCERS. Reliability for the ERS tools will be considered
collectively, regardless of which version (ECERS. ITERS, or FCCERS) is used most frequently by the rater or
during the inter-rater reliability “buddy visit.” Reliability in CLASS and Toddler CLASS will be considered
separately. Inter-rater reliability “buddy visits” do not preclude the requirement to annually maintain
reliability through the author’s video reliability procedures or the need to attend mandatory “refresher
training” as determined by the Hub.
In the event that ratings have not been conducted for a significant period of time (more than six (6) months
and Raters have not had the opportunity to maintain reliability, the Hub may determine that a “refresher
training” will signal a new cycle of consistency for a group of raters.
Scheduling:
Inter-rater reliability “buddy” visits will be done in pairs. One Rater (Candidate) will be seeking reliability and
one Rater will be designated as the Principal Rater. The Principal Rater need not be a Master Star Quality
Rater. However, the Principal Rater must be currently reliable and hired by a local coalition to conduct the
assigned rating. Every effort should be made to vary the individuals paired in inter-rater reliability “buddy
visits”, so that Raters are not conducting inter-rater reliability “buddy visits” with the same individuals
repeatedly. In the event that there are no opportunities to conduct reliability visits during official VSQI ratings’
visit, an alternate option is to visit a center who has volunteered their site for this purpose. The same policies
and procedures, including conflict of interest, apply to these visits.
The Rater (Candidate) seeking reliability will contact a currently reliable Rater to serve as a Principal Rater and
arrange to join the Principal Rater on a rating’s visit. A list of current Raters will be provided, but the Rater
(Candidate) seeking reliability is responsible for arranging a visit with an active, reliable Rater. If the Rater
(Candidate) seeking consistency is unable to arrange the inter-rater reliability “buddy visit”, they will contact
the local coordinator for assistance. Raters (Candidates) seeking reliability must notify the local coordinator
and VECF/VSQI when they have scheduled their inter-rater consistency visit. They should submit their intent
to QRIS@vecf.org. The director of the center, or the provider at the family child care home being rated
should also be notified by the local coordinator that an additional person will be joining the Rater on one day
of the rating visit.
Inter-rater Buddy Visit:
The Rater (Candidate) seeking reliability and the Principal Rater will visit one classroom/family child care home
together and score in the same setting for the same duration of time (for CLASS consistency in classroom
settings, the same twenty (20) minute segments will be used.) Interviews with the Directors and staff must be
done by the Principal Rater, in the presence of the Rater (Candidate), for accurate scoring. Each individual will
complete score sheets which they will submit individually to VECF/VSQI (by either mail, email (QRIS@vecf.org)
without discussing scores with one another. It is imperative that raters do not discuss the rating visit with one
another. It is not necessary to send original score sheets to the Hub: a scan, a fax or a copy is acceptable, but
must contain the full record of scoring. Both individuals should retain a copy of the score sheets in their
records. All score sheets submitted to the Hub remain confidential. The Hub will evaluate the scoring and
9|Page
report back to the Rater (Candidate) seeking reliability and specific members of the Hub. OECD/VDSS then
places the reliable Rater on their Rater’s listing for distribution.
Compensation:
The rater seeking consistency/reliability will participate in the inter-rater reliability visit pro-bono. They will
submit CLASS and ERS score sheets (FCCERS only in family child care homes), but do not write a summary
report. The Principal Rater, who is hired by the local coalition to conduct the rating, will be paid for the rating
(provided that it is an official VSQI ratings’ visit), and will be responsible for writing the summary report. The
Principal Rater’s scores will be the scores that are entered into the website and used in calculating the
program’s/home’s Star Rating.
Consequences if reliability/consistency is not met:
If the Rater (Candidate) seeking reliability is not successful at reaching reliability with the Principal Rater,
feedback on items/areas of inconsistency will be provided to the Rater (Candidate) for review and self-study.
Following this feedback, the rater must schedule another inter-rater reliability “buddy visit.” For this followup visit, the Principal Rater must be a Master Star Quality Rater, unless there are so few MSQR’s in the region
that an exception is granted by the Hub. In order to serve as the Principal Rater, the MSQR must be currently
reliable and actively conducting official VSQI ratings. The same procedures are to be followed as they were in
the first inter-rater reliability “buddy visit.”
Once the “Hub” has reviewed the second inter-rater reliability “buddy visit”, the Rater (Candidate) will be
notified of his/her inter-rater reliability results, along with specific persons at the Hub. Should the Rater
(Candidate) still be unreliable, an individual decision will be reached by the Hub in consultation with that
Rater’s local coordinator. Rater (Candidate) will not be able to conduct ratings until they have successfully
met reliability on the assessment tools.
FREQUENTLY ASKED QUESTIONS
What months will the observation take place?
All programs are on a revolving two-year cycle from the time they receive their official VSQI rating. Exceptions:
if a star rated program changes physical locations VECF should be notified and the rating will re-assessed.
Some localities have the funds available to do annual “check up” visits in the year programs are not receiving
official VSQI ratings.
What do you do if the director is not there?
Inquire to find out who is in charge in the absence of the director. You will need to ask for all the necessary
information required of the director (i.e. randomly selecting a classroom, providing a daily schedule for the
classroom to be observed and being available for the interview questions at the end of the observation) If you
are able to conduct the observation, but the person who is in charge cannot answer the questions, you may
then email or phone the director the following day. If there is not someone left in charge, it will be difficult to
continue with an observation. At this point, you should contact your local coordinator as soon as possible.
What if I cannot finish my observation in one day?
By careful pacing of your observation, this should not be a problem. If you absolutely cannot finish because
the program only meets for 3 hours a day, then you will need to go back for a second day to complete your
10 | P a g e
observation. The second day should also fall within the 3-week window for the observation. If this is not
possible, contact your local coordinator to advise you.
What if I get sick during the observation?
You must stop the observation and notify your local coordinator as soon as possible. You would need to
schedule a new 3-week window (if it has expired). You would begin a new observation upon your return to the
program.
What is a “non-typical” day?
This can occur when a field trip has been unexpectedly planned or the daily hours of the program have been
reduced due to an early dismissal. This can also occur when the children are sent outside so that ratios can be
met. You want to observe a “typical day” and none of the above examples would allow you to do that. Check
to see if there is another classroom you can observe that day. If not, explain this to the director and inform
your local coordinator.
How many children must be present to complete an assessment?
At least 2/3 of the children enrolled in a classroom should be present on the day of the observation for you to
experience a “typical day”. At this time, you may ask to observe another classroom of the same aged children
or return on another day. You would need to notify your local coordinator of this change.
Who do I contact if I have a question about any part of the rating process?
You may contact the Master Star Quality Rater who is supporting you. If you are not provided with this
information during rater training, you should contact the Hub.
Who do I contact if I have a question about ERS,FCCERS or CLASS?
You may contact the Master Star Quality Rater who is supporting you.
What if the program I have been assigned to rate is one that I personally feel is a competing
program of mine?
This would be considered a “conflict of interest”. You should notify your local coordinator and refrain from
rating that program.
What if the program offers me a gift (like chocolate or hand lotion)?
All raters should refrain from accepting any gifts.
What if the director asks how they did on the observation?
Raters are observers only. Explain that the observation is only a part of the four Standards in the VSQI. Tell
them they will receive a comprehensive report which their mentor will aide them in reviewing.
What if the program that I rated contacts me to provide technical assistance?
Refer the program to your local coordinator or assigned mentor.
Revised May 2012
11 | P a g e