Tulare County
Workforce Investment Board, Inc.
MONITORING
PROCEDURES MANUAL
WIB Administrative Offices
4025 West Noble Avenue, Suite A
Visalia, California 93277
TULARE COUNTY WORKFORCE INVESTMENT BOARD
MONITORING PROCEDURES MANUAL
1
TULARE COUNTY WORKFORCE INVESTMENT BOARD, INC.
MONITORING PROCEDURES MANUAL
TABLE OF CONTENTS
I.
PURPOSE, REFERENCES, POLICY ..........................................................................3
II.
PROCEDURES AND METHODOLOGIES .................................................................7
III.
ADMINISTRATION - Exhibit "A".............................................................................19
IV.
INTAKE - Exhibit "B" ................................................................................................22
V.
ASSESSMENT - Exhibit "C" ......................................................................................27
VI.
FINANCIAL ACCOUNTING
AND INTERNAL CONTROL OUTLINE - Exhibit "D" ...........................................37
VII.
YOUTH PROGRAMS - Exhibit "E" ...........................................................................69
VIII.
CLASSROOM TRAINING - Exhibit "F" ...................................................................85
IX.
ON-THE-JOB TRAINING - Exhibit "G"....................................................................96
X.
PRE-AWARD SURVEY - Exhibit "H" ....................................................................117
XI.
ONE-STOP SYSTEM MONITORING GUIDE- Exhibit "I" ...................................120
XII.
EQUAL EMPLOYMENT OPPORTUNITY – Exhibit “J”.......................................128
XIII.
SERVICE PROVIDER MONITORING – Exhibit “K” ............................................131
XIV. MONITORING RESOURCES ..................................................................................132
2
*TULARE COUNTY WORKFORCE INVESTMENT BOARD, INC.
MONITORING PROCEDURES
PURPOSE:
This manual contains Tulare County Workforce Investment Board, (WIB) policy and standards for
monitoring and oversight of contracted operations and other programs operated by the Board. WIB
Policy WIBD-08-13 (WIB Monitoring Standards) are incorporated by reference. These procedures
are developed in response to State and Federal guidelines concerning WIB oversight, evaluation and
monitoring.
REFERENCES:
 WIA Sections 117(d)(4), 184(a)(2)(A), 184(a)(3)(A), and 184(a)(5)(A)
 Title 20 Code of Federal Regulations (CFR), Sections 627.475(e) and (f), 667.400(c)(1),
667.410(a), and 667.410(f)(6)
 Title 29 CFR Part 97
 Title 29 CFR Part 95
 Office of Management and Budget Circulars (OMB) A1-21, Cost Principles for Educational
Institutions
 Office of Management and Budget Circulars (OMB) A-87, Cost Principles for State, Local and
Indian Tribal Governments
 Office of Management and Budget Circulars A-122, Cost Principles for Non-Profit
Organizations
 Office of Management and Budget Circulars (OMB) A-133, Audits of States, Local
Governments, and Non-Profit Organizations
 WIA Directive WIAD00-2, Subject: Procurement (August 24,2000)
 WIA Directive WIAD07-6, Subject: Nondiscrimination and Equal Opportunity Procedures
(January 16, 2008)
 WIA Directive WIBD00-7, Subject: Standards for Oversight and Instructions for Substate
Monitoring (April 10, 2001)
 Tulare County Workforce Investment Board, Inc. WIBD-05-14, Subject: Procurement (October
17, 2005)
 Tulare County Workforce Investment Board, Inc., WIBD-08-15, Subject: Nondiscrimination
and Equal Opportunity Procedures (December 17, 2001)
 Tulare County Workforce Investment Board, Inc., - Contracts with Subrecipients (Service
Providers)
 California Unemployment Insurance code (CUIC) 15032 (d)
 SEC. 103. (a) It shall be the responsibility of the Workforce Investment Board to provide
policy guidance for, and exercise oversight with respect to, activities under the job training
plan for its service delivery area in partnership with the unit or units of general local
government within its service delivery area (SDA).
(b)(2) The Board is authorized to provide oversight of the programs conducted under the job
training plan in accordance with procedures established by the Board.
In order to carry out this paragraph, the Board shall have access to such information
concerning the operations of such programs as is necessary.
* Update 7-8-09
3
 SEC. 141 (i) Each administrative entity shall be responsible for the allocation of funds and
the eligibility of those enrolled in its programs and shall have responsibility to take action
against its subcontractors, subgrantees, and other recipients to eliminate abuses in the
programs they are carrying out, and to prevent any misuse of funds by such subcontractor,
subgrantees, and other recipients. Administrative entities may delegate the responsibility for
determination of eligibility under reasonable safeguards, including provisions for
reimbursement of cost incurred because of erroneous determinations made with insufficient
care, if such an arrangement is included in an approved job training plan.
(e)(1) The WIB pursuant to standards established by the Governor, shall establish
specific policies for monitoring and oversight of SDA performance which shall be
described in the job training plan.
(2) The WIB may exercise independent oversight over activities under the job training
plan which shall not be circumscribed by agreements with the appropriate chief elected
officials(s) of the SDA.
(f) The WIB and chief elected official(s) may conduct such oversight as they, individually or
jointly, deem necessary or delegate oversight responsibilities to an appropriate entity
pursuant to their mutual agreement.
 20 CFR 628.410 (b)(3)
(b) Responsibilities of the WIB Pursuant to section 103 of the Act, the WIB shall:
(3) As specified in Subpart D of Part 627 of this Chapter, exercise independent oversight
over programs and activities under the job training plan, which oversight shall not be
circumscribed by agreements with the appropriate chief elected official(s) of the SDA.
 20 CFR Part 34 -Implementation of the nondiscrimination and equal opportunity
requirements of the Job Training Partnership Act of 1982 as Amended.
POLICY:
It is the policy of the Tulare County Workforce Investment Board, as the Administrative entity, to
provide monitoring and oversight of its contracted Services Providers and other programs operated
by or on behalf of the WIB. Monitoring and oversight will be designed to comply with State and
Federal standards.
1. Conduct on-site visits of all service providers or training sites twice each program year. If the
duration of the program or activity monitored coincides with the program year or is longer than
the program year, the first monitoring visit will occur in the first six months of the program year
and the second during the second six months. If the activity is shorter than the program year but
longer than six months, monitoring will be scheduled once during the first half of the activity
and once during the second half of the activity. If an activity being monitored is less than six
months, only one on-site visit is required. At least one of the on-site monitoring visits will
include a complete review of the agency’s fiscal operations as it relates to the contract with the
WIB and a review of the Equal Employment and Training Opportunity (EEO) program. NOTE:
The self-evaluation checklist in the WIB EEO Policy will be utilized to review compliance with
the Americans with Disability Act (ADA).
4
2. Monitoring will identify whether outcomes are within the confines of the Service Provider’s
contract. Monitoring will ensure that programs are operated in compliance with applicable
Federal and State regulations, and the contract between the Provider and WIB.
3. Monitoring will ensure fiscal accountability. Such monitoring shall verify that WIA funds are
being utilized for allowable and budgeted activities, and that applicable fiscal records are
maintained and that such provides an adequate audit trail.
4. Monitoring shall also ensure that Service Providers and the LWIA remain in compliance with
program limitations provided by the WIA. Review of the following program limitations, and/or
other areas of concern as identified by the LWIA/WIB, shall be included in the scope of the
monitoring process.
a. Non-duplication of Services
141 (b)
-Services must add to available services
107 (b), 141 (h)
-Prohibits duplication of services
b. Selection of Service Providers
107 (a)(b)(c)(d)(e)
-c. Consultation and Coordination
107 (d)
-WIB must approve skills levels
143 (c)(2)
-Labor unions must be consulted
d. Local Conditions, Prohibition of Displacement
141 (a)
-Most in need and equitable distribution
141 (d)(1)
-Training only for occupations in demand
203 (b)(4)
-Ratio of public to private sector OJT
143 (b)(1)
-No displacement of current workers
143 (b)(2)
-No union contract impairment
143 (b)(3)
-No displacement of laid off workers
143 (b)(4)
-No infringement of promotions
e. Prohibitions of Activities
141 (1)
-143 (c)(1)
-167 (a)(3)
-141 (c)(1)(2)(3)(4) --
Prohibits political activity
Prohibits union organizing
Prohibits sectarian activities
Prohibits relocation of establishments that cause layoffs
f. Training Limitations and Fee Prohibitions
141 (g)(1)(2)(3)
-Limits OJT reimbursement length of training
141 (i)
-WIB responsibility for intake & eligibility determination
141 (j)
-Prohibits placement and referral fees
141 (k)
-Limits subsidized private employment to youth
5
If during the monitoring process the monitor suspects that he/she has discovered possible
fraud/abuse, monitoring of the Service Provider will be terminated and findings reported
immediately to his/her immediate supervisor and the WIB Administrator. The WIB Administrator
will then make the necessary contact with the grant recipient and the Chairman of the Workforce
Investment Board. The WIB Administrator shall make a determination regarding submission of an
Incident Report to the State EDD.
Adam Peck
Executive Director
Updated July 2009
6
PROCEDURES & METHODOLOGIES
WIA
MONITORING UNIT
The procedures contained in this document are intended as a
general guide to the operation of the Tulare County
Workforce Investment Department’s Monitoring Unit.
Special circumstances may require a variance from these
procedures and will be guided with direction from the
management of the Department and/or WIB Administration.
Subsequent releases shall be numbered and dated.
Release7, dated: July 8, 2009
E:\Monitoring\SOP Monitoring.doc
7
GENERAL INFORMATION
All monitoring shall be conducted under the guidance of the Tulare County Workforce Investment
Board, Inc. Executive Director and/or Deputy Director. Unless otherwise directed, all
correspondence generated prior to, during and resulting from monitoring reviews will be sent under
the signature of the Administrator or his/her designee.
The Tulare County Workforce Investment Department Organizational Chart provides guidance
regarding the lines of responsibility to be followed in the ordinary execution of the duties of the
TCWID Monitoring Unit. This does not limit or restrict direct communication between staff in
the normal work process.
Introduction: This document contains operating procedures for the Tulare County Workforce
Investment Department (TCWID) Monitoring Unit; and a thorough description of the
elements of monitoring and its role in the regular conduct of business in the
administration of Workforce Investment Act (WIA) funds in the County of Tulare.
I.
The Tulare County Workforce Investment Board, Inc. has set requirements for the frequency
and conditions for monitoring as follows:
A. Monitoring of all subrecipients will occur at least once each program year.
B. The onsite review of each subrecipient is both fiscal and programmatic; consistent with
the requirements stated in Title 20 CFR Sections 667.400(c)(1) and 667.410(a). WIBD00-14.
C. The monitoring of subrecipients will follow a standardized review methodology that will
result in written reports which record findings, any needed corrective actions, and due
dates for the accomplishment of corrective actions. This methodology shall include
requirements that on-site visits of all contractors providing training services shall occur
twice each program year for activities longer than six months in duration, and at least
once regarding activities lasting six months or less. A Monitoring Schedule shall be
published every six months and shall outline the review to be conducted (compliance,
fiscal and/or program).
D. Systematic follow-up will be conducted to ensure that necessary corrective action has
been taken. WIA Section 184(a)(5)(A)
E. All written reports and other documentation pertaining to monitoring and other oversight
activities will be made available for review by federal and State officials. Title 29 CFR
Sections 95.53(e) and 97.42(e)
F. Reports and other records of monitoring activities will be retained for three years from
the date of submission of the final expenditure reports regarding the funding sources
monitored. If any litigation, claim, audit or other action involving the records has been
started before the expiration of the three-year period, the records must be retained until
completion and resolution of all such actions or until the end of the three-year period,
whichever is later. Title 29 CFR Sections 95.53(b) and 97.42(b)(c)
8
G. The Tulare County Workforce Investment Board, Inc. Monitoring Procedures Guide
contains the procedures, tools and methodologies by which monitoring will be
conducted.
II.
Types of Monitoring
A. Compliance Monitoring
The purpose of compliance monitoring is to ensure that the requirement of a specified
agreement, contract, or document is met. The WIB staff is responsible for monitoring all
program activities and services, and program administration and management practices,
supported with funds under the Act, to ensure legal, fiscal, administrative, and
programmatic compliance. Other items shall include a review of program administration
and management practices, adherence to Americans with Disabilities Act, and Equal
Employment Opportunities Act.
B. Plan vs. Actual Monitoring (Performance)
The purpose of plan versus actual monitoring is to provide the Service Provider, WIB,
and LWIA staff with current information on the extent to which programs and program
components are achieving established goals. The results of plan-versus-actual
monitoring analyses shall be to assess progress toward goals and objectives, and to
identify existing or emerging problems. This method of monitoring may be conducted as
scheduled or as often as deemed necessary utilizing the Management Information System
to ensure Service Provider compliance with agreements.
C. Program Quality Review
The purpose of program quality review shall be to review the excellence and
effectiveness of services to participants. Such monitoring shall consist of “flagging”
problems identified through desk review or compliance review, and survey reviews;
determination of corrective action, and facilitation of technical assistance through WID
program analysts will ensure that corrective action occurs. Primary concerns are “what
is happening” and “why it is happening”.
III.
Monitoring – Areas for Emphasis
A. Participant Eligibility Process
Ensure that the Service Provider’s eligibility systems are in compliance as required by
WIA Eligibility Determination and Verification Directives currently in effect. Ensure
that the participant has received core services which are documented in the case file.
B. Participant Assessment and Individual Employment Plan (IEP)
1. Assessment of all participants.
a. Development of Assessment Report
i. Long Term, Short Term Goals
ii. Results of Assessment
iii. Assessment Recommendations
b. Development of Individual Employment Plan (IEP)
c. Periodic review and update of Individual Employment Plan (IEP)
2. Review of participant progress.
3. Review of quality and success of training and customer satisfaction
9
C. EEO/Grievance Procedures
1. Grievance Procedures and Equal Employment Opportunity (EEO) practices.
D. Procurement Practices (WIB Property and Procurement Procedures)
a. Adherence to contract requirements and instruction for purchases.
b. Procedures for prevention of conflict of interest.
c. Political Activities Prohibited
d. Sectarian Activities Prohibited
e. Property Management
E. Financial Management – Fiscal Monitoring
1. Probable Major Findings & Conditions
a. Internal Control finding/conditions such as but not limited to:
i. No Separation of Duties
ii. Lack of approval/authorization
iii. Lack of written policies procedures
iv. No reconciliation of cash balances
v. Cash payments
vi. Inconsistencies between job descriptions and actual duties
vii. No audit trail between accounting records and financial reports
viii.Insufficient documentation/records
ix. No signatures on time sheets
x. No inventory of equipment
b. Financial Management Systems
i. No supporting documentation
ii. Lack of procedures
iii. Idle property – not in use
iv. Inequitable allocation of costs to WIA
v. Improper allocation of expenditures
vi. Overstated expenditures
vii. Expenditures not properly classified
viii.Non-competitive bids
ix. Property used by non-WIA participants, no use fee charged
x. Bank cash balance not in agreement with books
xi. No written cost allocation plan
IV. Reports generated as a result of monitoring shall not contain subjective or qualitative
statements.
10
MONITORING
PROCEDURES & METHODOLOGIES
WIA
I.
A Desk Review will be conducted prior to any on-site visit to a provider. Desk Reviews may
also be scheduled periodically to review providers determined to be “high-risk” or on
probationary status. The review shall consist of, but shall not be limited, to:
A. Review contracts, files, documents, and related correspondence;
B. Review applicable Federal and State Regulations;
C. Review Management Information Systems data, including current enrollment and
placement results;
D. Review Budget and Expenditure reports;
E. Review prior audits, if available;
F. Review prior monitoring reports.
II.
Initiating the on-site monitoring visit.
A. Prior to the monitoring visit a letter, signed by the WIB Administrator or his/her
designee shall be sent to the agency to be reviewed. This letter will be sent to the
Director, Executive Director, or Administrator of the agency.
B. The letter will be sent no later than two (2) weeks prior to the on-site visit.
C. The letter will contain:
1. A list of items the service provider will want to have available to the monitor for
review.
2. A list of enrolled participants as reflected by the TCWID/MIS system.
3. A list of participants exited from programs and data regarding their follow-up status
as reflected by the TCWID/MIS system.
4. Copies of any surveys and/or instruments to be completed by the service provider
prior to the date of monitoring.
D. The letter shall identify the contact person for the Monitoring Unit.
III.
Field or on-site review shall consist of, but shall not be limited, to:
A. Entrance Conference - A description of the scope and purpose of the monitoring shall be
provided. This description shall include programs to be reviewed and a determination
that information required to complete monitoring visit is available at the site.
(Note: The Service Provider shall receive a current listing of all participants in the JTA’s
data system prior to the review.)
1. At this time the monitor may request information regarding the appropriate parties
with whom interviews will be conducted:
-- Staff involved with the WIA contract activity;
-- Program participants at training sites;
2. Scheduling for the exit interview may also be discussed if the provider desires an exit
interview.
B. The on-site review will generally consist of:
1. Review fiscal and program records for compliance with WIA contract and generally
accepted accounting principles.
2. Review systems for controlling program administration, including those for
determination of participant eligibility where applicable. (Youth programs of Service
Providers and WID Adult and Dislocated Worker eligibility services.)
11
3. Review pay records (time cards/sheets) and attendance reports to ensure controls are
established for preventing unauthorized payments to participants in training.
4. Interview participants to determine training meets guidelines approved by the WIB
and is consistent with the career goals of the individual.
5. Review (OJT, internship, work experience, etc.) contracts/agreements between
training site (employer) and the Service Provider to ensure that training is being
provided and payments are appropriate. Contracts shall be reviewed for dates,
authorized signatures, amounts paid, and supporting documents to justify payment.
6. Review staffing levels, operating procedures, and contracts to determine Service
Provider capability to fulfill contract obligations.
7. Review Service Provider records on expenditures of WIA contract funds and
supporting documents.
8. Review affirmative action plans as applicable.
C. A sample of the following records will be reviewed either in a Desk Review or on-site at
the provider’s place of business. Copies will be maintained in the monitoring file where
appropriate to document the sampling process utilized and/or any concern surfaced
during the review.
1. Administrative and EEO records
(a) Personnel policies
(b) Affirmative Action Plan, (most current)
(c) Job Descriptions for Staff
a. EEO Poster, display and location will be noted.
2. Contract files
(a) Contract
(b) Modification(s)
(c) Contract Exhibit(s)
3. Participant Files - See Sample Size and Random Sample Selection, page 9.
(a) Registration Form and eligibility support documentation
(b) Client Action/Enrollment Forms
(c) Civil Rights Form, signed by participant and dated
(d) Job Performance Evaluation/Assessment
(e) Individual Service Strategy (ISS) or Individual Employment Plan (IEP)
(f) Termination/ Placement documents
(g) Documentation of competencies, certifications (youth and adults)
(h) Case Notes
4. Fiscal Records
(a) For cash basis accounting system
-- Cash Receipts Journal
-- Cash Disbursement Journal
-- Payroll Journal
-- Administrative Cost Allocation Plan
-- General Ledger
-- Bank Reconciliations
(b) For accrual basis accounting
-- Accounts Payable Journal
12
-- Purchase Journal
-- Cash Receipts Journal
-- Cash Disbursement Journal
-- Payroll Journal
-- Administrative Cost Allocation Plan
-- General Ledger
-- Bank Reconciliations
(c) Property Control Records
(d) Insurance Policies
-- Worker's Compensation, General Liability, Fidelity Bond
(e) Source Documents, such as:
-- Time Sheets - staff and participants
-- Vendor invoices
-- Travel reimbursements
-- Petty cash
-- Canceled checks
-- Advances (Review of cash-on-hand.)
5. Property Records Review
(a) WIB property is tagged and utilized for contract purposes.
(b) Buildings and facilities paid for by contract funds are being used for project
proposes and are adequate/inadequate.
(c) Major property improvements or alterations in process or plan have been
approved by WIB Administration.
(d) Verification of prior approval of (a) and (c).
(e) Conduct a random check to determine if proper care and attention, is being given
to the maintenance, repair, and protection of WIB property.
D. The Exit Interview, if requested or appropriate, will be held with appropriate
administrative staff of the Service Provider when the monitoring visit is completed.
Problem areas will be discussed in general terms. At a minimum, two WID staff shall
conduct the exit interview.
E. Monitoring Report - A monitoring report shall be written as a result of the desk review,
field review, participant interviews, and/or other related activities. The report will state
the objectives, scope and methodology of the monitoring; and will clearly state
observations, requirements, findings, concerns, and whether corrective action is required.
Findings will contain a statement of criteria (Regulation, Directive, or Contract Clause),
the condition found, the cause of the problem, and the effect or consequence that will
result if corrective action is not taken. The report will be circulated to appropriate
program analysts and WID management for review, comment, and recommendations.
This shall be a informal e-mail process which will be documented and maintained on file
with the report. The report which is circulated, shall be in the format used as the letter of
transmittal.
A letter transmitting the results of the monitoring report shall be developed for the
review and action by WIB Administration and shall incorporate the report content. It
will initially be transmitted in draft form to the respective Service Provider and if
corrective action is required, a response from the Service Provider will be secured. A
13
final report, titled “Final Monitoring Report” will be issued and shall include the
response of the respective Service Provider and the conclusions of the WIB
Administration.
Technical Assistance will be provided by the appropriate WID Program Analyst and
shall be conducted at the direction of WID/WIB Administration.
Resolution and documentation of corrective action shall be sought until all issues are
concluded. Where appropriate findings shall remain open pending follow-up on
successive monitoring visits, (See Procedure for Closing a Monitoring Report.), the
report will so state.
F. If during the monitoring process the monitor suspects that he/she has discovered possible
fraud/abuse, monitoring of the Service Provider will be terminated and findings reported
immediately to his/her immediate supervisor and the WIB Administrator. The WIB
Administrator will then make the necessary contact with the grant recipient and the
Chairman of the Workforce Investment Board. The WIB Administrator shall make a
determination regarding submission of an Incident Report to the State EDD.
14
PROCEDURE FOR CLOSING A MONITORING REPORT
1. Upon completion of monitoring fieldwork, a monitoring report will be written and submitted to
WID Administration for circulation and comment by WID Analysts. The report/draft
transmittal letter containing findings, or the lack thereof, will be issued to the Service Provider
within 30 days of the exit conference or the date monitoring concluded, whichever is later. A
copy of the report, routing documentation and comments, and correspondence generated from
the report/letter will be filed in the master contract file at the WIB Administration Office.
2. When corrective action is recommended to resolve a finding or address a concern: the Draft
Report letter will be issued to the service provider and provide twenty (20) working days from
issuance for response a return receipt request will activate this date. A copy of the letter will be
distributed to the appropriate program analysts for their records.
3. If the service provider response is deemed sufficient after review by Administration or corrective
action has been demonstrated, a closure letter (Final Monitoring Report) will be initiated by WIB
Administration. A response is due thirty (30) days from the date of receipt. Where appropriate
findings shall remain open pending follow-up by the monitoring unit with a follow-up visit or in
subsequent monitoring visits.
4. Should the service provider response be inadequate, WIB Administration will direct the
corrective action in writing. The service provider will have five (5) working days to file a
written appeal, provide further information, or request an extension.
5. If corrective action was implemented within 30 days of the closure letter, validation of corrective
action taken will be made and documented by the monitoring unit.
6. All observations, findings, concerns, and resolutions will be recorded in the Monitoring Log
maintained by the Monitoring Unit. The log will record the name of the provider, contract
number, date of the report, note whether the issue is Program or Fiscal, whether the matter is a
finding or a concern, if the matter is closed, open or pending future review, and a brief narrative
of the Item. A copy of the log will be placed on file with the Tulare County Workforce
Investment Board Inc. general files.
7. Files and filing responsibilities:
A. Monitoring Files will be maintained with working papers developed as a result of
monitoring. The file shall contain a copy of documents pertinent to the review and copies of
any reports or correspondence generated during or as a result of the monitoring review.
B. Tulare County WIB, General Files maintained by the monitoring unit.
1. The following files will be maintained in the General Files of the agency. The
Monitoring Unit is responsible to provide clerical staff with appropriate documents to
keep these files current.
a. A Monitoring Schedule File, which shall be developed at the beginning of each
Program Year. This file shall be updated on a quarterly basis to show the actual date
of the monitoring visit, date the draft report was issued, date the final report was
issued, and an indication of whether or not there was a finding and its status.
b. The Monitoring Log File. This file contains a listing of findings and concerns and a
statement of resolution.
15
c. The Audit Log File. This log is required by State directive (Audit Resolution) and
must contain a copy of service provider audits, listings of audit findings, auditor
opinion, and status of the resolution of any audit findings by the service provider.
2. The Monitoring Unit will ensure that a copy of any draft reports, service provider
responses, final reports, and documents related to the circulation of such report shall be
provided to WID clerical staff for inclusion in the Master Contract file for that Service
Provider.
C. The Monitoring Unit will ensure that a copy of any reports resulting from monitoring
reviews of WID activities will be provided to WID clerical staff for inclusion in an WID InHouse Monitoring file in the General Files of the Tulare County Workforce Investment
Board, Inc.
16
OTHER RESPONSIBILITIES
Monitoring Unit
I.
Role in preparation for visits from the State EDD Compliance Review Division.
A. The Monitoring Unit shall from time-to-time develop a “library” of current local
Directive to respond to monitoring visits scheduled by the State EDD Compliance
Review Division (CRD).
B. Response to sections of the Guides distributed by the CRD relating to monitoring will be
completed by the person assigned from the Monitoring Unit.
C. Copies of all CRD Monitoring Guides completed/compiled by the TCWIB staff will be
kept on file with the TCWIB General Files with support documentation.
II.
Role in risk assessment process.
A. The Monitoring Unit shall provide up to date copies of the Monitoring and Audit Logs to
Program Analysts responsible for negotiation of awards of funds after the procurement
process.
B. Risk assessment will guide the extent and frequency of the monitoring review of a
service provider.
III.
Any other roles as deemed appropriate by management and administration of the Tulare
County Workforce Investment Department and/or Tulare County Workforce Investment
Board, Inc.
PROCEDURE FOR CONTACTING WID ANALYSTS
FOR MONITORING PURPOSES
I.
Prior to site visits and desk reviews Monitors will contact the responsible Program
Coordinator to schedule an informal meeting. The monitor will inquire if the Program
Coordinator(s) have any concerns, or any request to focus on a particular subject area during
monitoring.
II.
An appointment will be scheduled with the appropriate Program Coordinator after a site visit
and prior to composition of the report to review findings, concerns, and/or observations
during the monitoring.
III.
All meetings will be set by e-mail appointment and a copy of the e-mail placed in the
appropriate monitoring file.
TIME SHEETS & TIME RECORDS
The Monitoring Unit will record their time as follows:
A. Time monitoring programmatic matters shall be recorded in the appropriate category,
e.g., core, intensive; and in the appropriate fund source, e.g., Adult, Dislocated Worker,
Youth or the appropriate fund source.
B. Time monitoring or reviewing TCWIB activity shall be recorded as Administrative.
C. Questions regarding the correct recording of time should be addressed to the
Monitoring Unit Supervisor or the Deputy Director Fiscal at the TCWIB.
17
EXHIBIT "A"
1. DESK REVIEW:
A. Review Individual Service Strategy (ISS) guidance
B. Eligibility Verification Determination System
C. Coordinate with MIS/WIB Intake Program Coordinator
D. MIS Reports (Client transaction information)
2. FIELD REVIEW:
A. Interview with Management staff--entrance conference
B. Review operational documents and files
18
GENERAL ADMINISTRATIVE
QUESTIONNAIRE
Purpose: To be used reviewing either Service Providers, One-Stop Operators or other contractors
whose services are formally procured by the TCWIB.
Service Provider:
Date:
Person Interviewed:
Title:
Monitor:
1. INTERNAL CONTROL/SELF-EVALUATION:
A. Do you have an Agency Policy or procedure Manual, including policy/guidelines, by which
you conduct self-evaluations or monitoring to validate your administrative, management, and
operational controls?
(yes
) (no
)
Comments:
B. Do you have sufficient staff to carry out the program activities as described in your contract.
(yes
) (no
) Note: Conduct a sample of staff qualifications and review related job
descriptions.
Comments:
C. Do you have a procedure for referral of clients to other services/training where appropriate,
and if so how? (yes ) (no ) (Are clients referred back to the LWIA for possible reevaluation when services are not provided by the Service Provider?)
Comments
D. Have you distributed to staff, or have accessible copies of:
Directives or information from Tulare County WIB
19
Current WIA regulation and laws
Personnel Policies
Grievance Procedures
Comments:
E. Are all of your staff members who are paid from, or work within, WIA programs familiar
with requirements of the contract and its exhibits? (yes
) (no
) How? (Conduct a
random sampling by questioning staff, the agency should provide written certification that
appropriate staff are familiar with the contract.)
Comments:
F. How do you insure that sectarian (religious activity) is avoided as required by the WIA?
(Ascertain if the staff is familiar with this provision.)
Comments:
G. How is staff informed to refrain from involvement in improper political activity? (The
agency should show documented proof that this information is routinely provided, especially
to new staff.)
Comment:
H. How do you insure that relatives by blood, adoption, or marriage of any executive or
employee of you agency does not receive favorable treatment for enrollment in services?
Comments:
20
I. When it is in the public interest for you as an agency to conduct business with a friend or
associate of an executive or employee of your organization, or area elected officials, or
Board members or staff of the Workforce Investment Board, do you retain a record of the
transaction? (yes ) (no )
Comments:
J. Do salaries and bonuses of executives for WIA programs exceed the latest limit under Public
Law 109-234?
Yes
No If YES, please identify the LWIA executive(s) and the
amount exceeded:
Comment:
CONCLUSION:
2. FRAUD AND ABUSE:
A. What procedure(s) do you follow to provide identification and reporting of Fraud and
Abuse?
B. Do you have available a copy of the WIB Fraud and Abuse Reporting Policy?
(yes
) (no )
C. When and how was information regarding Fraud and Abuse provided to your staff? Do
you have written verification
D. Does the Service Provider have adequate safeguards to prevent employees from engaging
in private business or contractor time or using contractor materials or equipment for
private purpose? (yes ) (no ) (n/a )
21
EXHIBIT "B"
INTAKE/ELIGIBILITY/ASSESSMENT
1. INTAKE STAFF INTERVIEW
A. How do you insure that target groups identified in the contract are being served at the levels
indicated? (n/a ) (This question is applicable to Service Providers that conduct their own
intake, refer to the contract for service levels.)
Comments:
B. What process is used by the service provider or WIB to purge out-dated client files? Are
client records maintained for at least three (3) years?
Comments:
C. How do service providers/WIB insure that your intake staff are familiar with current WIB
eligibility requirements? (Does the staff have adequate reference material readily available?)
Comments:
D. Are files easily accessible but safeguarded? (yes
) (no
)
Comments:
E. Aree intake staff familiar with participant selection criteria? Does the staff appear to be
knowledgeable of client eligibility, assessment, testing and referral as it relates to training
and other community services?
22
Was this confirmed in the file review?
(yes
) (no
)
F. When applicants are interviewed at intake, how are they informed about:
1. Rights & benefits
of participants
Verbal
Written
Other:
2. Grievance
procedures
Verbal
Written
Other:
3. Agency programs
& training
expectations
Verbal
Written
Other:
4. Tulare County
WIB Services
Verbal
Written
Other:
Was the material reviewed in an orientation? (yes
2.
) (no
) How?
ASSESSMENT
STAFF INTERVIEW GUIDE
INSTRUCTIONS:
The following questions may be directed to more than one of the contractor's staff since there are
a variety of subjects covered. Explain the purpose of the interview and quickly review the
subjects to be covered.
Interviewees' Names
Title
A. Describe briefly how your assessment and referral activities are organized? How do
applicants get into assessment?
The purpose of this question is to obtain a general orientation to the assessment and referral
component. What is the sequence of events or activities? Is there any screening, other than
23
for eligibility that occurs before participants reach assessment? Summarize the response
below.
B. Where do the assessment and referral activities take place? Where are participant records
kept?
Response to this question locates where data sources are and identifies logistic, of the
activity.
C. How many people have been served thus far? How have they been distributed in terms of
the referral options?
This question establishes how large or active the assessment component is and whether it is
achieving planned objectives. Is there a wide variance in where participants are being
referred or does it appear that most participants are being directed to the same activity? To
what extent is the timing of available slots determining choice of referrals?
D. What were the factors that determined the approach used to assess participants and make
referrals?
The purpose of this question is to determine how rational the reasoning is behind the choice
of assessment tools and procedures. Are these decisions driven by the selection criteria of
the referral options? What were the assumptions made about participants' general
capabilities and interests? How does the assessment process fit into the development of an
individual service strategy (ISS)?
E. List the assessment techniques used and indicate what percent of the participants served
received assessment.
24
The purpose of this question is to establish how often assessment tools and procedures are
applied and what accounts for any variation in what type of assessment is given participants.
Are assessment techniques varied to accommodate participants with different
characteristics? Assessment techniques include interviews, paper and pencil tests,
questionnaires, and any other process that is designed to obtain information from the
participant relative to employment planning.
Technique
%
Objective
Type of Participant
(Youth/Adults)
F. What kinds of qualifications must staff have to administer or interpret the results of
the assessment tools described above?
Some assessment instruments, particularly projective tests, require special training or a postgraduate degree in psychometrics. At a minimum, staff who are administering or
interpreting tests must be trained on the specifics of a particular instrument. What are the
education levels and specialties of the staff charged with administering and interpreting
tests? What kinds of training are they given? Keep in mind that administering tests is
generally easier than interpreting tests and requires less qualified staff.
G. How do you assure that any tests used are appropriate for participants?
Are written questionnaires (forms) or tests used without first checking to verify that the
participant is literate in English? What effort was made to investigate the validity and
reliability of the tests? What authoritative source is used to support the validity and
reliability of the tests for the culture represented by the target group?
H. How does the information collected through the assessment tools listed above relate to the
referral options and services available?
25
The purpose of this question is to establish that there is a direct link between the assessment
activities described thus far and information needed to make referral decisions. Is the
information generated by the assessment relevant to the kinds of decisions that must be
made? Make sure that each assessment product can be justified by a need for the
information.
I. CONCLUSIONS:
Are the assessments tailored to the individual differences of participants or does everyone
get the same assessment? Is the information generated by the assessment useful for
employment planning in that it is relevant to the referral options open to participants? Do
you generally feel that the assessment activities described above can be justified in terms of
the usefulness of the information for employment planning?
3. QUALITY OF INDIVIDUAL SERVICE STRATEGY:
A. Please demonstrate through written records or documents how it is determined that
participants have an interest or compatibility for the occupation and/ or training reflected in
their Individual Service Strategy Plan (ISS).
The purpose of this question is to establish that the assessment data supports the participants'
referral to a particular training or service strategy. Answering this question may require
some reference to several participant files. Look for indication in the documents or
interviewee's response that participants' interests and capabilities were considered in
developing the ISS. Is there a connection between assessment results, past work history, or
vocational interest in work related to the training occupation? Is the ISS specific and
reflective of individual differences or do they all read much the same?
B. What role does the participant play in developing the ISS? What feedback do participants
receive from assessment results?
What procedures and methods are used to help participants select training and employment
services? Is the participant active in the employment planning process? Are they given
sufficient information to make informed decisions about their job training? This would
26
require information about entrance criteria for various training options, information about job
requirements, salaries and benefits, etc. Are ISS's signed by the participant?
C. Who does the testing, counseling, ISS development, and referrals?
If different individuals are involved, how are their efforts coordinated? To what extent is a
counselor involved in employment planning?
4. OUTCOMES AND PARTICIPANT BENEFITS: (SERVICE
PROVIDER/CONTRACTOR ONLY)
A. How many participants have received, training, employment planning and referral services
thus far? What other services, other than assessment, have they received?
Briefly, summarize services received by participants.
B. Of those participants referred to training, education, or work experience, what percent
completed the activity successfully?
The Service Provider may have to conduct some type of follow-up to get this information.
The purpose of this question is to determine the attrition rate for Service Provider referrals.
Participants poorly matched to a training service frequently dropout or are unable to perform
at a level to be successful. Since the best evidence of a good referral system is the success of
the participants, this is an important question.
Percent:
% Comment:
CONCLUSIONS:
27
Does it appear that the Service Provider is in compliance with the terms of the contract?
What observations support or refute the above information?
28
5. PARTICIPANT INTERVIEW GUIDE
INSTRUCTIONS:
Participants (or applicants) should be interviewed individually unless they were assessed as a
group, an unlikely occurrence. Select participants who have been assessed and referred and are
sufficiently into their ISS to have some opinion about the relevance of the employment plan and
objectives to their needs, interests, and capabilities. You will need a copy of the interviewee's
ISS and, if possible, a summary of the test results, or other assessment data to compare. Note:
This interview may be accomplished during other points of your monitoring visits where
Participant contact is made.
INTERVIEWEES:
A. Recall your experience when you were interviewed and counseled regarding your training or
employment goals. What sort of tests did you take? How long did it take before you
completed the process and were referred to training?
Have the participant describe the activities and events leading up to his/her referral. How do
they compare to the procedures described by the contractor?
B. Do you think that the tests measured your abilities? If not, why not? Were you surprised at
the test results?
Look for examples of test anxiety that may depress test scores, statements suggesting that
participants' test scores are much different from grade level achieved or other evidence that
the tests are not good measures of ability. How clear is the participant about the results of
any tests taken?
C. Do you think that the staff (use name if possible) have a good understanding of your
background, interests, and capabilities?
Does the participant have confidence in the kinds of information obtained about his/her
29
vocational interests and aptitudes? Did the staff take enough time or did the participant feel
rushed through the process? Look for how much individual attention was given the
participant.
D. How well are you doing in the class (or job)? How close is this to the objective set out in
your ISS or training plan?
Probe for information regarding the success of the participant in the training situation or job.
Does it appear that the assessment and subsequent counseling resulted in a good or poor
match between the participant and the labor market? If not successful, can the problem be
related to failure of the participant to possess essential qualities needed for the job, e.g., basic
education, physical requirements, interest, etc.?
E. What would you like to see changed about the procedures leading to your employment plan
and referral to training? Why?
Look here for factors relating to the inconvenience and time required as well as the
procedures themselves. Do participants have any special reaction to the staff involved?
How comfortable were they throughout the process? Do they generally find it a helpful
process or just another bureaucratic hoop to jump through?
30
6. RECORDS REVIEW SUMMARY
A. PARTICIPANT RECORDS
INSTRUCTIONS:
Complete record review sheets for an appropriate percent sample of participants. If a review
of this sample reveals inadequate record keeping, increase the sample 10 percent more
records. (Use the Sample Size Table in the Monitoring Resource Section of this Guide.)
Summarize below the percentage that:
_____ documents period of participation
_____ documents activities
_____ records all services received, date service received
_____ contains complete ISS
_____ ISS is signed by participant
_____ contains follow-up information
_____ shows assessment of basic education
_____ shows assessment of interests
_____ shows assessment of work history
_____ shows assessment of vocational aptitudes
_____ shows assessment of general intelligence
_____ shows assessment of life style preferences
_____ shows assessment of physical fitness
_____ shows assessment of supportive service needs
_____ shows ISS implemented
_____ shows ISS objectives achieved
B. CONCLUSIONS:
Note any discrepancies between participant records, observations, and interviewee responses.
Ask for assistance from contractor staff in reconciling discrepancies. Do the participant
records show that assessment and the employment planning process results in good referrals?
If the ISS is not implemented or the objectives not realized, what explanation do the staff
have?
C. What Assessment tools were used? (List)
31
D. What determines which tool is used for a particular customer?
E. Based on your observation do you feel that agency staff is qualified to conduct the
assessment? (yes ) (no )
Comments:
F. Who interprets the test results? Do the same staff that administer the test read the results?
G. How are the test results then used? What is the name of the testing/assessment tool?
(Are the results used in developing the ISS?)
H. Is in-service training provided to the assessment staff? (yes
adequately trained and provided with current information)
Comments:
I. If F is yes, when and what type of training was provided?
32
) (no
) (n/a
) (Are staff
J. Is the assessment process adequate to provide for a thorough ISS development?
(yes ) (no ) (n/a ) If no, explain:
Conclusions:
33
NAME
shows IEP (ISS) objectives achieved
shows IEP (ISS) implemented
shows assessment of supportive svcs needs
shows assessment of physical fitness
shows assessment of life style preferences
shows assessment of general intelligence
shows assessment of vocational aptitudes
shows assessment of work history
shows assessment of interests
DATE:
MONITOR:
shows assessment of basic education
contains follow-up information
IEP (ISS) is signed by participant
contains complete IEP (ISS)
records all svcs rec'd, date svc rec'd
documents activities
documents period of participation
PARTICIPANT IEP/ISS RECORD REVIEW FORM
SERVICE PROVIDER:
COMMENTS
Optional, the State Participant Review Sheets for Adult, Dislocated Worker, and Youth may be substituted.
34
NAME
shows IEP (ISS) objectives achieved
shows IEP (ISS) implemented
shows assessment of supportive svcs needs
shows assessment of physical fitness
shows assessment of life style preferences
shows assessment of general intelligence
shows assessment of vocational aptitudes
shows assessment of work history
shows assessment of interests
DATE:
MONITOR:
shows assessment of basic education
contains follow-up information
IEP (ISS) is signed by participant
contains complete IEP (ISS)
records all svcs rec'd, date svc rec'd
documents activities
documents period of participation
PARTICIPANT IEP/ISS RECORD REVIEW FORM
SERVICE PROVIDER:
COMMENTS
Use in conjunction with the Record Review Summary, Page 30, Item 6.
35
EXHIBIT “C”
Tulare County Workforce Investment Board
FINANCIAL ACCOUNTING AND INTERNAL CONTROL OUTLINE
PURPOSE
This Agency Fiscal Questionnaire with the attached packet of worksheets (1-6) is being used to collect
information for determining whether a service provider with a cost reimbursement contract follows sound
accounting principles, regulatory requirements and its own established procedures in recording and documenting
financial transactions.
PRIMARY CONCERNS
Adequacy of internal controls
Existence and adequacy of procedural guidance
Prompt and accurate posting and reconciliation of bank statements
Adequacy of documentation supporting expenditures and allocations
Effectiveness of monitoring of limits for maximum expenditures
Accuracy of allocation of costs
Accuracy of classification of costs
Control over allowable costs
Reasonableness of cost of space,
utilities, etc.
Security of petty cash
Cash vs. Accrual basis for reports
Adequacy of insurance coverage
Effectiveness of cash management
CONTRACTOR IDENTIFYING DATA
Agency Name:
Address:
Service Providers Fiscal Representative:
Type of Service Provider:
( ) private-for-profit
Is your agency multi-funded? ( ) yes
Phone
( ) private non-profit ( ) public
( ) no Date of most recent audit:
Agencies receiving over $500,000 in federal funding must attach a copy of most recent Audit.
Indirect cost rate used:
%
Are WIA funds obtained by: ( ) Advance
Approved by Cognizant Agency? ( ) yes
( ) Reimbursement
List WIA Contract Number(s) in operation currently:
36
( ) no
( ) Working Capital Advance
Attach a copy of your agency's current organizational chart indicating both the titles and names of all individuals
employed by the agency whether or not they are currently working on WIA contracts. School Districts should list the
individuals within the Department(s) contracting with WIB. If the organization chart does not list the Board of
Director's names, please submit a separate listing of current names and titles.
AGENCY QUESTIONNAIRE
INSTRUCTIONS
The following questions should be answered by the Principal Accountant. In larger agencies, it may be necessary to
address some questions to accounting staff directly responsible only for the WIA program Answer "N/A" to any
questions that do not apply. Answer "Unknown" to any questions of which you have no knowledge.
PROGRAM OVERVIEW
1. Approximately what percentage of agency income comes from WIA funds?
List all other funding sources:
Total of all funding sources
%
100%
2. Which of the following financial records does your agency maintain?
General ledger
Cash receipts journal
Cash disbursement journal
General journal
Payroll register
Accounts receivable ledger
Accounts payable ledger
Purchase journal
Obligation control register
Other:
3. Does your agency's financial manual(s) adequately describe financial procedures in the following areas and
is coverage up-to-date? Indicates yes or no and indicate the date the procedure was added to your manual. Do not
write in the "Verified" column. This will be completed by the monitor at the On-site review.
Date:
Y N
Y N
Verified:
bonding/insurance
payroll-staff
Date:
Y N
Y N
37
Verified:
audit requirements
cash management
Date:
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
N
N
N
N
N
N
N
N
Verified:
Date:
Verified:
payroll-participants
Y N
advances/reimbursements
petty cash
Y N
purchasing procedures
cost classifications
Y N
bank reconciliation
allowable costs
Y N
non-allowable costs
chart of accounts
Y N
cost allocations
travel
Y N
posting to books of account
reconciliation's between subsidiary and official records
internal control procedures and requirements for cash controls
responsibility for recording transactions and for approving transactions
4. Have you received adequate guidance from Tulare County Workforce Investment Board about their
requirements for financial records and reporting? ( ) Yes ( ) No If no, explain
5. Is accounting done ( ) manually or ( ) automated? If automated, explain type of software used for recording
information
Is accounting done on a ( ) cash, ( ) accrual basis, or ( ) modified accrual basis?
6. How are WIA funds and expenditures recorded to separate them from those of other activities/programs?
( ) separate checking account
( ) code(s) used to classify expenses
( ) other
The code(s) used to designate WIA costs:
Code:
Code:
Code:
Code:
Code:
Code:
Code:
Code:
Code:
Code:
Desc:
Desc:
Desc:
Desc:
Desc:
Desc:
Desc:
Desc:
Desc:
Desc:
7. Does your organization receive any income (interest, profit, etc.) from WIA programs? How are these funds
handled and what controls ensure the proper use of income?
(
(
(
(
)
)
)
)
interest from bank account?
sales of products?
"profit"?
other?
38
Comments:
8. How many bank accounts are there with deposits of WIA funds?
.
How many are: zero balance
,
clearing accounts
payroll accounts
,
other
,
.
9. Are all accounts containing WIA bank deposits covered by the Federal Deposits Insurance Corporation
(FDIC)?
( ) no. If no, explain
10. Do you have a source of funds to make reimbursement for costs that might be disallowed?
( ) yes ( ) no
What is the source of reimbursement?
11. What were the findings of the last audit? What corrective action was taken? ( ) None or see below:
12. How long are agency financial records retained?
years
13. Complete the attached Worksheet describing Separation of Duties.
14. How is access to accounting records controlled? Who has access and what is necessary to gain access?
manual records:
automated records:
15. Explain the steps from the authorization of an expenditure to the posting to the general ledger.
16. Are procedures different for payroll than for other disbursements?
( ) yes ( ) no If yes, describe procedures
Is a separate bank account used for payroll?
( ) yes
( ) no If yes, describe payroll reconciliation procedures?
39
17. Are procedures for payment of indirect costs different from other disbursements?
( ) yes ( ) no If yes, describe
18. What is the policy on mandatory vacations or rotation of duties among accounting personnel?
19. Do individuals with check-signing authority have access to accounting records?
( ) yes ( ) no If yes, describe
20. Where are blank checks and signature machines kept? Who has access to them?
21. Are there any automatic teller cards issued?
( ) yes ( ) no If yes, where are they kept and how are they controlled?
22. Can authorized signers sign their own expense or payroll checks?
( ) yes ( ) no If yes, describe
23. How many signatures are required on a check?
Provide samples of all signatures
24. When a person is no longer authorized to sign checks, what action is taken?
25. How are long-term outstanding vendor checks handled?
How are long-term outstanding payroll checks handled?
26. How are undelivered checks (checks to be picked up rather than mailed) handled?
.
40
27. How are voided checks handled?
28. How do you handle situations when there is an immediate need to disburse a check but the
authorized signers are not available?
29. Under what circumstances are checks signed in advance?
30. When can checks be made out to “Cash” or "bearer"?
( ) only petty cash ( ) never ( ) other, describe
31. When can disbursements be made other than by check?
( ) never ( ) from petty cash ( ) other, describe
32. Are funds sometimes transferred (other than by writing checks) from one bank account to another?
( ) yes ( ) no If yes, when? What documentation is required to support the transfer? Why aren't
checks used?
33. When you run short of funds for a project, does anyone or any other fund make internal shortterm loans to it? ( ) yes ( ) no If yes, what documentation is required to support the loan?
Under what circumstances does this occur and how are funds so transferred repaid?
Are these transfers considered to be accounts payable and documented as loans in the general ledger?
Are there written procedures guiding these transactions?
34. Do you have policies with respect to salary advances or loans to staff members?
41
How are advanced or loaned funds protected?
Can salary advances exceed earnings-to-date?
35. How do you ensure that an invoice is not paid twice?
36. How do you ensure that payments are made within a discount period?
37. How often is the bank statement reconciled? Who is responsible for the reconciliation?
38. How often are the subsidiary accounts reconciled with the general ledger?
How often are other accounts/records/systems reconciled?
39. Where are the following items kept, who has access to them and how are they organized?
bank statements:
canceled checks:
documents supporting purchase of service or merchandise (purchase orders, receiving receipts, etc.):
invoices that have not yet been, paid (accounts payable):
documents supporting disbursements (paid vendor files or files containing original paid invoices and
supporting documentation):
payroll records including disbursement journal and reconciliation's of payroll deductions per pay
period:
time and attendance source document records:
Federal and State quarterly and annual payroll tax reports and proof of payment:
42
various subsidiary ledgers (accounts payable, receivable, fixed assets, etc.):
various journals (cash receipts, cash disbursements, general journal, etc.):
original staff travel records (mileage logs for each staff member, including travel dates,
destination, start/end mileage and purpose of trip):
petty cash (access by staff, reconciliation methods and period of reconciliation):
telephone records (phone logs for each staff member, including all long distance and cellular use,
dates, person called, purpose of call and reconciliation to actual telephone billings:
COST ALLOCATION
INSTRUCTIONS:
This section is to be completed whether or not your agency receives only WIA funds or funds from any other
sources in addition to the WIA funds being reviewed. Attach a copy of:
a. the descriptive cost allocation plan (the written methodology used to establish expenditure allocation
percentages),
b. the agency's projected % allocations for all sources including the current year WIA contract(s).
The information may be presented in your agency's current format if the information can be converted
to the attached worksheet 2 format or you may use the copy of the attached format. Information from
documents submitted will be cross-checked for accuracy during the on-site review by examining your
current year journal entries.
Written cost allocation plan available: ( ) yes ( ) no If no, explain
Current year projected allocation worksheet available ( ) yes ( ) no If no, explain
40. Which costs are sometimes shared by different projects/funding sources?
Circle "I" if the following costs are included in an indirect cost pool, "D" if they are direct costs, or
"E" if they can sometimes be either.
I
I
I
I
D
D
D
D
E
E
E
E
space
utilities
office supplies
office equipment
I
I
I
I
43
D
D
D
D
E
E
E
E
administrative staff
program staff
postage
other:
41. In the past three years, were there any major (over $1,000 each) purchases of equipment (that
may be used in on going WIA activities)? ( ) Yes ( ) No If yes, described
Were any major purchases also used by a non-WIA project?
42. How often does your agency review and update it's cost allocation plan?
43. How do you ensure that expenditures, especially non-routine ones, are charged to the correct
project/contract?
Who actually makes this decision and/or how is this information communicated to the bookkeeper?
44. How are shared administrative costs handled?
Is there an indirect cost rate approved by a cognizant Federal agency? ( ) none ( ) provisional
( ) fixed
Current year rate allowed by cognizant Federal agency:
% Which Federal Agency established it?
Does your agency bill WIB for the exact amount of the indirect rate? ( ) Yes
( ) No, If no, explain
45. Which staff members, other than those charged exclusively to indirect cost, work for more than one
project? List names and job titles:
46. What is the basis of allocating above salaries and benefits?
How do you know if this proration is reasonably accurate?
Does the procedure differ for salaries of administrative staff?
47. What is the basis for allocating use of space and utilities?
space
utilities
44
48. What is the basis for allocating telephone and cellular baseline (flat monthly) charges?
What is the basis for allocating long distance and cellular per minute charges?
How are charges for personal calls reimbursed to the agency?
49. What is the basis for allocating travel, supplies, and other routine expenses?
Travel
Supplies
routine expenses
COST CLASSIFICATION
INSTRUCTIONS:
To be completed by all agencies if their contract/agreement requires reporting by WIA cost categories:
50. Has Tulare County WIB provided adequate guidance on how costs are to be classified
( ) yes ( ) no
51. Is your chart of accounts compatible with the cost categories by which you report to the WIB?
52. How do you handle costs which should be charged to more than one category, such as utility costs
which are associated with both classrooms and the administrative office?
53. How do you ensure that a one-time expenditure is charged to the right cost category?
Who makes the decision? How was he/she trained in the cost classification system?
54. Are there any direct costs charged to training that might be more appropriately charged to
administration? ( ) Yes ( ) No Which ones?
PETTY CASH
INSTRUCTIONS:
55. What is the typical total of petty cash disbursements for a month? $
45
56. What is the maximum withdrawal permitted? $
57. Describe how the system works.
prior approval required? ( ) yes
( ) no
documentation required? ( ) yes
( ) no
restrictions when it can be used? ( ) yes
who reconciles and when? ( ) yes
how replenished? ( ) yes
( ) no
( ) no
( ) no
periodic surprise counts? ( ) yes
( ) no
rules and procedures in writing? ( ) yes
( ) no
58. Can the approving officials authorize disbursements to him/herself as payee? ( ) yes
If yes, explain
( ) no
Provide samples of all authorized approving signatures:
TRAVEL
59. What are the procedures for the request and approval for overnight WIA travel?
Who approves the director's request?
60. What are the policies regarding travel advances and their reconciliation?
Clarify procedures used to follow up on advances and how and where they are recorded.
How soon after completion of travel must employees submit vouchers and reimbursement?
46
Days
What procedures are required for local travel in personal vehicles? Allowance:
mile
per
Does staff travel directly to training sites rather than traveling to the agency offices first? ( ) yes ( ) no
When staff travel directly to training sites, is a round-trip adjustment (home to agency office mileage)
deducted from the daily mileage incurred by staff? ( ) yes ( ) no If no, explain
Are copies of staff personal vehicle insurance coverage's kept in ( ) employee's personnel file ( ) a
central file with agency insurance verification ( ) other, explain?
Who monitors the insurance records to verify that they are currently up to date?
Is staff required to wear seat belts while driving privately owned or agency owned/rented vehicles? (
) yes ( ) no If yes, you have written authentication from staff and where is it located
Is staff allowed to transport adult or out of school youth clients in personal vehicles? ( ) yes
If yes, explain circumstances
( ) no
Is staff allowed to transport in-school youth clients in personal vehicles? ( ) yes ( ) no If yes, explain
What procedures are required for local travel in organization-owned or leased vehicles?
What procedures are required for local travel in other organizations' vehicles on non-WIA projects?
SPACE USE
61. Does your agency ( ) rent or ( ) own the building(s) used to house WIA activities?
List all addresses for both administrative and training operations connected with all WIA programs.
62. If space is rented, who owns the building(s)? Attach copy of lease(s) to this questionnaire.
Why was the space(s) selected and how was the amount of rent determined?
Is there any type of relationship between the landlord(s) and anyone on the Agency's
management/board, Agency's WIB program staff, Tulare County WIB staff or management/board?
47
63. If rented, was a security deposit required?
( ) yes ( ) no If yes, how was it accounted for in your general ledger?
64. If the building is owned by the Agency, how is the organization compensated for use of its building(s)
and how was this amount determined?
Do you have documentation which supports the amount charged? ( ) yes ( ) no If no, explain.
INSURANCE
65. Please complete the attached Insurance Verification Worksheet 3 there is insurance coverage
applicable to this contract.
CASH MANAGEMENT
66. How and when do you receive funds from Tulare County WIB? How is the timing and amount of
the request for funds determined?
( ) advance or ( ) reimbursement
( ) draw down or ( ) working capitol advance;
Time laps from date of expenditure to date of request for reimbursement:
Time laps from date of request to receipt of funds:
days;
Number of cash requests each month:
.
days.
If advances are based on immediate need, answer the following questions:
The points at which requests for funds are initiated:
(
(
(
(
)
)
)
)
when an agency's vendor invoice is processed;
when agency's check is written;
when agency's check has cleared the bank;
no policy requirement.
If other than just before check distribution, why?
What is the usual interval between the request for an advance and the actual distribution of the agency/s
checks?
days
67. Does this procedure frequently result in ( ) a shortage of available funds ( ) in substantial
funds in the WIA account that are not immediately needed?
How is the amount of cash requests determined?
Is the amount of request more than necessary or could a lesser amount be requested more
frequently?
48
68. If payment is made by advances, how much is usually kept in your bank account between payrolls
$
. What balance, if any, is required to avoid bank service charges? $
.
69. If advances are made for immediate need, is the gross or net payroll used to determined the need
for funds or to determine the amount of the request for reimbursement? ( ) gross ( ) net
Specifically, are requests for funds for the following made only when payment is due?
FICA & Federal Income Tax
State Payroll Tax
Union Dues
Health Insurance
Life Insurance
Indirect Payroll Costs
Is this cash control ledger kept separately for different WIA titles? ( ) yes
If no, explain how cash is accounted for by different WIA titles
( ) no
How is it done?
How does management make sure that cash balances are not excessive?
MONITORING FUNDING LIMITATIONS
70. How do you ensure that you do not exceed the contract maximum limitations for expenses in any cost
classification and the overall contract?
Who tracks expenditures relative to those planned? How often?
In past years, have you had to slow or discontinue operations because funds were used earlier than
planned?
In past years, have expenditures within cost categories exceeded that budgeted by more than the allowed
amount?
71. Are In-kind contributions required by your current contract(s) with the WIB? ( ) yes ( ) no If yes,
explain the required dollar or percentage amount and the types of contributions required by your
contract(s):
Are any non-required In-kind contributions donated toward the cost of running your current WIA
contract(s)? ( ) yes ( ) no If yes, please explain?
49
How are in-kind contributions reported to the WIB? ( ) Monthly ( ) Annually ( ) Other
Explain what types of in-kind expenses are contributed by your agency, how the cost basis of each
contribution is determined and which cost category (Admin, Direct training or Training related) the
contributions are associated with:
volunteer services (including estimated fringe benefits)
space (including janitorial)
utilities
telephone
insurance
supplies
postage
duplicating/printing
publications/advertising
travel
equipment including rented/leased equipment and maintenance
contracted services including auditing and accounting
direct training or training related equipment or supplies
50
Certification by Agency:
Answers prepared by:
date:
date:
I hereby certify, under penalty of perjury, that the above information is true and correct and can be
corroborated by agency staff and agency records.
Signed:
Title:
Date:
Verification by Tulare County WIB:
Date received:
Reviewed by:
Date reviewed:
Title:
WIB recommendations to existing procedures:
Discussed with agency representative:
date:
Recommendations to be implemented:
Accepted by agency representative:
date:
Title:
Approved by Program Manager:
date:
51
INSTRUCTIONS FOR REVIEW OF ORIGINAL SOURCE DOCUMENTS FOR
DISBURSEMENTS AND RECEIPTS
Answer subsequent questions by circling Y (yes), N (no) or ? (doubt because information lacking or because
the findings are borderline). The desirable answer is first, permitting a quick scan for problem areas. Any ?
response should be explained. Undesirable responses should be explained or indicated by highlighting
appropriate examples on the worksheets 5 and 6.
80. Using Worksheet No. 5, select and trace a sample of transactions from the disbursement journal or
record of expenditures. These should be dated within the three-month period being reviewed. The
selections are made to determine if: proper approvals were obtained; the recorded amounts are accurate;
adequate original support documentation exists; the cost was allowable (per EDD WIA DO3-10
Allowable Costs); the cost was correctly allocated (to each WIA or non-WIA program or activity); and
the cost was classified (to appropriate cost categories). Since this analysis is not intended to be
representative, select entries that seem more likely to be questionable. For example, routine payments
to vendors such as the landlord, utilities, etc., might be omitted. A pre-visit desk review if the FADS
spreadsheets (which include the comparison to the contract budgets) and/or the invoices for the specified
period should identify any non-routine payments that should be selected for review. During the desk
review and/or the source document review, make copies of any questionable transactions to attach to the
worksheet and annotate the reasons the item was questioned and any response made by the accountant or
bookkeeper concerning the item.
81. Using Worksheet No. 6, complete an examination of selected vendors from the agency’s vendor file.
This task is similar to task 80, but it omits the linkage to the record of expenditures or disbursements. If
documents are filed within vendor folders, it permits a more efficient and rapid examination of
transactions, especially with respect to whether they were reasonable and necessary. Select vendors on
the basis of increased probability of questionable transactions as identified in the desk review (e.g.,
unusually high or one time purchases, spending overages in certain budget line items, etc.). In selecting
files, ask the accountant or bookkeeper about the vendors and what they provide to the contractor.
During the desk review and/or the source document review, make copies of any questionable
transactions to attach to the worksheet and annotate the reasons the item was questioned and any response
made by the accountant or bookkeeper concerning the item.
Answer the following based on Worksheets 5 and 6 provide explanation where appropriate:
Y N ?
Were all transactions traceable to canceled checks?
N Y ?
Were there any discrepancies between the check amount or date and the journal
entry?
N Y ?
Did any transactions appear to be unnecessary, unreasonable or not allowable?
Y N ?
Did adequate documentation (original invoices vs. statements) support all
transactions?
Y N ?
Were payments made within the discount period and discounts taken?
Y N ?
Were the check numbers and dates of payment entered on all original source
documents, invoices or statements to indicate that payment was made?
Y N ?
Was there proper and prior authorization by an authorized signatory of the
agency?
52
Y N ?
Was the cost classification and/or G/L # (identifying program and cost category)
entered on all invoices (or equivalent back up)?
82. Examine the numerical sequence of checks in the disbursement journal to determine if any check
number is missing.
Period for which check numbers will be examined:
Check Numbers of missing entries
Or none ( )
If there is a missing entry, ask for an explanation. Check subsequent bank statements to see if the
checks cleared.
84. Look at the data entry dates in the disbursement journal and general ledger for the review period.
What are the dates of the last posting during the review period?
To the disbursement journal:
To the general ledger:
Y N ?
Are postings reasonably up to date (within one week of date of disbursement
and within one month to posting into the general ledger?
Y N ?
Are books of original entry in ink (and corrected by lining out the amount and
inserting the correct amount) or on computer generated media (that cannot be
revised or changed except via initiating a correcting journal entry)?
Y N ?
Are corrections made by white out or erasure (unacceptable) or by revising
original computer entries (unacceptable)?
85. Compare the list of payments from WIA to the contractor’s entries in their cash receipts journal, using
Worksheet 4 completed during the Desk Review.
Y N ?
Are the two records always consistent in dollar amounts and deposits made
within reasonable period? Inquire about apparent differences in amounts or
dates.
85. Review several hundred actual canceled checks and make copies as necessary:
Y N ?
Are any checks made out to Cash? List or copy all checks made out to
Cash. Use this information for a review of petty cash disbursements posted
to the G/L.
Y N ?
Do any endorsements appear unusual, e.g., more than one endorsement, or
any authorized signatures that appear different?
86. After comparing checks made out to Cash to the petty cash items in the disbursement journal, note
any unmatched items. Record an explanation for each check not used for the petty cash fund.
87. Follow up to determine if voided checks were actually traced and filed so that all checks in the series
picked for review can be readily accounted for.
88. Ask the person who reconciles the bank statement about the procedures used to verify that the balance
is consistent with ledgers. This should include:
53
frequency of bank reconciliation’s (within two weeks of receipt of bank statement or by the end of
the following month)
how long-outstanding payroll and vendor checks are handled (written off after what period)
how bank transfers (if done) are handled
that canceled checks (including voided checks) are actually compared to the disbursement
journal for date, amount, and payee
that the recording of disbursements and receipts (by)
and the signing of checks (by)
is done by a person other than the reconciler: (by)
89. Examine the bank statements and bank reconciliations for the review period to determine if proper
reconciliations were done. Make copies of the review period documents as needed.
Date of the last review period reconciliation:
Y N ?
Were outstanding checks itemized?
Y N ?
Does it appear that the statement was reconciled with the general ledger?
Y N ?
Does the reconciler sign and date the statement or worksheets?
Y N ?
Were the reconciliations done by a employee who was not responsible for
recording receipts or disbursements or signing the actual checks?
90. If the agency receives advances from WIB, examine a sample of bank statements for the past 6-12
months.
Y N ?
Were there transfers between accounts? (If so, ask for documentation to support
the transfer and determine if this procedure was necessary.)
Y N ?
Did the bank pay interest on the balance? (If reimbursement is required, ask for
canceled checks showing reimbursement.)
91. Ask to see records over 2-1/2 years old (or within the number of years required by the contract).
Y N ?
Have they been retained?
92. If accounting is done on a cash basis, ask to see the worksheets for the last reporting period which show
the adjustments made to obtain accrued figures in the General Ledger report.
Y N ?
Does the worksheet agree with the report for that period?
Y N ?
Are accrued leave obligations included in the accrued computations?
Y N ?
Does it appear that the proper adjustments were made?
54
CONCLUSIONS:
Are there adequate audit trails for all disbursements?
Are disbursements documented properly to support costs as allowable, allocable, reasonable and
necessary?
Are disbursements made by check only, except for petty cash?
Are all disbursements recorded accurately?
Were internal controls followed?
Are there any discrepancies between this record review and other information obtained either during the
interviews or the desk review? Clarify any discrepancies at this time.
55
Source Document Review:
Before departure, secure and review the following prior to going on-site (Requested in letter of
Introduction):
Service Provider Identifying Data
Most Recent Audit
Cost Allocation Plan
Agency Fiscal Questionnaire
Disbursement to Service Providers (Worksheet 4)
Current Organizational Chart
Financial Manuals from Agency (on-site).
Previously monitored or audited: Date(s):
After reviewing previous program and fiscal monitoring reports and documents relating to corrective actions,
note areas that require special attention:
Other recent problems or issues likely to need special attention:
Review, contract proposals, and other documentation provided during the procurement process.
Discussion with Program Coordinator assigned to Agency:
Review of MIS statistics (Performance Operating Plan and Statement of Work, etc.) related to
performance of agency:
Specific items for review:
56
If the Service Provider receives advances, complete Worksheet 4 (Disbursements to Service Provider) or
worksheets on which payments by WIB to the Service Provider are to be recorded. Compare amounts to
the contractor’s record of deposits and bank statements.
Other desk review information:
Specific questions to review with Agency:
Regulations/guides/instructions on hand and in use by fiscal staff at agency:
Contract # (s)
Boilerplate dated
DOL WIA, Final Rule Provided by WIB
WIB Local Policies
Other
57
WORKSHEETS
Referenced in the
Financial Accounting and
Internal Control Outline
58
WORKSHEET NO. 1
Separation of Duties
Enter names of staff who have responsibilities at top. Place checkmark below name if person
has responsibility for that function.
Name
Approves Purchase Orders
Verifies receipt of order
Prepares request for funds
Authorizes disbursement
Prepares checks
Signs checks – manual
- machine
Custodian of check signing device
Custodian of blank checks
Compare checks with vouchers
Distributes checks
Post Disbursements
Computes cost allocations
Receives "cash"
Post receipts
Deposits receipts
Custodian of petty cash
Petty cash replenishment
Audits petty cash
Bank reconciliation
Maintains general ledger
Prepares financial report
Approves financial report
59
WORKSHEET NO. 2
VERIFICATION OF COST ALLOCATION
This worksheet is to verify whether direct costs were disbursed according to the cost allocation
plan.
Columns 1 and 2:
Select up to 12 which were allocated among projects and list them by
check number (columns 1 and 2).
Column 3:
Enter the amount (%) to be allocated to WIA according to the cost
allocation plan, or in its absence, the claimed prorating policy.
Column 4:
Verify that the amount charged to WIA is consistent with that scheduled
(planned)
Service Provider:
Allocation
Check Number
(1)
Payee/Purpose
(2)
1
2
3
4
5
6
7
8
9
10
11
12
60
Scheduled
(3)
Actual
(4)
WORKSHEET NO. 3
INSURANCE
Contractor: ___________________________________________
Carrier
Amount
of
Coverage
Date
Cost
Start
End
Fidelity Bond*
General Liability
Auto Liability
Worker Compensation
xx
Property/Casualty
Unemployment Insurance
xx
xx
*Type of staff covered:
61
xx
xx
xx
Paid
Check
#
WORKSHEET NO. 4
DISBURSEMENTS TO SERVICE PROVIDER*
Service Provider:
Invoice Payments
Check No.
Recorded by Service Provider
Date
Amount
Date
*To be completed if contractor receives advances.
62
Correct?
INSTRUCTIONS FOR WORKSHEET NO. 5
Select entries from the disbursement journal and examine supporting documentation for each
selected entry. Enter the following information:
From disbursement journal:
enter basic data as indicated
From canceled checks:
check?
- enter check mark if present
accuracy?
- enter check mark if amount, date and vendor name correspond
From the support documentation:
purpose
- enter stated purpose or “not indicated” or “not clear” as appropriate
allowable
- enter check mark if expenditure is allowable, a question mark (?) if
not or if questionable
invoice?
- enter a check mark if an original invoice is present; this should be in
addition to any statement summarizing several purchases
adequate?
- enter check mark if the invoice is adequate (quantity, description and
unit cost shown? amount the same?)
discount?
- enter check mark if paid within discount period; enter NR if not
eligible for discount
authorized?
- enter a check mark if there is a proper signature giving approval for
the disbursement
check#?
- enter a check mark if the invoice/statement of authorization form has
the check number entered to indicate that payment was made
cost category?
- enter a check mark if the cost classification is entered on the invoice,
statement, or authorization form
NOTE: Credit card statements should have original receipts as well. Purpose, allowability, and
cost category should be determined for each item on statement.
63
WORKSHEET
NO. 5
ACCOUNTING RECORDS
Contractor:
Cost Category?
Check #?
Authorized?
Discount?
Comments
Adequate?
Purpose
Invoice?
On the Basis of Support Documentation
Allowable?
Check?
Cost Category
Date
$ Amount
Check #
Payee
Accuracy?
Checks
From the Disbursement Journal
Comments
66
INSTRUCTIONS FOR WORKSHEET NO. 6
If the Service Provider files documentation by vendor, look through for the vendor files and
examine records from several vendors which appear atypical or of special interest. Enter the
following information:
Check #
- enter the check number of canceled check
Amount $
- enter amount of check
Correct $?
- enter check mark if check amount correspond to the amount billed
Purpose
- enter stated purpose or “not indicated” or “not clear” as appropriate
Allowable
- enter check mark if expenditure is allowable, a question mark (?) if not or if
questionable
Invoice(s)?
- enter a check mark if an original invoice is present; this should be in
addition to any statement summarizing several purchases
Adequate?
- enter check mark if the invoice is adequate (quantity, description and unit
cost shown? amount the same?)
Discount?
- enter check mark if paid within discount period; enter NR if not eligible for
discount
Authorized?
- enter a check mark if there is a proper signature giving approval for the
disbursement
Check#?
- enter a check mark if the invoice/statement of authorization form has the
check number entered to indicate that payment was made
Cost category?
- enter a check mark if the cost classification is entered on the invoice,
statement, or authorization form
NOTE: Credit card statements should have original receipts as well. Purpose, allowability, and
cost category should be determined for each item on statement.
65
Vendor/
Payee
Correct$?
Amount$
Check#
Purpose
Cost Category?
Check#?
Authorized?
Discount?
Adequate?
Invoices?
Allowable?
WORKSHEET NO. 6
Accounting Records by Vendor
Contractor:
Comments
68
EXHIBIT "D"
YOUTH PROGRAMS
Service Provider:
Date:
Person Interviewed:
Program:
Monitor:
COMPETENCIES
Employment workplace competency system: review the program activity and take in consideration
the activity design and expected results.
A. Are WIB approved competencies, being used and are required documentation in file to
support the need for training, i.e., ISS pre-test, etc.?
Comments:
B. Which competency areas have been identified as needed by the participant?
[ ] Work Readiness
[ ] Occupational Skills
[ ] Basic Skills
C. How does the agency document that the participant has successfully completed the WIB
approved competencies? Verify in file review.
Comments:
D. List the names of instructors/Counselors qualified to administer/teach employment
competencies and certify competency attainment.
1.
2.
3.
4.
5.
6.
YOUTH TEST SCORES
A. How do you document that the participant was deficient and remediated in at least two of the
three WIB approved competency areas before a competency completion is claimed?
67
F. If Job Specific competencies are used, how is training being provided?
G. If Pre-Employment/Work Maturity competencies are used, how is training provided?
H. If Basic Skills is used as a competency, how is training being provided?
I. How is length of time in training being documented?
Review the ISS in order to insure that it is being used to document the need for training
accomplishments, etc.
J. The following will be verified during Competency review. Attach copy of the file review
report upon completion. (Youth Competency Matrix)
a. Competency Assessment/Attainment Form must be complete and include all dates and
signatures.
b. Pre-test must include all tests; check for minimum deficiency levels.
c. Post-test must include a test of each area that was identified as deficient and include
back-up attendance records where required.
d. Curriculum: review a copy of the outline or competency acquisition plan.
e. ISS's must be updated.
f. A completed copy of WIB approved certificate.
g. Enrollment verification, review appropriate MIS documentation to check for enrollment
during time of instruction.
2. PRIORITY/TARGETED GROUPS
A. Is priority being given to High School dropouts? (yes
) (no
) Is priority given to
individuals who do not meet established levels of academic achievement and who plan to
enter full-time labor market upon leaving school? (yes ) (no )
1. How do you ensure that priority is being given?
68
B. Is the curricula designed to prepare the student to meet State and locally determined general
education diploma and basic education competency requirements? (yes ) (no )
1. What process do you use?
C. To verify level of need or competency requirements do you use:
WIB Competency
School Records
Pre-test
Post-test
D. How, and how often, are participants evaluated to determine their progress in training?
(training reviews and/or evaluations should be conducted at least every 30 days)
1. Where is this documented (ISS, case record, etc.)?
E. Are services in excess of 200 hours being provided? (yes
insure that this limit is not exceeded?
) (no
) How do you
F. What process does the Service Provider use to insure that work activities do not occur
until the participant receives Pre-employment/Work Maturity Skills training?
G. How do you insure that participants have not held a regular, part-time or summer job of
more than 250 hours paid employment, prior to enrollment.
H. Are youth working more than 20 hours a week during the school year? (yes ) (no__)
(Sample) (Review appropriate records to insure that child labor laws are being adhered to
as appropriate.)
I. What processes are used by the Service Provider to select Work Experience/Internship
training sites in public agencies (or private sector for internship only)? (Review worksite
agreements to insure that they are properly executed.
69
J. When conducting Internship, is a written agreement developed between the Service Provider
and the employer? (yes
) (no )
If yes, review:
1. Does the contract:
Task orientated training plan.
Have a training outline or job description?
List the number of participants?
Contain a requirement to hire upon completion?
Contain a non-performance termination clause?
Comments:
70
YOUTH PROGRAM ACTIVITY
Work experience
Internship
( )
( )
INTERVIEW GUIDE
INSTRUCTIONS:
The following questions may be directed to more than one of the Service Provider staff since there
are a variety of subjects covered. Upon introducing yourself to the representative, explain the
purpose of the interview and quickly review the subjects to be covered. The representative can then
determine who the appropriate respondents are and arrange for their availability. This can be done
by telephone prior to the site visit. Also note that some of the information called for in this interview
may be available in the contract or other records available to you. If so, complete these sections
prior to the interview and use the interview to confirm and clarify those points. Note to Monitor:
Indicate if this is a work experience site or internship and use only the appropriate portions of this
checklist.
Interviewees' Names
Title
1. PROGRAM OVERVIEW
A. What is the general purpose of work experience/ internship? Describe briefly your approach
to developing positions and matching them to participants. How do you view your progress
in developing the number and type of worksites set out in the contract?
The purpose of this question is to obtain a general orientation to the Service Providers
operations and to assure that the understanding obtained from the desk review is accurate.
What are the criteria used to approve worksites? Summarize the response below.
B. Where are participant, worksite and contract records kept? Is there a written worksite
agreement that can be reviewed?
This question assumes that the Service Provider responsible for developing and supervising
worksites. Response to this question locates where data sources are and identifies logistical
71
problems around completing the monitoring visit in the event that some of the required data
is maintained offsite. If there are work site agreements, ask for a copy to determine how
much information can be secured from these documents.
C. What are the objective of the work experience/internship?
Is it intended that this teach good work habits to participants with no prior work history?
correct poor work habits identified earlier? teach specific job skills? provide for career
exploration? Is work experience or Internship linked to other training activities managed by
this or another service provider? Is WE/Internship the primary activity in the ISS or does it
complement concurrent activities?
CONCLUSIONS:
Review the information in this section. You should have a clear picture of the organization
and activities of the WE/Internship's Service Provider in terms of number of work sites,
general approach, key staff involved, and WE's role in the participant's overall ISS
development.
2. APPROPRIATENESS OF WORK EXPERIENCE OR INTERNSHIP FOR
PARTICIPANTS
A. Describe how participants are recruited for work experience/internship.
If the Service Provider is responsible for recruiting WE/IS participants, how is it determined
that they are WIA eligible? Are there any criteria for entering WE/IS activity other than the
eligibility criteria?
72
B. Please demonstrate through written records or documents how it is determined that the
WE/IS slots are suitable for participants. How are participant assessment data used to
match participants to particular WE/IS positions?
Whether the Service Provider is responsible for recruitment or not, there should be some
evidence in their records that supports the participant's referral to a particular WE slot.
Answering this question may require some reference to participant files or other documents.
Look for indication in the documents or interviewee's response that the work sites are
selected with participant employability needs in mind. What kinds of assessment data
support the referral to WE? Describe this process of reviewing work sites prior to the
assignment of participants.
CONCLUSIONS
Is there sufficient reason to believe that work experience or Internship is appropriate for the
participants from the information above? If not, do you require additional information?
Statements to the effect that WE or Internship meets financial needs or that particular work
sites are assigned because they are accessible to the participant does not satisfy the
requirement that the work sites address employability objectives of the participant.
3. PROCESS INDICATORS OF QUALITY OF TRAINING
A. Are there learning objectives stated for WE/Internship? Are they measurable?
Learning objectives should be stated in behavioral specific terms, e.g., 90 percent attendance
and 100 percent on-time, and be readily measurable, particularly as they relate to work
habits. Is the employer well acquainted with the learning objectives? Are they in writing?
73
B. How do participants learn how well they are doing on the job?
Feedback is critical to learning and may take the form of work evaluations, comments by the
supervisor, peer evaluations, or practice where they must perform a task to a set standard.
Identify the form of feedback used.
C. How do you determine when the WE/Internship objectives have been achieved? Is the
assignment open-ended or for a set period? Do supervisors hesitate to “graduate” successful
participants?
The purpose of this question is to determine if WE/Internship is directly related to
employability objectives and is therefore ended when those objectives are obtained.
D. How do you assure that the worksite supports the employability plan? What form of
worksite feedback is used to determine that the participant is progressing toward the
employment objectives? How many participants have achieved their training objectives or
employment?
Recognizing that employers may object to disruption of work for the purpose of monitoring
the training and may be unwilling to expend much energy completing written evaluations,
the contractor must demonstrate, nevertheless, methods for assuring that the terms of the
contract are being met. Probe especially for how the quality of training is assured and
performance benchmarks are achieved.
E. How do you determine that working conditions meet the standards and requirements for
WIA WE/IS worksites?
This question is directed to the issue of work site compliance with the State and federal
safety laws, child labor laws, political, sectarian and maintenance of effort provisions. How
does the contractor orientate the employer to these requirements? Note that if participants
are less than 16, special work site requirements are imposed by the State.
74
CONCLUSIONS:
Determine if Work Experience/Internship serves the general aims of preparing participants
for employment as described in the contract and that the number of participants being served
meets contract performance benchmarks. Make note of points that you feel need further
confirmation from other data sources.
4. ACCURACY OF TIME AND ATTENDANCE RECORDS
A. How do you assure the accuracy of participant attendance and time records that support the
payroll?
This question is concerned with the procedures used to track and record participant
attendance at the work site. Do both the work site supervisor and participant sign the time
sheet? How often are time sheets collected? What is the length of the pay periods? Do
participants punch a time clock?
B. If the Service Provider is responsible for participant payroll, how do you ensure that
wages do not exceed the limits set by the contract?
Because wages are paid on an hourly attendance basis, participant attendance controls the
payrolls. How does the contractor monitor participant attendance so that budgeted
participant wages is consumed but not exceeded?
75
5. WORK SITE SUPERVISOR INTERVIEW GUIDE
INSTRUCTIONS:
Worksites must comply with various administrative requirements in addition to providing an
appropriate experience for participants. Interviews accomplished while the participant is on-site
provides opportunity to observe the training experience and the working conditions while
obtaining the work site supervisor's perceptions of the purpose of WE. The work site supervisor
should be the person who directly supervises the participants and is most responsible for their
learning experience.
Work Site
Interviewee and Title
A. How long have you functioned as a worksite supervisor for trainees? How long have you
supervised this participant?
This question establishes the supervisor's experience and familiarity with the participant now
under his or her supervision. Is the most qualified employee serving as the work site
supervisor?
B. How does (do) the participant(s) fit in with the work done here?
The purpose of this question is to determine if the participant is compatible with the work
environment in terms of both basic skill requirements and personal attributes. Probe for how
the participant's appearance, social skills, attention to detail, basic education level and
personality match the work site.
C. What are the participant’s duties?
Probe for specific responsibilities, kinds of equipment used, products expected and level of
difficulty represented by each. If you are unfamiliar with the nature of the job done by the
participant, ask these questions of the supervisor. How much responsibility does the
participant have?
76
D. How is the participant’s attendance tracked? What records of the participant’s attendance are
maintained?
This question checks on the accuracy of the attendance records that support wages paid the
participant. Are they complete and verified by the supervisor and the participant? How does
the supervisor check attendance? Does the work site supervisor's response corroborate
information given by the contractor?
E. Are you familiar with requirements for WIA work experience sites? (I.E., Child Labor
Laws) (yes___) (no___) How did you become familiar with them?
CONCLUSIONS:
How would you judge the value of this work site in achieving the stated employability
objectives for the participants? What are its strengths and weaknesses in terms of number of
participants it can serve, type of learning objectives it can achieve, types of participants it can
serve, accessibility to participants, attractiveness of the work environment, and quality of the
supervision? In terms of your observations, how effective do you think this supervisor is with
disadvantaged participants? How would you characterize the supervisor's relationship with the
participants paternalistic, officious, friend, peer? Does the information obtained from this
interview comply with the contract statement of work?
77
6. PARTICIPANT INTERVIEW GUIDE
INSTRUCTIONS:
Participants may be interviewed individually or as a group if they share the same work site. If
interviewed as a group, it is important that no one individual dominate the discussion and that
the participants not feel that they or their supervisor are being investigated. Use the following
questions to learn the participants' perception of their work experience. Begin the interview with
introductions, recording their names below. Use respondent initials to attribute significant
responses to a particular individual, particularly if the information requires some additional
follow-up.
INTERVIEWEES:
A. How long have you been at this worksite?
This question establishes the respondent's experience with the work site.
B. Is attendance recorded every day?
This question confirms whether necessary records of attendance are made.
C. What happens if you are late?
The purpose of this question is to determine if being on time, an important work related
behavior, is being reinforced by the work site. Probe to learn if the work site supervisor's
response is similar to that of an employer of a regular employee.
D. What happens if you miss time at work?
Are good work related behaviors being reinforced? Can lost time be made up? Are
attendance requirements understood by the participants and enforced by the contractor?
78
E. Do you think that you get enough attention from your supervisor?
Do the participants think that they get enough job coaching and help from their supervisor?
If not, what do they think accounts for that? too many participants or other people to
supervise?
F. Do you ever feel that you are wasting your time? When?
The purpose of this question is to determine the participants' perceptions about the usefulness
of the work experience. Probe for why they may think their time is being wasted. Are such
feelings associated with a particular area of the training?
CONCLUSIONS:
To what extent does the information from the participant interviews confirm or refute
information from other sources? Where there appears to be discrepancies, restate the
question or probe more thoroughly to assure that there is no confusion. Note areas of
disagreement with other data sources for further investigation.
79
RECORDS REVIEW
PARTICIPANT RECORDS
WORKSHEET NO. 1
INSTRUCTIONS:
Complete Participant Record Review forms for a sample of the trainees covered by this contract. If evidence of poor record
keeping is found, increase the sample. Summarize below the percentage that:
_____ documents period of participation
_____ documents attendance in the form of signed time sheets
_____ documents contacts and communications with participants
_____ documents progress in completing WE objectives
_____ contains information which substantiates appropriateness of work experience
_____ contains information regarding reason for termination of work experience
_____ documents wages and benefits received and position as an employee of the employer of record
_____ contains work permits for participants under 16 years of age
CONCLUSIONS:
Are participant records orderly and complete? Is there any reason why they cannot be easily reviewed? Note any
observations that confirm or refute information obtained from other sources. Try to reconcile any disparities with
information contained in the participant records and information obtained from interviews. Note any discrepancies below.
80
INSTRUCTIONS FOR WORKSHEET NO. 2
Complete the Work Site Checklist by reviewing a sample of work site records. List on the form in
the first column the name of the work site supervisor, the number of participants supervised, and the
occupations represented. Under the column A “Time Documented”, check if time sheets or some
other form of documentation of attendance at the work site is in evidence. Also check the next two
columns if starting and ending dates for each of the participants assigned to the work site are
recorded.
The column, “Work Site Standards” refers to indications that the work site complies with all
applicable laws governing WIA work sites and Child Labor Laws. Check this column if all
standards are in force, or note which ones are not in force.
“Work Permit” column should be checked if the work site or contractor has on file work permits for
any youth assigned to the work site.
81
WORKSHEET NO. 2
WORK SITE CHECK LIST
Date
Work Site
Supervisor
Number of
Participants
Occupations
Time
Documented
Start
End
Work Site
Standards
Work
Permit
84
EXHIBIT "F"
CLASSROOM TRAINING
STAFF INTERVIEWS
1. APPROPRIATENESS OF TRAINING FOR PARTICIPANTS
A. Describe how participants are recruited for Classroom Training.
How is it determined that participants are WIA eligible? Are there any criteria for
entering Classroom Training other than the eligibility criteria? What is the primary
source of referrals? LWIA? Others? Employers?
B. Please demonstrate though written records or documents how it is determined that the
Classroom Training is suitable for participants?
There should be some evidence in their records that supports the trainees' referral to a
particular training. Look for indication in document or interviewee's response that
participants either asked for vocational training or reflected in their past work history or
vocational assessment an interest in work related to the training occupation.
C. What are the employment and vocational training backgrounds of the participants?
The primary purpose of this question is to ensure that the participants are not already
trained in this occupation and, in fact, need the training provided in this contract. Look
for any evidence that the training duplicates previous training or pre-existing knowledge
and skills.
83
2.
PARTICIPANT INTERVIEWS
INSTRUCTIONS:
If participants are to be interviewed as a group, try to assemble them in the classroom if
possible. Use the following questions to learn the participants' perception of their training
experience. It is important that no one individual dominate the discussion and that the
trainees not feel that they or their instructor are being investigated. Begin the interview with
introductions, recording their names below. Use respondent initials to attribute significant
responses to a particular individual, particularly if the information requires some additional
follow-up. Respondents should be representative of the class, not the "best" or "poorest"
students.
INTERVIEWEES:
A. How long have you been in training? Did you have this training or some similar course
before?
This question establishes the respondent's experience with the course and any prior training
and confirms the extent to which the class is fully operational.
B. How did you get into the course?
This question verifies the institution's statements regarding how appropriateness of the
training was determined. Look for confirmation that participants were interested and were
involved in the decision to enroll in the training.
C. How do you know how well you are doing?
Do participants seem to know where they stand as far as class standards for acceptable
progress? How is this feedback provided?
84
D. What happens if you fail a test or have trouble with the class work?
The purpose of this question is to confirm how slower students are accommodated in the
training and whether the first signs of difficulty are responded to in some constructive way.
How do the participants feel about the response given to slower students?
E. Is attendance recorded every day?
This question confirms whether necessary records of attendance are made.
F. What happens if you are late?
The purpose of this question is to determine if being on time, an important work related
behavior, is being reinforced by the classroom training. Probe to learn if the instructor's
response to lateness reinforces good work habits.
G. What happens if you miss class time?
As in question F, are good work related behaviors being reinforced? How is lost time
made up so that progress in the class is maintained? Are attendance requirements
understood by the participants and enforced by the instructor?
85
H. Are there enough books, materials, time in the lab for practice and attention from the
instructor?
The purpose of this question is to learn if the class is adequately supplied and equipped and
that the instructor/trainee ratio is adequate. Ask about each item separately so that the
responses are focused clearly and can be recorded as follows:
Books:
Materials:
Instructor Time:
I. Who do you go to when you have a problem? What generally happens?
To what extent participants have access to assistance when they have problems. This
question should confirm that counseling or case management services are being provided by
the Classroom Training Unit. Do they know the name of their classroom counselor?
Are there any reported incidents of grievance, complaints, problematic issues?
By:
Date:
Issue:
By:
Date:
Issue:
By:
Date:
Issue:
86
CLASSROOM TRAINING
RECORDS REVIEW
3. PARTICIPANT RECORDS
INSTRUCTIONS:
Complete records review sheets for a sample of participants. If a review of this sample reveals
inadequate record keeping, double the sample. Summarize below the percentage that:
_____
documents period of participation
_____
documents attendance
_____
documents all contacts and communications with participants
_____
documents progress in completing training
_____
records all services received, date service received
_____
contains information substantiating appropriateness of training
_____
contains information regarding reason for termination of training
CONCLUSIONS:
Note any observations that confirm or refute information obtained from other sources. Try to
reconcile any disparities with information contained in the participant records and information
obtained from interviews. Note any discrepancies below.
87
EXHIBIT "G"
ON-THE JOB TRAINING
CONTRACTOR IDENTIFYING DATA
Name:
Address:
Contract Number:
Contractor Representative:
Date On-Site Visit:
Monitored by:
88
OJT INTERVIEW TOPIC GUIDE
INSTRUCTIONS:
The following questions may be directed to more than one of the staff since there is a variety of
subjects covered. Upon introducing yourself to the representative, explain the purpose of the
interview and quickly review the subjects to be covered. The representative can then determine
who the appropriate respondents are and arrange for their availability. This can be done by
telephone prior to the site visit. Also note that some of the information called for in this
interview may be available in the contract or other records available to you. If so, complete
these sections prior to the interview and use the interview to confirm and clarify those points.
Interviewee's Name
Title
1. PROGRAM OVERVIEW
A. Describe your approach to developing OJT positions and matching them to participants.
How do you view your progress in achieving contract performance goals?
The purpose of this question is to obtain a general orientation to the contract's operations
and to assure that the understanding obtained from the desk review is accurate.
B. How many participants have been served in this or previous brokering contracts and
what were the placement results?
This question seeks information about the contractor's past track record. Look for
differences between the current and the past contracts' trainees, employers or approach
that might account for differences in outcomes. Has the Service Provider demonstrated a
capability in either serving a similar target group or in developing effective OJT
contracts with similar employers?
89
CONCLUSIONS:
Review the information in this section. You should have a clear picture of the
organization and activities of the contractor in terms of its size, general approach, key
staff involved, and past track record. Reconcile any discrepancies in the information
given in the interview and the information gathered during the desk review.
D. Where are participant and contract (employer) records kept?
This question assumes that the contractor is responsible for developing and
supervising employer OJT contracts.
E. Who are the staff who develop the OJT sites? Are they the same personnel who refer
the participants and develop and supervise the contract?
This question identifies the key actors so that when they are referenced in subsequent
discussions, you are clear about their roles relative to the services provided and the
administration of the contract.
2. APPROPRIATENESS OF TRAINING FOR PARTICIPANTS
A. Describe how participants are recruited for OJT.
If the service provider is responsible for recruiting OJT participants, how is it determined
that they are WIA eligible? Are there any criteria for entering OJT activity other than the
eligibility criteria? What is the primary source of referrals to OJT? LWIA? Other
vendors? Employers?
90
B. Please demonstrate though written records or documents how it is determined that the
OJT slots are suitable for participants? Describe how participants are matched to
particular OJT positions.
Whether the Service Provider is responsible for recruitment or not, there should be some
evidence in their records that supports the trainees' referral to a particular OJT slot. Look
for indication in document or interviewee's response that participants either asked for
vocational training or reflected in their past work history or vocational assessment an
interest in work related to the OJT occupation. What role do employers play in selecting
participants for a particular slot?
C. What are the employment and vocational training backgrounds of the participants
enrolled in OJT?
The primary purpose of this question is to ensure that the participants are not already
trained in this occupation and, in fact, need the training provided in this contract. Look
for any evidence that the training duplicates previous training or pre-existing knowledge
and skills.
CONCLUSIONS:
Is there sufficient reason to believe that the OJT training is appropriate for the
participants from the information above?
3. ADEQUACY OF FOLLOW-UP AND CONTRACT SUPERVISION
A. How often is the employer site visited and what services are offered to participants
during these occasions? Must be visited at least every 30 days.
This question is aimed at the quality and frequency of contracts between the broker's
91
staff, the employer and the participant. How often is the employer site visited? Is the
participant contacted by telephone or in person? Are other services available to the
participant in the form of counseling for work adjustment, remedial education etc.? How
would you characterize the nature of the interactions between the broker and the
participant once the OJT begins?
B. If a participant quits or has difficulties, how do you know about it and how do you
become involved?
This question gets at the methods by which the broker mediates problems arising
between the participant and the employer. What options are available to the
contractor when a participant quits or is having difficulties?
C. How does the service provider assure that the terms of the contract are met?
Recognize that employers may object to disruption of work for the purpose of
monitoring the training and may be unwilling to expend much energy completing written
evaluations, the broker must demonstrate, nevertheless, methods for assuring that the
terms of the OJT contract are being met. Probe especially for how the quality of training
is assured.
D. How do you determine that working conditions meet the standards and requirements for
WIA OJT requirements?
This question is directed to the issue of employer compliance with the various WIA
requirements for OJT contracts. How does the broker orient the employer to these
requirements?
92
CONCLUSIONS:
Does the broker demonstrate procedures and activities sufficient to assure that the terms
of the OJT contract are being met? How would you assess the quality and nature of the
contacts between the broker and the employer? The participant?
4. TRAINING OUTCOMES AND OTHER PARTICIPANT BENEFITS
A. Are employers honoring their commitments to retain participants at the wages agreed
upon?
Outcomes in the form of employment is the chief benefit expected from the training.
Evidence that this is probable may be past experience (or current placements if the
contract is far enough along). Are placement rates likely to meet contract performance
goals?
B. What are the typical wages and fringe benefits received by those participants enrolled in
OJT thus far?
Are participants being paid wages comparable to other new hires? If they are paid less,
how is it justified? Do they receive fringe benefits in the form of FICA, workman's
compensation, health insurance, sick and vacation leave with pay, bonus, etc., as other
employees do? If not, how is it justified? Of those participants retained by the
employer, have they received raises and/or up-grades on the same basis as other
employees?
CONCLUSIONS:
Is there sufficient information from the responses above to expect that participants
enrolled in OJT will retain employment with their employers on a permanent basis?
Are the wages and fringe benefits equal to or greater than what was earned by the
participants in prior employment?
93
5. ADEQUACY OF THE OJT CONTRACT
A. Who is responsible for reviewing and approving OJT contracts before they are submitted
for signatures? How do you assure that each contract contains the stipulations and
provisions mandated by the WIB?
This question examines the process by which OJT contracts are written. It is advisable
that at least one person other than the signatory parties review the contract for
compliance to LWIA policies and requirements. Are there checks and balances to assure
that OJT contracts meet uniform requirements?
B. How are the training plans or outlines developed? Are they incorporated into the
contract with the full knowledge of the employer?
The details of the training procedures are key to the value of the OJT contract as a
training mode. Are they specific and clear? Are there measurable objectives so that
progress toward training objectives can be evaluated?
C. How is the employer and participant involved in the development of the contract and the
training outline? How does the employee responsible for the actual training become
aware of the contents of the training outline?
Explore how the training plan becomes actualized to learn whether the training
outline is written to serve as a meaningful guide to the training as opposed to a
perfunctory exercise to comply with LWIA policy. How is the length of training
determined. Are the DOT and SVP codes used?
94
D. How is the length of training, the base for reimbursement and the reimbursement rate
determined for OJT contracts? Are any OJT contracts reimbursed at a rate other than 50
percent? If so, what are the circumstances that explain this?
This question is to determine whether the responsible person is close enough to the
participants and the training to be able to justify costs included in the reimbursement rate
in terms of their necessity or reasonableness. Does he/she confer with other staff or the
employer on these matters?
CONCLUSIONS:
Is the process described above adequate to ensure the development of an OJT contract
that provides quality training? Does it provide for the inclusion of all WIA requirements
for OJT contracts and an effective training outline? Are all parties to the contract
contributors to the development of the contract? Is the reimbursement base, length of
training and reimbursement rate correct and in accordance with the broker's contract with
the WIB?
95
6. OJT CONTRACTS REVIEW
INSTRUCTIONS:
If the broker is responsible for developing or supervising the OJT contracts, it is necessary to
review the contracts for their compliance to WIB policies, WIA requirements and cost
limitations. Summarize the results below by indicating the percent of contracts reviewed that
met the following requirements:
Identifies occupation and ONET code for which training is being provided
Training outline is incorporated in the contract and contains a schedule with
appropriate time periods for learning each skill
Specifies wages to be paid and total hours to be completed
Identifies reporting and payment procedures in contract
Sets out beginning and ending dates of the contract
Provides for signatures of participant, employer and WIB prior to beginning date
of contract
Reimbursement rates to the employer is no more than 50 percent as documented
by employer payment records
Contains standard general provisions required by WIB and State for adherence to
all WIA provisions as they relate to OJT contracts
CONCLUSIONS:
96
CHECKLIST FOR STANDARD PROVISIONS
Note if any of the following standard provisions are absent or there is some question about the
adequacy of the provision. Add any other provisions required by the WIB or State for OJT contracts.
Advise the administrative entity if provisions are missing or inadequate.


Trainees will be covered by Workers' Compensation, or, if not applicable, alternative
equivalent. {Sect. 639.22 (a)}

Displacement of currently employed workers or infringement of promotional
opportunities is prohibited. {Sect. 143(b)}

Contains provisions that cover liability, sanctions & debt repayments{Sect. 164(d)}

Records must be retained by the contractor, including time and attendance and
payroll records.

Contains provision regarding contract modification, handling of disputes and
termination, including termination for the convenience of the government.

Provides for maintenance, retention and access to pertinent records by staff from the
LWIA/SSA, State and U.S. Department of Labor. {sect. 163 and 165}

Training will be in accordance with laws and regulations governing safety and health
standards applicable to the worksite. {Sect. 143 (a) (b)}

Describe employer's commitment to retain participant(s) at the end of training.

Requires contractor comply with all applicable business licensing, taxation, and
insurance requirements.

Employer will comply with Workforce Investment Act and regulations.

Contains provision for nondiscrimination and prohibition against sectarian activities.
{Sect. 167}

Contains provision for union concurrence/comment (as required in Sect. 143(b)(2)
and 143(c).

Provides compensation of the participant at the highest of the Federal, State and local
minimum wage or the prevailing wage rate of similarly situated employees. {Sect.
142}
97
WORKSHEET NO. 1
Employer
Participant Name
SSN (Last 4 Digits)
Occupation ONET
Contract No.
Hourly Wage
contract
Reimb.
Dates
contract
reimb
Training Hours
contract
reimb.
Signatures
yes
no
Reimbursement
Rate
yes
no
Training
Outline
yes
no
WIA
Provisions
Yes
no
97
OJT EMPLOYER INTERVIEW GUIDE
INSTRUCTIONS:
The following questions may be directed to either the personnel officer, supervisor, or CEO, if a
small company. Upon introducing yourself to the employer's representative, explain that the
purpose of the interview is to conduct a review of the OJT activities to assess their effectiveness
and general compliance to contract requirements. Quickly review the subjects to be covered so
that the employer can determine appropriate respondents and arrange for their availability. This
can be done by telephone prior to the site visit. Also not that some of the information called for
in this interview may be available in the contract or other records available to you. If so,
complete these sections prior to the interview and use the interview to confirm and clarify those
points. Questions that refer to the participant may have to be repeated if this contract covers
more than one participant.
Individual(s) Interviewed:
1. PROGRAM OVERVIEW
A. How did you become involved with the WIA/OJT Program? Are you familiar with the
OJT Contract between your company and the service provider/ WIB?
The purpose of this question is to learn how well acquainted the respondent is with
WIA's OJT program in general and this contract in particular. If the respondent
expresses a long history or good orientation to the OJT program, less time must be spent
explaining terms or why certain areas of inquiry are necessary.
B. How were you informed about the program requirements? If you have questions, from
whom do you get assistance?
These questions reflect on the orientation given the employer and to what extent this has
been passed down to the person responsible for the training. It is possible that the
employee responsible for the participant's training has never seen the OJT contract. If
the respondent learned from other employees rather than directly from the OJT contract,
how did the employer's other staff learned about program rules. Does it appear that they
are informed about OJT contract requirements? You may wish to briefly review the
main points of the contract at this point to confirm that the employer has an adequate
knowledge of the contract's requirements.
99
C. How was the length of the training period and participant wages determined?
This question is intended to assess the rationale for the primary contract provisions.
How does the duration of training compare with the DOT or LWIA's policies for
determining length of training? Look for explanations for any variance. Are participant
wages comparable for other trainees in similar occupations? How was the rate of
reimbursement determined? Was it negotiated with the employer or was it simply set at
50%?
D. Does the training outline in the contract accurately state the skills necessary for the job?
Review the training outline or the notes taken during the desk review with the
respondent. Does the employer have a copy of the training plan? Does it appear that the
information contained therein is accurate and being used to guide the training?
E. Does the training being provided OJT trainees differ from that given your regular hires in
similar positions? If so, in what way?
Probe for information suggesting that OJT trainees get only the same training given hires
with comparable experience and skills. Look for justification for the OJT support. Give
the respondent every opportunity to show why OJT training is necessary for the
participants.
100
CONCLUSIONS:
Compare the responses from the interview thus far with the information from the desk
review. Does the contract as written and/or implemented constitute a good OJT
opportunity for WIA participants?
2.
APPROPRIATENESS OF TRAINING FOR PARTICIPANTS
A. How do you go about getting WIA trainees?
Are trainees recruited by the employer? "Reverse referrals" are not prohibited, but are
cause for questioning whether the OJT contract is serving the interests of the employer
rather than the participant.
B. Is there evidence that the participant is making satisfactory progress in learning the new
occupation?
This would indicate that there is a good match between participant interests and
capabilities and the training. How is progress measured? How is the participant rated in
terms of the employer's standards for learning new skills, work habits, and general
productivity? Are records of progress being documented? Review the training plan.
CONCLUSIONS:
Is there sufficient reason to believe that the training is appropriate for the participants
from the information above? Does it appear that the participant is suited to the
occupation and working conditions represented by the training? Does the ISS support
this type of training?
D. If basic education or classroom training in the occupation is needed, how is it
coordinated with the OJT?
101
If basic education or any type of classroom training is included in the training, it should
be well coordinated with the occupation training done on the job.
E. How is the instruction provided and who is the person most responsible for training the
participant?
Is the training interactive? Is the person responsible for the training knowledgeable in
the occupation? What position does he or she hold in the organization? Does the trainee
receive sufficient attention from the trainer? Use this question to get the respondent to
describe the training, including the instructional methods, in his or her own words. Does
this description match the description in the contract?
F. How do trainees learn how well they are learning?
Feedback is critical to learning and may take the form of work evaluations, comments by
the supervisor, peer evaluations, or practice where they must perform a task to a set
standard. Identify the form of feedback used and determine if it is sufficiently structured
and regular in occurrence to promote learning.
CONCLUSIONS:
Determine if this training serves the general aims of preparing participants for
employment as described in the contract. Is it reasonable to expect that the training will
be effective in achieving the skill levels required for this job. Make note of points that
you feel need further confirmation from other data sources. Clarify any confusing
responses before continuing with the next section.
102
3. TRAINING OUTCOMES AND OTHER PARTICIPANT BENEFITS
A. To what extent are you committed to retaining this trainee as a permanent employee after
the OJT program is completed? What conditions must be met?
Outcomes in the form of employment is the chief benefit expected from the training.
Evidence that this is probable may be past experience with other OJT placements.
B. What other benefits do participants receive as a result as a result of their participation in
training or their completion of training? Do participants receive the same benefits as
regular employees?
At a minimum, normal fringe benefits include FICA, workman's compensation, health
insurance, vacation and sick leave. In addition, participants may also be eligible for
bonuses, pay increases, certificates, or other benefits. You may include improved selfesteem or some other behavioral change if you are sufficiently convinced that it is an
intangible but important benefit.
CONCLUSIONS:
Is there sufficient information form the responses above to expect that participants with
this training will be retained in full time permanent employment as described in the
contract? If not, why? Does this constitute a violation of the contract or a breech of
faith? Are participant benefits equal to or better than the contract requires? Is any
problem correctable?
4. COMPLIANCE WITH CONTRACT GENERAL PROVISIONS
A. OJT contracts have a number of general requirements established by state or federal law.
Are you familiar with these requirements contained in your contract? Were you given
an orientation to them by the case manager?
Refer to your desk review and note what general provisions are applicable to this
contract. You may want to review them quickly to refresh the respondent's memory.
103
The point of this question is to determine if the employer has a good working knowledge
of these requirements. Reimbursement of cost related to the OJT contract is one of the
most important contract provisions to be reviewed at this time. Comments on general
compliance provisions:
CONCLUSIONS:
Are you satisfied that the employer is aware of and is in compliance with contract
provisions? Is the reimbursement base and rate as described by the employer consistent
with the terms of the contract?
104
OJT TRAINEE INTERVIEW TOPIC GUIDE
INSTRUCTIONS
The primary objective is to obtain the trainee's perception of his/her training and to verify key
points of information obtained during the employer interview. Confidentiality should be assured
to trainee and adverse comments should not be shared with the employer if it might appear that a
trainee was the source.
Trainees Interviewed:
A. How did you find out about the job here? What did you do before you could begin onthe-job training?
This question establishes how the trainee came into contact with the employer and the
procedures involved in the assessment and referral process. Look for confirmation that
the screening and/or assessment process was followed and that the trainee was not
current employee simply sent to the LWIA/Service Provider for referral.
B. Does the job fit your interest? Does it take into account your education and past
experience and training?
Does it appear that the trainee is an appropriate match to the job?
C. Describe the training you are getting? Does the training match the training outline
approved by you at the beginning of your training?
Look for confirmation that the participant was aware of the type of training to be
provided and that the training is effective in teaching the desired work.
D. Who is the person most responsible for your training? What methods does he or she
use to teach the necessary skills?
105
The purpose of this question is to confirm that the approach described in the contract
is in effect. What is his or her opinion of the employer's implementation of the
training outline?
E. What is your hourly wage when you started $
receive?
? What fringe benefits do you
This question confirms wages being paid and benefits received. Is this what the
trainee expected? In what way is it different?
F. Do you have any comments about the working conditions?
Look for indications that the employer is not in compliance with requirements
governing work place conditions.
G. Who is your WIA counselor or contact person?
often do you see him/her?
How
Determine whether the participant is receiving on-going contact from the responsible
party.
H. Have you ever worked less than
hours a week? How is your time on the job
recorded? Have you ever had a complaint about your time? Have you ever signed a
blank time sheet?
This question confirms that time sheets are maintained and are the basis for paychecks.
Time on the job should comply with the OJT contract.
106
CONCLUSIONS:
Does the participant agree with the choice of occupation, the choice of training site and
the training plan? Based on the participant's comments, and perspective, how should the
training be changed to make it more effective? Do any of the participant's remarks
suggest a violation of the OJT contract? Is there any indication of a need for counseling?
Does the participant believe that training will lead to a job that he or she wants?
107
EXHIBIT "H"
PRE-AWARD SURVEY
Proposed Service Provider/Vendor
Review Date
Activities:
When the Workforce Investment Board is considering a new service provider/vendor, a site visit
may be necessary. This checklist is intended to be a guide in conducting the visit/interview.
A. Describe prior WIA or other projects which provide an indicator of performance attainment. Is
this the first proposal submitted? (n/a )
B. Describe past experiences in performing this or a similar activity.
C. If any contracts have been terminated or not re-awarded within the past two years, please
explain. (n/a )
D. Describe the qualifications of key staff who will be working under the proposed contract.
108
E. Describe the accounting system currently being used. Can the agency adequately account and
safeguard WIA funds?
(yes
) (no
)
F. What exceptions were noted in the past two years audits? Was corrective action necessary?
(yes
) (no
)
G. Does the agency have an AA/EEO plan that meets federal and state AA/EEO guidelines?
(yes
) (no
) Retain a copy.
If no, when is the proposed plan due to be submitted to the WIB?
H. Does the proposed activity duplicate facilities or services available in the area which funded by
federal, state or local entities. (yes
) (no
)
I. Are facilities and equipment adequate for the type of activity being proposed?
(yes
) (no
)
Comments:
J. Is proposed activity an occupation skills training program? (yes
) (no
)
1. If yes, does the proposed level of skills meet the guidelines established by the WIB?
(yes
) (no
)
2. Is the school an accredited post-secondary institution? (yes____) (no___)
K. Can the agency provide a certificate of insurance for comprehensive general public liability
insurance with combines single limit coverage of at least $500,000 and worker's compensation
insurance--with the County of Tulare and Workforce Investment Board specified as additional
insured? (yes
) (no
)
109
If yes, when will the binder be sent to the WIB?
L. Does the agency have adequate personal property insurance coverage for WIA property when in
their possession? (yes
) (no
)
1. If no, when will the agency provide the coverage binder to the WIB?
M. Are any officers or the agency subject to federal debarment? (yes
and document (i.e., Attach any document or directive.)
N. Are any lobbying activities conducted by the agency? (yes
) (no
) (no
) Describe
)
If so, can assurance be given that, if contracted is entered into, WIA funds will only be used
for allowable purposes? (yes
) (no
)
REFERENCES:
1. LWIA
Performance:
Cost per Participant:
Status:
Vendor ( )
Service Provider ( )
One-Stop Operator ( )
Service Provider ( )
One-Stop Operator ( )
Service Provider ( )
One-Stop Operator ( )
Comments/Contact:
2. LWIA
Performance:
Cost per Participant:
Status:
Vendor ( )
Comments/Contact:
3. LWIA
Performance:
Cost per Participant:
Status:
Vendor ( )
Comments/Contact:
110
EXHIBIT “I”
ONE-STOP SYSTEM MONITORING GUIDE
Purpose: To review compliance with one-stop operational guidance. This section should integrate
the prior sections to complete a comprehensive review of one-stop operations and systems.
I. Customer Access
Key Elements
Current Expectations
Met
(yes/no)
A. Mandatory Customers
1. All job, education, and training seekers
2. All employers
All job, education, and training seekers
must have access to the core mandated
services.
All employers must have access to the
core mandated services.
Attach a copy of the procedures that will assure all job, education and training seekers, as well as
employers, have access to the one-stop center/system.
B Physical Locations
1. Central full-service one-stop center
Each system must have at least one
physical location that will act as a full
service hop site. Facility must meet
Americans with Disabilities Act
requirements.
Attach a floor plan of the physical location(s) including an address and telephone number.
Document a walking tour of the one-stop centers.
2. Other local partner locations
Each center/system may be linked to
other partner locations.
Attach a list of other partner locations. Demonstrate how partners communicate with the hub.
Provide website address if appropriate.
Comments:
111
II. Partners Included
Key Elements
A. Mandated Partners
Met
(yes/no)
Current Expectations
Each comprehensive center must have representation of the
mandatory programs with partners located in the local area.
List the name of the cognizant agency, contact and the telephone number
Agency
Contact
1. Programs authorized under the
Wagner-Peyser Act
2. Workforce Investment Act
Title I funded Adult Programs
3. Workforce Investment Act
Title I funded Youth Programs
4. Workforce Investment Act
Title I funded Dislocated Worker
Programs
5. Senior Community Service
Employment Activities
6. Unemployment Insurance
7. National Veterans
Employment Service
8. Native American Services
9. Housing and Urban
Development Employment &
Training Programs
10. NAFTA/TAA Programs
11. Rehabilitation Services
12. Welfare-to-Work Formula
Programs
13. Small Business Development
Centers
14. Postsecondary Vocational
Education & Applied Technology
Act Programs
15. Job Corps funded programs
located in the Workforce
Investment Area
16. Employment & Training
Services carried out under
Community Services Block
Grants
17. Nationally funded Migrant &
Seasonal Farm Worker Programs
18. National Veterans
Employment Service Programs
19. Technical Assistance to
Needy Families Act Programs
112
Phone #
II. Partners Included
Key Elements
B. Other Partners
Current Expectations
Each center has other important partners
Attach a list of other partners of the one-stop center/system including the contact person and
telephone number.
C. One-Stop Center/System
staffing
Partner staff will contribute to overall staffing
of the centers/system.
Attach a list of core staff and their duties.
Comments:
113
Met
(yes/no)
III. Core Services for all Job, Education, and Training Seekers
Key Elements
Current Expectations
A. Mandatory Core Services
1. Eligibility determination for
available funding sources.
Met
(yes/no)
Services that will be available universally and not
require an eligibility determination.
A process that determines the eligibility of customers
to participate in the programs offered by mandated
and other partners of the local Workforce Investment
System including the establishment of eligibility for
welfare-to-work activities and programs of financial
assistance for employment and training programs.
Attach a copy of the Procedures for Determining Eligibility
2. Directory of Services
A directory that provides a descriptive “menu” of
services available at the one-stop center/system.
Attach a copy of the Directory of Services
Provision of information on how to access supportive
3. Resource Referral Services
service agencies available in the local one-stop
center/system area.
Attach a list of training dates and staff who were trained to provide resource and referral services.
Attach a list of available supportive service agencies.
4. Job Service/Employment
Assistance to customers in locating, applying for, and
Information
obtaining a job. This may include but is not limited
to:
a.
b.
c.
d.
e.
f.
g.
h.
job-finding skills
resume writing assistance
work search plan assistance
job development
job referral and placement services
job clubs
job search workshops
vocational exploration
Attach workshop curriculum, calendar, or other written material and a list of software that will
demonstrate the availability of the job service/employment information.
a. Provision of occupational wage and
supply/demand information
b. Identification of geographic and occupational
areas of potential growth or decline.
5. Labor Market Information
c. Assessment of the affects of such growth or
decline on individuals, industries, and
communities.
d. Individual career information
Attach a list of available information that addresses each of the Labor Market Information areas.
114
III. Core Services for all Job, Education, and Training Seekers
Key Elements
Current Expectations
Met
(yes/no)
A. Mandatory Core Services
6. Intake
a. An orientation that explains the programs
available through referral from a one-stop
center including information about the
program, its services and the availability of
those services.
b.
A procedure to provide for intake of the
programs of the mandated partners at the
one-stop center.
Attach the orientation agenda and the referral procedures.
7. Outreach to disadvantaged
populations
A plan that describes outreach strategies for
disadvantaged populations.
Attach the plan for outreach to disadvantaged populations.
The process used to determine the employability
needs of individuals as well as their progress towards
achievement of redefined occupational goals.
Attach a list of available assessment tools and workshops. Attach a calendar to demonstrate its
availability.
The provision of information to all potential
9. Integrated Services
customers concerning available career resources and
Orientation
labor market information.
8. Assessment
Attach an outline of the integrated services orientation and a calendar to demonstrate its availability.
10. Initial Screening and
Coordination System for the
next level
A method by which an applicant is made aware of
other programs or services and receives appropriate
assistance in being referred.
Attach the procedure for coordination of the system.
A service that allows applicants to file their initial
claims for unemployment insurance benefits.
Attach the procedure that explains how applicants will be able to file for unemployment insurance
benefits at the one-stop center(s).
Information on local area performance as well as the
performance of eligible training providers that can be
12. Performance Information
used by one-stop customers to make informed
choices.
11. Unemployment Insurance
Attach documents that are used to provide performance information to customers.
13. Other
A one-stop center/system is a reflection of the
community that sponsors it. Other services reflecting
the needs of that community many be provider.
Attach a list of any other job, education, and training services provided or linkages available.
115
IV. Core Services for Employers
Key Elements
Current Expectations
Met
(yes/no)
A. Mandatory Core Services
1. Directory of Services
A directory that provides a descriptive “menu” of
services available at the one-stop center(s).
Attach a copy of the Employer Directory of Services
2. Resource Referral Services
Provision of information on how to access business
assistance/incentives available in the local one-stop
center(s)
Attach a list of assistance and incentives as well as a procedure to connect businesses with the resources.
A service that will allow employers to fill job
openings by accessing a “talent bank” that matches
3. Job Applicant Data Bank
job seekers’ skills and/or occupations to the
employer’s specifications.
Attach a list of staff trained in the use of America’s Job Bank, America’s Talent Bank, America’s
Learning Exchange and/or other local/state job system and procedures for assuring that employ know
how to use these internet based systems.
Occupational wage, supply/demand information,
geographic/occupational areas of potential
4. Labor Market Information
growth/decline, assessment of the effects of
growth/decline on individuals, industries,
communities, and employers.
Attach a list of information that addresses each area.
5. Economic Development
(Basic Incentives)
Information on basic incentives (e.g., tax credits,
enterprise zones, or other incentives) as well as local
planning and zoning commissions or boards,
community development agencies, and other entities
responsible for regulating, promoting, or assisting in
local economic development.
Attach a list of information that addresses each area.
6. Outreach to Employers
A plan that describes outreach strategies to
employers.
Attach a plan for outreach to employers.
7. Other Services
A one-stop center is a reflection of the community
that sponsors it. Other services reflecting the needs
of that community may be provided.
Attach a list of other employer services available at the one-stop center or in the one-stop system.
116
V. Service Integration
Key Elements
A. Cross-Trained Staff
Current Expectations
Met
(yes/no)
Capacity Building provided to staff and managers.
Attach a list of cross training accomplished including the names and titles of the staff who were trained.
Attach a calendar of future training sessions with an outline of the materials to be provided and
identification of the staff to be trained.
B. Shared Job Development
Job Development is coordinated to ensure a system is
in place to work with employers
Attach a description of the job development coordination occurring at the one-stop center or in the
system.
VI. Community Support
Key Elements
Current Expectations
Met
(yes/no)
A. Local Elected Official
Local Elected Officials endorse the plan for the onesupport
stop center/system.
Attach a copy of the Local Elected Officials letter of support and/or Memorandum of Understanding
between the Workforce Investment Board and the one-stop center/system operator, or other instrument of
agreement if applicable.
B. Partner Identification
All partners within the one-stop center/system are
identified.
Attach copies of a letter of support from each partner and/or Memorandum of Understanding between the
partner and the one-stop center/system operator describing the role and responsibility of each party.
C. Name of the one-stop
Each one-stop center/system must have a name that
center/system
reflects the community partnership.
Name:
Show how the name has been used for location identification, letterhead, signage, marketing, etc.
Comments – Discussion:
117
One-Stop Partner Questionnaire
(To be used in conjunction with Section II, C.)
Partner Identifying Data:
Agency Name:
Address:
Partner Classification:
Mandated Partner
Partner Type:
private-for-profit
Other Partner
private-non-profit
public
Attach a list of staff currently providing services in the local one-stop environment.
Instructions:
The following questions should be answered by the person most knowledgable with the one-stop’s
operations and the MOU. Answer N/A if the question does not apply, “Unknown” if you have no
knowledge of the matter.
1.
Can you demonstrate that your agency is:
a. Providing all services as agreed in the MOU?
‫ٱ‬
Yes
b. Providing all personnel services as agreed in the MOU?
‫ٱ‬
No
‫ٱ‬
Yes
c. Cash or in-kind contribution (MOU/Cost Sharing Agreement)?
‫ٱ‬
‫ٱ‬
No
Yes
‫ٱ‬
No
d. Other:
2.
Provide documentation of payment of proportional share paid toward the one-stops’
operation and services.
3.
Do you participate in:
a. A site management committee/council?
‫ٱ‬
b. A committee/council of the WIB?
Yes
‫ٱ‬
Yes
‫ٱ‬
No
4.
Do staff participate in center/system cross-training?
5.
Are Core services provided?
6.
A one-stop center/system is performance driven.
a. Are performance goals set?
‫ٱ‬
‫ٱ‬
Yes
Yes
‫ٱ‬
‫ٱ‬
‫ٱ‬
No
‫ٱ‬
Yes
‫ٱ‬
No
No
No
b. How is your performance against the goals? (Attach performance data and period.)
118
EXHIBIT "J"
EQUAL EMPLOYMENT OPPORTUNITY
Service Provider:
Interviewee:
Date:
Monitor:
The following questions should be directed to the person that has primary responsibility for the
agency EEO functions.
A. Do you have an affirmative action plan/grievance procedure on file? (yes
1. Have you adopted the WIB's AAP/Grievance Procedure? (yes
) (no
) (no
)
)
2. If no, has your plan been submitted and approved by WIB Administration?
(yes___) (no
) If no, retain copy for review and include in main file at WIB.
This question seeks to confirm that the agency is in compliance with WIB EED policy and
contracting requirements.
B. Are EEO posters prominently displayed in a location frequented by WIA
applicants/participants? (yes
) (no
) Where?
The purpose of this question is to obtain information relative to dissemination of information to
staff and the client population.
C. Have you received any written complaints within the past program year? (yes
yes,
1. What type(s) of complaint(s)?
3. How was/were the complaint(s) resolved?
119
) (no___) If
This question is intended to insure that complaints are handled appropriately and
resolutions are reached as required under governing directives.
Conclusion:
D. Are facilities accessible to disabled persons seeking services under WIA? (yes
(no )
1. If not, are alternative provisions made? (yes
) (no
)
) What provisions?
2. Insure that the agency is providing services to disabled persons or that appropriate referrals
are being made to other agencies. Comments:
E. Are services provided on an equitable basis among substantial segments of the population within
the service area? (yes
) (no )
This question seeks to insure that the agency outreach and recruitment efforts are effective and
that service levels are representative of the LWIA eligible population
F. Is your hiring process accessible to qualified disabled applicants? (yes
accommodations are made for this group?
) (no
) What
This question elicits information relative to the agencies hiring practices. Insure that persons
with disabilities are not artificially excluded. Comments:
120
G. For youth Service Providers that provide intake services: Do clients receive a copy of the LWIA
grievance procedure at the time of intake? (yes
) (no
) Verify in File Review.
This question is intended to insure that applicants are informed of EEO policy and their
rights as an applicant/participant.
CONCLUSION:
This question elicits information relative to the agencies hiring practice
CHECKLIST:
EEO Posters (location_______________________)
EEO/Grievance Procedure
Acceptable facility for Section 504/American with Disabilities Act (ADA)
Compliance
Complete "504" checklist annually (View.)
121
EXHIBIT "K"
SERVICE PROVIDER
MONITORING
EXIT INTERVIEW CHECKLIST
Service Provider
Contract #
PY
( ) 1st Round Monitoring
( ) 2nd Round Monitoring
Date of Monitoring Visit
Date Exit Interview Completed
Meeting
Telephone
Monitor
Program
Fiscal
1. ( ) Recommendation(s) discussed
2. ( ) Finding (s) discussed
Itemize findings and recommendations
Is technical assistance advised/required?
( )
Follow-up
Date
122
MONITORING RESOURCES
SAMPLE SIZE
SAMPLE SIZE - - An adequate sample will generally be between 5-25% of the population. As a
rule, a small proportion of a large population and a large proportion of a small population will
provide an adequate starting point for determining sample size. At a minimum, the following sample
size requirements should be met.
Population Size
100 or under
101 – 500
501 – 1000
1001 or more
Minimum Sample Size
20 % of population
15% of population
10% of population
5% of population
In addition to population size, there are other factors to consider in determining sample size. These
factors are listed below. Sample size should be increased beyond the minimums listed above to
address conditions under each factor.
1. Eligibility Criteria Complexity – a larger sample size is indicated in programs with
multi-faceted or complex eligibility criteria.
2. Suggestion of Irregularity – a larger sample is indicated if there is a suggestion or clue
that ineligible individuals have been enrolled.
3. Relative Risk – a larger sample may be indicated if the intake/enrollment/verification
functions are not under the direct supervision of the Tulare County Workforce
Investment Department, or if the functions are being performed by new staff members or
subcontractors.
RANDOM SAMPLE SELECTION
SAMPLE METHODS – A random sample can be drawn by using many methods. Management may
choose any method that serves local needs. Three sampling techniques are described below:
Simple, Cluster and Systematic. The Cluster and Systematic methods require the use of a Table of
Random Numbers (Attachment), which is explained below. Automated spreadsheets also provide
random number generators and may be used to execute the Cluster and Systematic methods.
Simple Random Sampling is analogous to picking individual participant cases out of a hat.
This technique becomes impractical as the size of the population increases. The entire
population must be available for each selection of a sample case in order to avoid
unintentionally biasing the sample.
Cluster Sampling may be used in situations when cases fall into natural groupings such as
time frames, file drawers, participant teams, etc. Each cluster is assigned a number (from
one to ‘x’) depending on the number of clusters in the population. Then, using a Table of
Random Numbers (explained below) clusters are selected to fill the sample size requirement.
Systematic Random Sampling is often used to select a sample from a large population.
123
Individual cases are assigned a number (from one to ‘x’) depending on the number of cases
in the population. Then, using the Table, cases are selected at random to be part of the
potential sample. The number of cases in the potential sample is reduced by choosing every
‘nth’ case, for example, every tenth case. When the number of cases selected equals the
sample size, the sample selection is complete.
To use a Table of Random Numbers:
Give each case in the population a number starting at one and continuing until all cases are
numbered.
Note the exact number of cases, and the number of digits comprising the figure representing
the total number of cases in the population.
Decide on the sample size.
Select a starting point on the Table of Random Numbers, e.g. the upper left had corner. To
select sample cases, use the number of digits that represent the total population. Move
across or down the table and select the first number which is within your population. If the
first number exceeds your population size, skip it and progress down or across the columns
until you find a number that is within your population. When enough numbers have been
selected to fill the predetermined sample size, remove the individual cases from the
population; this is your sample.
As an example of use of the attached Table of Random Numbers, consider the need to sample a
population of 535 cases. The individual choosing the sample would assign a number from 1 to 535
to each case. Note that there are three digits in the number 535. The numbers on the Table of
Random Numbers are comprised of five digits. The sample selection may be based upon either the
first [three] or the last three digits in the five-digit number, as long as the choice of the digits is
consistent through the entire sample. For this example, use the digits in the ones, tens and hundreds
columns. Starting from the upper left hand corner, and moving down, the individual selecting the
sample would choose case number 097, skip number 542 (because it is too large for the population),
identify cases 422, 019 and 065 as part of the sample. This selection process will continue until the
number of cases identified satisfies the sample size requirement.
124
Attachment
A BRIEF TABLE OF RANDOM NUMBERS
10097
37542
08422
99019
12807
85017
16719
65842
79875
93640
84532
82789
27672
20684
39160
13618
69041
82186
39187
41453
23157
05545
14871
38976
97312
86952
44109
22115
94324
41548
02348
05403
86529
43204
93437
76520
64894
19645
09276
80157
66069
31060
85269
63573
73796
19478
65119
70322
58133
44655
70086
26486
21529
41278
81255
71265
47353
48233
11697
31133
11742
43361
93806
49540
36768
18226
99436
32584
61777
60452
29004
42753
21828
67954
38537
34072
45571
02051
05325
03529
98520
11805
83452
88685
99594
65481
80124
74350
69916
09893
02295
85035
01197
97907
63268
52841
53722
11434
62375
28337
13487
34881
86935
19078
96905
39684
71399
34908
99292
20923
98662
35587
28021
40646
28797
67411
10916
62171
21177
87929
07092
43310
61570
31352
57048
09243
07959
93732
72721
61020
44673
48897
23350
48625
46359
56092
21225
26958
66995
62841
61303
48493
65710
44369
74594
84369
13018
02400
07289
31374
14905
39808
06288
86507
87517
17468
17727
77402
66252
14225
91499
80336
44104
12550
63606
61196
15474
94557
42481
23523
38631
49172
08232
02844
40387
80240
44910
33663
86430
31379
79430
16332
57327
15026
65406
44177
99321
86347
19102
68588
62421
44670
34679
32439
37920
51171
72173
00926
67420
81675
97959
35089
62235
58537
08709
08723
56239
49115
41976
15694
67422
17691
79655
48274
60623
39323
04595
34823
76559
43438
69992
89246
81336
81330
02237
05798
10836
51770
24358
36879
68479
26940
85157
11100
16505
26457
95270
67897
97344
73208
04493
00549
35963
59808
46058
98086
03345
80951
79752
18633
32533
04805
68953
02529
99970
17767
05431
99634
40200
67348
14523
94598
81949
73742
49329
52494
97654
15307
08391
95236
24826
16232
00406
49140
32537
75246
64051
26898
45427
01390
32179
69234
19565
45155
94864
74029
54178
11664
18324
69074
74717
10805
77602
32135
45753
17674
35635
99817
26803
20505
90446
45266
28673
16513
78317
00597
61406
41430
14938
31994
45240
41941
96382
71961
98145
87379
20117
01758
19476
36168
Sources Leonard K. Kazmier, Statistical Analysis for Business and Economics (2 nd Ed.), copyright
125
1973 by McGraw-Hill, Inc.
GLOSSARY OF TERMS
UTILIZED BY THE MONITORING UNIT
Desk Review: A review of information available in the TCWIB contract file, fiscal files, and management
information system.
On-site or Field Review: A review conducted at the subrecipient or contractors physical location.
TCWIB General Files: The master files located at the Tulare County Workforce Investment Board, Inc.
Administrative Offices.
Audit Log: A list of audit findings developed by reviewing an agency’s annual audits. This is a requirement
by state directive.
Monitoring Log: A list of findings and the resolution of findings.
Risk Assessment: In accordance with WIBD-01-XX, the method by which the level and intensity of
monitoring is determined.
EDD Compliance Review Division: The monitoring unit at the state level. Also called CRD.
Draft Transmittal Letter: The Draft Report initially sent to a subrecipient within 30 days of completion of the
on-site monitoring visit.
Final Report: The report, in letter form, which informs the subrecipient of the TCWIB’s conclusions
regarding any findings which may have been transmitted in the Draft Transmittal/Draft Report Letter.
Observation: The condition or deficiency which has resulted in a monitoring finding and/or concern.
Random Sample: An unbiased selection of data to be reviewed to verify its accuracy.
Requirement: The requirement of the legislation, regulation, and/or directive (state or local) which the
condition or deficiency violates.
Recommendation: The action(s) which will correct the condition or deficiency which has resulted in a
finding or concern. Recommendations are developed in consultation with all responsible TCWID Analysts
and TCWIB/WID management.
Service Provider Response: The reply from a service provide which either refutes the condition or deficiency;
or corrects the condition or deficiency.
TCWIB Conclusion: TCWIB’s acceptance or denial of the Service Provider Response. Acceptance is not an
indicator that the finding or concern is closed. The matter may remain open pending follow-up or future
monitoring review.
Follow-up: Service Provider Responses accepted by the TCWIB must be verified in either an additional
review, or in future monitoring visits, as appropriate.
Technical Assistance: Resolution of a concern or finding with guidance or advice from the
appropriate TCWID Program Analyst (programmatic), or the Business Services Officer of the
TCWIB (fiscal/internal control).
126