Section 254.610  Statewide Evaluation of Vocational Education Programs

a)         The State Board of Education will utilize the evaluation system set forth in this Subpart in evaluating vocational education programs administered by eligible recipients.

1)         Each eligible recipient administering a program of vocational education which is supported with vocational education funds will be required to complete a self-assessment of the vocational education programs offered by the recipient.  The self-assessment shall be completed for each vocational program offered by the eligible recipient and shall include evaluations in the areas of program placement, employer satisfaction, student satisfaction, student performance, enrollment trends, labor market justification, and program costs.

2)         The State Board of Education will direct an on-site review and analysis of the eligible recipient's assessment of its program of vocational education. The on-site review will be conducted by a team composed of individuals selected by the State Board of Education from the Board's professional staff and other groups on the basis of their knowledge of and experience in vocational education and employment training, student needs and services, and employment needs in the community and state.

3)         The self-assessment of an eligible recipient and the associated on-site review shall be conducted in accordance with a schedule of evaluations developed annually by the State Board of Education.  The evaluation schedule will be developed such that each eligible recipient will be evaluated once during a four-year period.

b)         The evaluation process is comprised of four phases, a vital signs assessment, an in-depth analysis, a summary report, and a program improvement plan, as follows:

1)         The initial review of a vocational program focuses on the vital signs of that program.  Seven vital signs which address the viability of a program have been identified:  placement, employer satisfaction, student satisfaction, student performance, enrollment, cost containment, and labor market justification.  The collection of information for each vital sign is done through the examination of existing information, conducting surveys, and/or student testing.  Information for each vital sign is then summarized, resulting in a vital sign rating.

2)         Local personnel analyze the graph and determine whether one or more of the vital signs indicates a concern.  If all vital sign ratings are judged acceptable, then no further evaluation is necessary and a program improvement plan is developed.  If one or more of the vital signs indicates an area of concern, a more thorough in-depth analysis, which focuses only on those vital signs which indicate a concern, is conducted.  After the in-depth analysis has been completed, results are summarized and conclusions are drawn.  Local personnel then make a determination of program status.  In most instances, a decision would be made to continue the program while working toward program improvements.  In others, evaluation results may clearly indicate the need to discontinue a program.  A plan to phase out the program should then be developed.

3)         Local personnel develop a summary report covering the activities carried out pursuant to subsection (b)(2) and their results.

4)         The last phase of the evaluation process is the development of a program improvement plan by local personnel, with assistance from State Board of Education personnel, for each continuing program.  If all vital signs are strong, then the program improvement plan may consist of general recommendations for maintaining and improving the program.  If one or more of the vital signs indicates a concern, the program improvement plan is focused on the findings of the in-depth analysis and plans for remediating identified problems.  Local personnel then identify ways in which the program could be improved as well as the methods for doing so and the necessary resources.  This plan is submitted to state personnel for approval and subsequent allocation of resources.  To be approved, the program improvement plan shall identify areas of deficiency, specify remediation activities, list resources utilized for remediation, and identify the person responsible for coordinating the remediation activity.  A verification process will be utilized by the State Board of Education to assist in the approval of the program improvement plan. This process will include a review of the data and findings from the evaluation process and interviews with selected personnel during an on-site visit.  The purpose of these activities is to verify that solutions proposed are related to problems identified and that the data indicate that such problems exist. Technical assistance will be available to local personnel throughout this process.

c)         The vital sign assessment phase of the evaluation process consists of collecting data related to the seven vital signs of the program.  These data will be utilized by local personnel in determining the status of the program. The seven vital signs and their definitions, the methods by which data are collected, and the types of information to be gathered are the following:

1)         Placement:  a count of former program participants who are now either employed, unemployed, in continuing education, in the military, or other, obtained through surveying a sample of former students.  The information gathered identifies the former students' vocational curriculum and current employment, as –

A)        Employed, (at job title),

B)        Unemployed,

C)        Continuing Education,

D)        Military, or

E)        Other.

2)         Employer Satisfaction:  an assessment of employer views related to the technical, employability, and basic skills of employees who have participated in the program, as compared to other employees of similar age and education, obtained through surveying a sample of such employers.  The information gathered indicates whether the employers have employed or supervised students who have participated in this program and the employer's judgment (which may include "no opinion"), on a five-point scale on which "1" means poor and "5" equals excellent, as to –

A)        how well the program prepared students to perform technical skills required on the job such as utilization of equipment, understanding of work-related terminology, etc.,

B)        how well the program prepared students to perform reading, writing, and math skills on the job,

C)        how well the program prepared students with general employability skills (such as work attitudes, being on time to work, receiving supervision, proper dress, etc.,) and,

D)        in general, how program graduate(s) rate as compared to other employees.

3)         Student Satisfaction:  an assessment of current and former student views related to course content, instructional methods, instructor readiness, instructional resources, support services, and preparation for employment, obtained through surveying a sample of current and former students.  The information gathered indicates the students' judgment, on a five-point scale on which "1" means not satisfied and "5" means very satisfied, as to –  

A)        the material covered and level of difficulty in their vocational program,

B)        the lectures, lab experiences, and the group and individual projects used by instructors in their vocational program,

C)        their vocational instructors' knowledge of the field and current employment opportunities and trends in their program,

D)        the equipment, facilities, and materials used in their vocational program,

E)        the guidance, tutoring, and vocational counseling available in their vocational program, and

F)         for former students, to what extent they are satisfied that their training prepared them for a job.

4)         Student Performance:  an assessment of students' skills on state-developed employability skills tests.  The information gathered identifies the mean student score by program and encompasses the following areas –  

A)        identifying employment opportunities,

B)        applying employment seeking skills,

C)        interpreting employment capabilities,

D)        demonstrating appropriate work behavior,

E)        maintaining a safe and healthy environment,

F)         maintaining a businesslike image,

G)        maintaining working relationships with others,

H)        communicating on the job,

I)         adapting to change,

J)         understanding how a business works, and

K)        performing mathematical skills.

5)         Enrollment:  a tabulation of currently enrolled students, as well as those enrolled over the past three to five years, resulting in trend information, obtained through retrieval of enrollment information from existing records. The information gathered identifies the –

A)        curriculum,

B)        year/term, and

C)        number of students enrolled.

6)         Cost Containment:  a calculation of the unit cost for the program, obtained through retrieval of program cost information from existing records. The information gathered identifies the cost per unit of credit to include components such as the following –  

A)        revenue (such as reimbursements, tuition, special projects/grants, services, sale/resale of equipment, supplies and projects), and

B)        expenditures (such as personnel, contractual fees, materials/supplies, equipment, travel, and maintenance/repair).

7)         Labor Market Justification:  Regional, state, and national labor market data will be provided by the State Board of Education in conjunction with the evaluation.  This information, together with such additional information as the applicant may consider necessary, shall be reviewed by the applicant as a basis for determining needs related to its program.

(Source:  Amended at 13 Ill. Reg. 8459, effective May 22, 1989)