U.S. DEPARTMENT OF LABOR
Employment and Training Administration
Washington, D. C. 20210

CLASSIFICATION

WIA/Performance Accountability

CORRESPONDENCE SYMBOL

OWS

ISSUE DATE

September 21, 2000

RESCISSIONS

None

EXPIRATION DATE

Continuing

DIRECTIVE

:

TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 06-00

 

TO

:

ALL STATE WORKFORCE LIAISONS
ALL STATE EMPLOYMENT SECURITY AGENCIES
ALL STATE WORKER ADJUSTMENT LIAISONS
ALL ONE-STOP CAREER CENTER SYSTEM LEADS

 

FROM

:

LENITA JACOBS-SIMMONS
Deputy Assistant Secretary

 

SUBJECT

:

Guidance on Obtaining Customer Satisfaction Results for Participants and Employers Receiving Services Under Title I of the Workforce Investment Act (WIA)

 

  1. Purpose. This guidance letter outlines: 1) the parameters States are to use in conducting State level customer satisfaction surveys, 2) the formula weights to be used in computing performance for the State American Customer Satisfaction Index (ACSI) scores for participants and employers, and 3) the parameters States are to use in capturing customer satisfaction outcomes at the local level for local performance accountability purposes.

  2. References. The Workforce Investment Act of 1998, section 136; 20 CFR part 666; Training and Employment Guidance Letter (TEGL) No. 7-99, March 3, 2000; Federal Register issuance on proposed reporting requirements, 65 Federal Register, pages 17537 through 17540 (Apr. 3, 2000).

  3. Background. Section 136 of WIA specifies core indicators of performance and their application to workforce investment activities in adult, dislocated worker, and youth programs. Four core measures apply separately to the adult, dislocated worker and older youth participants, three measures apply to younger youth, and two measures of customer satisfaction apply across the three funding streams.

    On March 3, 2000, the Department issued in TEGL 7-99 policies on core and customer satisfaction performance measures under Title I of WIA. The Department also published in the April 3, 2000, Federal Register a request for comments on proposed reporting requirements on WIA participation and outcomes. Both issuances outlined the core customer satisfaction questions, survey approach, and scoring methodology for States to use in deriving statewide customer satisfaction outcomes. Since the release of the issuances, the Department has received several inquiries about conducting and reporting the outcomes of customer satisfaction surveys for the State measures as well as requirements for assessing local Workforce Investment Board (WIB) performance. This guidance letter summarizes the parameters States are to use in conducting customer satisfaction surveys at the State level, specifies the weights States are to use in calculating performance, and outlines the parameters States are to use in conducting surveys and reporting results for local WIBs.

    The parameters outlined in this guidance letter represent minimal requirements for States to follow to ensure the survey results are representative of the populations surveyed. States have maximum flexibility in the design and implementation of customer satisfaction measurement approaches that will capture outcomes for local WIBs. Each State is responsible for ensuring compliance with the guidance outlined in this issuance when designing and implementing these approaches.

    Formal instructions for reporting customer satisfaction outcomes to the Department will be released as part of the overall WIA performance accountability reporting system.

  4. Populations for the Surveys. Each State is responsible for obtaining customer satisfaction outcomes for registered participants who exit the program and for employers who received substantial services through One-Stop operations. The universe of participants and the universe of employers for the State level customer satisfaction surveys and for the customer satisfaction surveys used to assess local WIB program performance are described below.

    1. Participant Universe.  WIA Title I-B participants who are exiters as defined in the core measures detailed in TEGL 7-99, who are either Adults, Dislocated Workers, Youth 19-21, or Youth 14-18 will be surveyed. All individuals from all funding streams in an exit cohort are eligible for the survey.

    2. Employer Universe.  Those employers eligible to be surveyed include those who have received service where the service has been completed or, if it is an ongoing service, when a full segment of service has been provided (e.g., after listing an open job order, the employer has received some referrals or if no service, 30 days have elapsed after the initial request). All employers who have received a substantial service involving personal contact with One-Stop staff are eligible to be chosen for inclusion in the survey, excluding those employers who request a brochure or standard mailing, ask a question that is answered with little expenditure of staff time, or use electronic self-services. Examples of services include staff facilitated job orders, customized job training, customized labor market information requests, and on-the-job training activities.

      In some instances, an employer may receive multiple services and have multiple contacts listed for the employer. In this situation, the employer contact receiving the greatest amount of service should be surveyed about the employer's satisfaction. An employer should only be surveyed once during the course of the program year.

      States should build the capacity to track WIA Title I services received by employers into their information and reporting systems policies. This information should include accurate contact data needed to initiate the survey.

  5. Parameters for Conducting Surveys to Obtain State Level Results. As stated in TEGL 7-99, the Department has secured rights to use the ACSI to capture and report statewide customer satisfaction results for each State. Each State is responsible for sampling, administering the surveys, and reporting results to the Department. States may enter into contracts with vendors to provide the actual sampling and interviewing services.

    The ACSI was developed by the National Quality Research Center at the University of Michigan Business School. ACSI is co-sponsored by the American Society for Quality, Claes Fornell International (CFI) Group, and the University of Michigan Business School. The index was developed to provide useful information on quality to complement present measures of the U.S. economy.

    To ensure ACSI results for individuals are collected in a consistent and uniform manner, procedures used by States to obtain participant and employer customer satisfaction information must satisfy the following criteria:

    1. As stated in TEGL 7-99, participant and employer customer satisfaction responses must be collected by telephone interviews. In-person interviews and mail questionnaires may be used only in situations where the individual does not have a telephone or where the person has a hearing impairment. Accommodations should be made to administer surveys and collect results from individuals who are not proficient in English.

    2. Participants should be contacted as soon as possible after the date of exit and no later than 60 days after the date of an exit. For registered individuals who exit the program due to non-participation in services for 90 days, these individuals should be contacted no later than 60 days after the 90 days have elapsed since the last date of service.

      Employers should be contacted as soon as possible after the completion of the service and no later than 60 days after the completion of the service. For employers who listed a job order where no referrals were made, contact should occur 30 to 60 days after a job order was listed.

    3. Interview questions must include the core questions outlined by the Department in TEGL 7-99 for participants and employers. The core questions are reprinted in item J in this section. Responses to these questions will be used by the State to calculate the ASCI, a proprietary product secured under contract by the Department to measure satisfaction at the State level. Respondents should be told that responding to the survey is voluntary and that the information they provide will be kept confidential. States may include other questions in the participant and employer surveys.

    4. States must complete a minimum of 500 participant surveys and 500 employer surveys during the program year to accurately assess performance on the ACSI.

    5. States must randomly sample both participants and employers to obtain the desired number of completed surveys. States with less than 1,000 individuals eligible for a survey during the year must attempt to reach each potential participant or employer respondent.

    6. The response rates for both participant and employer surveys must be a minimum of 50 percent.

      The response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number selected for the sample. A survey is considered complete when valid answers are provided by respondents for each of the core questions outlined by the Department.

    7. Attempts should be made to contact all exiters and employers in the samples.

    8. The sampling methodology used to select potential respondents for the surveys must ensure the consistent random selection of a sample of individuals or employers eligible for the surveys. With this in mind, no eligible participant or employer may be arbitrarily excluded from a sample.

      It is very important that all records for eligible participants and employers be entered in a timely fashion into the databases. The information in these databases will be used to develop lists of participants and employers who received services that are the subject of the surveys. Both participant and employer samples must be drawn on at least a monthly basis. Also, procedures used to draw a sample must conform to accepted statistical practices, such as using a table of random numbers.

    9. Every precaution should be taken to prevent a response bias.

    10. States must use the minimum core customer satisfaction questions specified by the Department to collect and report satisfaction outcomes. These mandatory items should be located at the beginning of the questionnaire. States may include other questions about the service experience.

      The introductory statements and questions are listed below for both participants and employers. Please note the introductory statements are examples that can be modified to meet the needs of the State and use the names of program services that are recognizable by respondents.

      Participant Questions

      [Introductory Statement]

      My name is ______________ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr.___________.

      Are you the Ms./Mr. ________ who was looking for a job a few months ago?

      I would like to ask you some questions about your recent experience looking for a job. Our purpose is to learn from you how to improve programs and services offered to people in XXX. The survey should take about XX minutes to complete. First I am going to read a list of services you may have received. Indicate as I read them those you recall receiving during the period in which you were seeking employment and/or training at the XX center.

        "  A thorough assessment of your needs

        "  Assistance about finding a job

        "  Assistance to develop an individual employment plan

        "  Assistance to decide about the best training to take

        "  Assistance from someone to support you during your job search or training

        "  Did you receive any training?

        "  Occupational training

        "  Training to give you general skills for the workplace

        "  Training in English or math

      Did you get any other help or services that I have not mentioned? (specify)

      [Questions]

      1)  Utilizing a scale of 1 to 10 where "1" means "Very Dissatisfied" and "10" means "Very Satisfied" what is your overall satisfaction with the services provided from ________?

      Very Very DK REF Dissatisfied Satisfied 1 2 3 4 5 6 7 8 9 10 11 12

      2)  Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? "1" now means "Falls Short of Your Expectations" and "10" means "Exceeds Your Expectations."

      Falls Short of Exceeds DK REF Expectations Expectations 1 2 3 4 5 6 7 8 9 10 11 12

      3)  Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? "1" now means "Not very close to the Ideal" and "10" means "Very Close to the Ideal."

      Not Close Very Close DK REF To Ideal To Ideal 1 2 3 4 5 6 7 8 9 10 11 12

       

       

       

      Employer Questions

      [Introductory Statement]

      My name is ____________ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr.___________.

      Are you the Ms./Mr. _________ who (describe the service).

      I would like to ask you some questions about your recent experience with ________. Our purpose is to learn from you how to improve programs and services offered to employers. The survey should take about XX minutes to complete.

      [Questions]

      1)  Utilizing a scale of 1 to 10 where "1" means "Very Dissatisfied" and "10" means "Very Satisfied" what is your overall satisfaction with the service(s) provided from ________?

      Very Very DK REF Dissatisfied Satisfied 1 2 3 4 5 6 7 8 9 10 11 12

      2)  Considering all of the expectations you may have had about the services, to what extent have the service(s) met your expectations? "1" now means "Falls Short of Your Expectations" and "10" means "Exceeds Your Expectations."

      Falls Short of Exceeds DK REF Expectations Expectations 1 2 3 4 5 6 7 8 9 10 11 12

      3)  Now think of the ideal service(s) for people in your circumstances. How well do you think the service(s) you received compare with the ideal service(s)? "1" now means "Not Very Close to Ideal" and "10" now means "Very Close to the Ideal."

      Not Close Very Close DK REF To Ideal To Ideal 1 2 3 4 5 6 7 8 9 10 11 12

       

       

    11. Calculating the ACSI.

      The ACSI scores reported for a State represent the weighted sum of the three ACSI question values which are transformed into 0 to 100 scale value. The weights are applied to each of the three questions to account for differences in the characteristics of the State's customer groups.

      For example, assume the mean values of three ACSI questions for Michigan are:

      1. Overall Satisfaction = 8.3

      2. Met Expectations = 7.9

      3. Compared to Ideal = 7.0

      Then, these mean values from raw data must first be transformed to the value on a 0 to 100 scale. This is done by subtracting 1 from these mean values, dividing the results by 9 which is the value of range of a 1 to 10 raw data scale, and multiplying the whole by 100:

      1. Overall Satisfaction = (8.3 - 1)/9 x 100 = 81.1

      2. Met Expectations = (7.9 - 1)/9 x 100 = 76.7

      3. Compared to Ideal = (7.0 - 1)/9 x 100 = 66.7

      The ACSI score of Michigan is calculated as the weighted averages of these values. Assuming the weights for Michigan are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3, respectively, the ACSI score for Michigan would be calculated as follows:

      (0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

      Weights were calculated by a statistical algorithm to minimize measurement error or random survey noise that exists in all survey data. State-specific weights were calculated using the relative distribution of ACSI respondent data for non-regulatory Federal agencies previously collected and analyzed by CFI and the University of Michigan.

      Specific weighting factors have been developed for each State for PY 2000 and are included in Attachment 1. New weighting factors will be published annually. Please refer to Attachment 1 for a listing of the weights to be used by each State in computing and reporting performance on the ACSI for employers and participants.

  6. Parameters for Assessing Local WIB Program Performance on the Customer Satisfaction Measures. Section 136(c) of the WIA requires the core performance measures and indicators of customer satisfaction be applied to local WIBs within each State. As discussed above, we require the use of the ACSI methodology to measure satisfaction with services at the State level to satisfy the need to have a consistent and uniform approach for assessing and comparing State performance. States have two options for capturing customer satisfaction results for local WIBs:

    1. States may use the ACSI questions and approach to measure satisfaction at the local level if individual license agreements are purchased from CFI by the State and the ACSI questions and measurement approach are uniformly administered throughout the State. States using the ACSI methodology to measure local WIB performance are not required to conduct separate statewide customer satisfaction surveys to obtain State level results as long as:

      • the surveys are administered by telephone,

      • respondents are contacted within the window of opportunity outlined in item B of section 5,

      • at least 500 surveys are completed statewide for each group -- participants and employers,

      • the surveys comply with the guidelines spelled out in this section, and

      • the local WIB results can be weighted to account for sampling differences, if appropriate, and aggregated to obtain state level outcomes.

      States following this guidance may aggregate local outcomes to obtain State level outcomes on the customer satisfaction measures. These aggregated results should be reported on the quarterly and annual reports.

    2. States may also develop their own customer satisfaction measurement instruments and methodology to be used by all local WIBs. States may use existing customer satisfaction instruments and methodologies as long as they follow the parameters discussed later in this section.

    In either case, the approaches used by States to capture customer satisfaction information for local WIBs within each State must be applied uniformly and consistently to all WIBs in a State. The procedures for measuring and reporting customer satisfaction results for two or more local WIBs within a State should conform to the parameters described in this section. Single WIB States should use the parameters outlined in section 5 for conducting customer satisfaction surveys.

    Purchasing Licenses from the University of Michigan Business School

    States wishing to use the ACSI core questions, scoring and the interpretation of results for their local WIBs must purchase a separate license for each WIB at an annual fee of $1,000 per WIB.States interested in purchasing one or more licenses from the University of Michigan Business School should contact Ms. Sally Petersen, CFI Group Director of Marketing, at (734) 930-9090 or e-mail her at spetersen@mail.cfigroup.com.

    Purchasing the license(s) permits the State to designate the local WIB outcomes as ACSI results and calculate the results using weights from the University of Michigan Business School. guidelines for using the ACSI.

    Please note the three core questions used to compute the ACSI are in the public domain and may be used without payingan additional license fee. However, the ACSI name, weights, formula and comparisons with other ACSI customers are proprietary and, therefore, are not in the public domain. ACSI is a recognized standard in measuring satisfaction and the data collected using the three questions cannot be referred to as "ACSI" data or even be considered comparable to the ACSI without a license from the University of Michigan Business School.

    Parameters for Collecting Customer Satisfaction Information to Measure Local WIB Performance

    As mentioned earlier, States may purchase and use the ACSI methodology to measure local WIB performance or they may develop their own customer satisfaction measurement instruments and methodology to be used by all WIBs. To ensure customer satisfaction results for individuals are collected in a consistent and uniform manner throughout the State, procedures used to obtain participant and employer customer satisfaction information on local WIBs should satisfy the following criteria:

    1. Procedures should:

      • Measure customer satisfaction of employers and participants with services received from the local workforce investment program under Title I-B (Sec. 136(b)(2)(B)),

      • Ensure the satisfaction of employers and participants with services can be reported in an objective, quantifiable and measurable way (Sec. 136(b)(3)(i)(I)),

      • Be consistently applied among all local WIBs,

      • Be collected in a uniform and equitable manner for each WIB throughout the State,

      • Conform with widely accepted methodological and statistical criteria for measuring customer satisfaction.

    2. The customer satisfaction data collected should be of public use quality. Sufficient safeguards should be used to ensure the integrity of the data.

    3. The core question(s) and survey approach (e.g., in-person, telephone or mail) used by States to collect customer satisfaction information at the local level must be uniformly applied throughout the State. A core question is one that requires a valid response to calculate customer satisfaction.

      If the State uses the ACSI approach for each local WIB and wants to use these results to derive State level outcomes, the customer satisfaction information for each WIB must be collected by telephone interviews to be consistent with the guidelines detailed in section 5, Parameters for Conducting Surveys to Obtain State Level Results. In-person interviews and mail questionnaires may be used only in situations where the individual does not have a telephone. The ACSI core questions outlined in item J of section 5 must also be used in the surveys.

    4. Participants and employers should be contacted and respond as soon as possible after the receipt of the service(s).

      If the State is using the ACSI approach to gather results for each local WIB and wants to use these results to derive State level outcomes, the following timeframes should be observed to be consistent with the provisions outlined in section 5:

      • Participants should be contacted as soon as possible after the date of exit and no later than 60 days after the date of an exit. For registered persons who exit the program due to non-participation in services for 90 days, these individuals should be contacted no later than 60 days after the 90 days have elapsed since the last date of service.

      • Employers should be contacted as soon as possible after the completion of the service and no later than 60 days after the completion of the service. For employers who listed a job order where no referrals were made, contact should occur 30 to 60 days after a job order was listed.

    5. Both of the questionnaires for participants and employers must include one or more core questions used to determine overall satisfaction with services. Respondents should be told that responding to the survey is voluntary and that the information they provide will be kept confidential. States and WIBs may include other questions in the participant and employer surveys. However, the core question(s) on overall satisfaction should be asked first on the questionnaire to avoid unintentionally biasing results that may arise from the ordering of questions.

    6. Guidelines for constructing the core question(s) are:

      • Expect only moderate knowledge, understanding and recall of services.

      • Write questions and response options that are clear and unambiguous.

      • Provide for varying degrees of satisfaction and dissatisfaction in the response options.

      • The questions should be worded in a neutral way to avoid leading the respondent.

      • With one or more questions, ask respondents to rate their overall satisfaction with services.

      • Before asking the respondent to rate his or her satisfaction with services, confirm early in the interview that the respondent is a customer.

    7. Attempts should be made to contact all exiters and employers selected in the sample, if applicable. Sampling is not required to obtain results for local WIBs.

    8. If sampling is used, all samples must be randomly selected from the sampling frame. The sampling frame is a list of individuals eligible for the survey for whom contact information is available.

      All samples should meet the minimum sample size requirements specified in item J (minimum sample sizes for surveys of exiters or employers). The sampling percentage should be constant over time to avoid under-representing or over-representing time intervals.

    9. Programs may randomly sample both participants and employers to obtain the desired number of completed surveys. The sampling method must conform to widely accepted statistical approaches.

      The response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number who were actually reached by phone who were eligible for the survey. A survey is considered complete when a valid answer is provided by a respondent for each core question.

    10. The minimum number of completed interviews for various sampling frame sizes for the two surveys (participants and employers) are outlined below. For a survey to project results confidently to the total population being studied, the responding sample selected must be large enough for each local WIB to reach the desired number of completed interviews. States are responsible for determining the sample sizes or sampling rates needed to reach the minimum number of respondents.

      Required Number Completed Interviews

      Size of
      Sampling Frame
      Completed
      Interviews
      Over 10,000
      5,001 - 10,000
      3,001 - 5,000
      2,001 - 3,000
      1,501 - 2,000
      1,251 - 1,500
      1,001 - 1,250
      901 - 1,000
      801 - 900
      701 - 800
      601 - 700
      501 - 600
      451 - 500
      401 - 450
      351 - 400
      326 - 350
      301 - 325
      276 - 300
      251 - 275
      226 - 250
      201 - 225
      176 - 200
      151 - 175
      126 - 150
      101 - 125
      91 - 100
      81 - 90
      71 - 80
      61 - 70
      56 - 60
      51 - 55
      45 - 50
      Under 45
      500
      370
      356
      340
      321
      305
      293
      277
      268
      259
      247
      233
      216
      206
      195
      182
      175
      168
      160
      151
      141
      131
      119
      107
      93
      79
      72
      65
      58
      52
      47
      42
      All in Sampling Frame

       

       

    11. Every precaution should be taken to prevent a response bias.

  7. Aggregating Local WIB ACSI Results to Obtain Statewide Results. As mentioned earlier in this issuance, States using the ACSI methodology to measure local WIB performance are not required to conduct separate customer satisfaction surveys to obtain State level results described in section 5, Parameters for Conducting Surveys to Obtain State Level Results. In situations where a single sampling rate is used throughout the State, the local WIB ACSI results on a customer satisfaction measure may simply be aggregated to obtain State level results.

    In situations where different sampling rates exist for the WIBs in a State, each WIB's ACSI score must be weighted before aggregating the outcomes to obtain results that are representative of the State overall. The information needed to compute these weighted scores are the sample frames for each WIB in the State and each WIB's ACSI score.

    To illustrate how these weights are derived, consider the following example.

      Participant
    Sampling
    Employer
    Sampling
    Name

    WIB 1
    WIB 2
    WIB 3

    State A
    Frame

    1,824
    1,025
    1,151

    4,000
    ACSI

    67.2
    78.1
    68.6

    /////
    Frame

    597
    533
    487

    1,617
    ACSI

    65.2
    80.0
    67.0

    /////

     

     

    The weighted score would be calculated for each WIB by multiplying the ACSI score by the number in the WIB sample frame. These weighted WIB scores would then be totaled and divided by the sum of the sampling frame sizes for all WIBs in the State to obtain a State ACSI score.

    Using the figures provided in the example, the ACSI scores for the State are calculated in the following manner.

      Participant Employer
    WIB 1
    WIB 2
    WIB 3
    (67.2*1,824)=122,572.8
    78.1*1,025)= 80,052.5
    (68.6*1,151)= 78,958.6
    Total = 281,583.9
    (65.2*597)=38924.4
    (80.0*533)=42640.0
    (67.0*487)=32629.0
    Total = 114,193.4

     

    State A ACSI score for participants...
    (281,583.9/4,000)= 70.4

    State A ACSI score for employers...
    (114,193.4/1,617)= 70.6

     

  8. Action. States should distribute this guidance letter to all officials within the State who need such information to implement the performance accountability policies under the WIA.

  9. Inquiries. Questions concerning this issuance should be directed to your appropriate Regional Office.

  10. Attachment. ACSI Formula Weights