Executive Summary
List of Tables
1. Background
     1.1. Development of the Denied Claims Accuracy Pilot Project
     1.2. UI Benefits Activity and Denials
     1.3. Administrative Data on UI Determinations and Denials
     1.4. Measurement of the Accuracy of UI Denials
     1.5. The 1987 Denials Pilot Project
2. Description of the Denied Claims Accuracy Pilot Project
     2.1. Reasons for a New Pilot Project
     2.2. Questions to Be Addressed and Issues to Be Resolved
     2.3. States in the Pilot and Their Characteristics
     2.4. Decisions Guiding the Design and Reasons for Them
     2.5. Design of the DCA Pilot Project
     2.6. Timing of the Pilot Project
3. Description of the Pilot's Operations and Findings from Monitoring Visits
     3.1. Conduct of the Site Visits
     3.2. New Jersey Site Visit
     3.3. West Virginia Site Visit
     3.4. Nebraska Site Visit
     3.5. Wisconsin Site Visit
     3.6. Summary of Findings from the Site Visits
4. Quantitative Analysis of Denials Accuracy
     4.1. Incidence of Erroneous Denials
     4.2. Responsibility for Erroneous Denials
     4.3. To What Extent Are Erroneous Denials Corrected by the System?
     4.4. Combined Wage Claims and Erroneous Denials
     4.5. Claimant Response and Denied Claims Errors
     4.6. Comparing the QPI Assessment with DCA Measurement
     4.7. Determinants of Erroneous Denials
5. Is It Possible to Estimate the Dollar Impact of Erroneous Denials?
     5.1. Description of the Problem
     5.2. Possible Solutions
     5.3. Discussion and Conclusions
6. Staff Costs of Conducting Investigations
7. Summary and Recommendations
     7.1. Summary of the Design of the DCA Pilot Project
     7.2. Summary of Main Findings from the Pilot
     7.3. Recommendations
References

During fiscal year 1998, UI agencies made over 10.6 million determinations of monetary eligibility for UI benefits, over 3.4 million separation determinations, and over 4.3 million nonseparation determinations (see Chapter 1 for details). The Denied Claims Accuracy Pilot Project was an operational pilot conducted during 1997-98 in five states with the purpose of gathering the information that will be needed to guide a program of measuring Denied Claims Accuracy in all 53 UI jurisdictions. This report describes and presents the main findings from the pilot project.

Design and Operations of the Pilot Project. Each of five pilot states that agreed to participate drew random samples of (roughly) 200 monetary denials, 200 separation denials, and 200 nonseparation denials and subjected them to intensive investigation (along the lines of the existing core Benefits Quality Control program) in order to determine their accuracy. Monetary denials were investigated using the BQC approach, which involved a review of all pertinent agency records, an interview with the claimant, and contacts with base-period employers to ascertain the correct wages, hours of work, weeks of work, etc., as prescribed by State law. Nonmonetary denials (both separations and nonseparations) were reviewed twice: first, under a BQC-type review involving a claimant interview and appropriate contacts with employers and/or third parties to determine the eligibility decision that would accord with a fully-informed application of State law and policy; and second, under a briefer review of applicable agency data alone, which resulted in a rating of the denial according to the existing Quality Performance Indicator (QPI) instrument. (Chapter 2 presents further details of the pilot's design.)

Each pilot state received resources for two DCA Pilot Project investigators. Sampling began in week 36 of 1997 (early September 1997) and continued until between week 33 and week 36 of 1998 (August through early September 1998), depending on the pilot state. Representatives of the National Office, Regional Offices, and the contractor made site visits to four of the five pilot states. Findings from the site visits include the following:

• The pilot states reported that 3 staff years would be needed to handle a case load of 200 of each type of denial (600 total).
  
  
• The pilot states reported greater difficulty in obtaining information from claimants during DCA investigations than they experience during core BQC investigations.
  
  
• The pilot states reported that difficulty in obtaining responses from claimants caused delays in completing cases. However, the pilot states do favor timeliness standards for the DCA program.
  
  
• Most pilot states agree that the sampling of monetary denials should be delayed for two weeks (tenwork days) from the date that the claim was filed, in order to avoid sampling claims that were initially denied but that will be redetermined based on confirmation of additional wages.
  
  
• The pilot states reported no greater difficulties with investigations involving claims than with those involving intrastate claims.
  
  
• Staffs of the pilot states generally indicated that they prefer intensive investigation of denied claims (as in the DCA Pilot Project) to the Quality Performance Indicator rating of nonmonetary denials
  
  
• The pilot states reported only minor (and easily resolved) start-up problems in implementing the COBOL sampling program and getting the DCA Pilot Project running.
  
  
• Most of the pilot states had suggestions for changes in the Data Collection Instrument. (See Chapter 3 and section 7.3.)
  
  
• Staffs in some of the pilot states indicated that training in the BQC methodology in general and in nonmonetary determinations in particular would be beneficial.
  
  

See Chapter 3 for further information on the pilot's operation and for the detailed findings from the site visits.

Summary of Main Quantitative Findings of the Pilot. Table ES-1 summarizes the essential findings of the 1997-98 DCA Pilot Project regarding the accuracy of denied claims. (See Chapter 4 for a full discussion of the quantitative findings of the pilot.) Column (1) shows the denied claim error rates unadjusted for the effects of appeals, redeterminations, or agency actions to resolve issues. Column (2) shows the error rates adjusted for the effects of appeals and redeterminations, column (3) shows the error rates adjusted for the effects agency actions to resolve issues, and column (4) shows the error rates adjusted for the effects of appeals, redeterminations, and agency actions to resolve issues.

Column (1) of Table ES-1 shows that, in the five pilot states together, the unadjusted monetary denial error rate averaged 16 percent, the separation denial error rate averaged 8.7 percent, and the nonseparation denial error rate averaged 15 percent. Both unadjusted and adjusted error rates tend to be lower for separation denials than for monetary and nonseparation denials. (Because of substantial differences among the five pilot states in laws, policies, and procedures, no conclusions about the quality of state administration can or should be drawn from interstate comparisons of the error rates.)

Column (2) of Table ES-1 shows that appeals and redeterminations had no impact on monetary denial error rates. Appeals and redeterminations did, however, reduce the separation denial error rate in the five pilot states together by about 2 percentage points (or 22 percent), and reduced the nonseparation denial error rate in the five pilot states by about 1 percentage point (or 6 percent). Appeals and redeterminations, then, are more effective in reversing erroneous nonmonetary denials than in reversing erroneous monetary denials.

Column (3) of Table ES-1 shows that the State UI agencies, through their own actions, reduced the monetary denial error rate in the five pilot states together by nearly 5 percentage points (or nearly 31 percent). The agencies' actions also reduced the separation and nonseparation denial error rates in the five pilot states, but by only about 1 percentage point in each case. Agencies' actions, then, are more effective in reversing erroneous monetary denials than in reversing erroneous nonmonetary denials.

Table ES-1 also displays summary results of the 1987 denied claims pilot. The unadjusted error rates in the 1997-98 DCA Pilot Project are generally lower than those in the earlier pilot (except for nonseparation error rates, which are about the same in both pilots). In fact, the unadjusted error rates in the 1997-98 pilot are roughly comparable to the error rates after adjusting for appeals and redeterminations in the earlier pilot (again, with the exception of the nonseparation error rates). This suggests that the determinations process, before any self-correction, has improved over the past decade. However, appeals and redeterminations now do less to reduce denial errors than they did a decade ago. (It appears impossible to adjust the error rates from the 1987 pilot so as to take account of agency resolution of issues. Accordingly, columns (3) and (4) of Table 7-1 have no data on the 1987 pilot.)

A main objective of the DCA Pilot Project was to compare the results of comprehensive field investigations with the QPI assessment of the quality of the determinations process. Table ES-2 shows cross-tabulations of the accuracy of separation and nonseparation denials by whether the denial determination passed or failed the QPI review. The results suggest that the correlation between QPI and DCA is highly imperfect: In only one case is the coefficient of correlation between the DCA assessment and the QPI assessment greater than 0.3. In all five pilot states and for both separation and nonseparation denials, a significant percentage of denials that were determined improper by DCA passed QPI, and a significant percentage of denials that were determined proper by DCA failed QPI. Moreover, the QPI suggests that the determinations process is less accurate than comprehensive field audits show. The conclusion is that the QPI is a very noisy predictor of the accuracy of denials.

Chapter 5 provides a brief discussion of the difficulties of estimating the dollar impact of erroneous denials. That discussion concludes that there are both conceptual and practical problems with estimating the dollar impact of erroneous denials. However, with further research, it may be feasible to overcome the problems and produce estimates of the dollar impact of erroneous denials that could be reasonable guides for policy. Recommendations. Chapter 7 summarizes the recommendations stemming from both the quantitative findings reported in Chapter 4 and the process of visiting the pilot states The main recommendations are as follows:

• It appears that, once allowance is made for supervisory review, about 2 to 3 staff years, not 2, are needed to investigate 600 denials in a year. The UI Performs budget currently includes only 2 full-time equivalent positions per state for denials. Accordingly, we recommend maintaining staffing at this level and making the denials annual sample size 150 for each type of denial.
  
  
• Four of the pilot states completed only between 43 and 79 percent of interviews in monetary investigations, and only between 68 and 83 percent of claimant interviews in nonmonetary investigations. We expect that interview completion will be a continuing problem in conducting investigations of denials. Accordingly, the experience of states and investigators with the best records should be drawn on and assembled in a handbook to provide advice and techniques for increasing contact rates. Standards for "reasonable attempts and effort" in attempting to contact claimants should be produced, consistent with the BTQ "reasonable attempt" standards.
  
  
• States reported no greater difficulty (or cost) in investigating and assessing the accuracy of interstate denials than intrastate cases. Accordingly, interstate cases should be included in the national program. • For the nationwide program, sampling of monetary denials should be delayed for up to 14 days so that wagerequest processes are completed and most of the denials in the sample are "true" denials. Nonmonetary denials should be sampled and assigned for investigation at the end of the week in which they occurred, just as in the pilot.
  
  
• The pilot showed little correlation between the results of the DCA review of denied nonmonetary claims and the QPI review of the quality of nonmonetary decisions. The clear conclusion is that the accuracy of nonmonetary denials is best estimated using the BQC reassessment and reverification method.
  
  
• States should be given the same flexibility to select among the various review techniques as they have for paid claim accuracy.
  
  
• The pilot states found that the DCA review process worked best when all Quality Control staff shared denials (DCA) and payment (core BQC) cases. Accordingly, we recommend that states be given flexibility in assigning staff to DCA cases.
  
  
• The DCA Pilot Project had no standards for timely completion of cases. We recommend that time lapse standards be established for denials in the nationwide program, with the understanding that these standards may differ from the BAM standards.
  
  
• The pilot was intended to identify data elements which should be captured for denials investigations, or for which definition should be modified. The pilot states identified several elements in the Denied Claims DCI for which edits needed to be modified or which could be pre-filled or "stamped" in case they were inapplicable in the state. The pilot states recommended eliminating all elements designed to capture dollar impacts of denials. A workgroup of State and Regional staff has met to review all other elements of the Data Collection Instrument to be used in both Denials and Benefits Quality Control and to develop a revised and integrated DCI.

Table ES-1
Summary of Denial Case Error Rates by Type, Unadjusted and Adjusted for Appeals, Redeterminations, and Agency Resolution, 1997-98 and 1987 Denials Pilot Projects (percentage erroneous denials for all pilot states with five-state range in parentheses)

Source: Tables 1-5, 4-10A, 4-10B, and 4-10C.

Table ES-2 Crosstabulations of Separation and Nonseparation Denial Accuracy by Modified QPI Pass/Fail, All Five Pilot States (frequencies with row percentages)

Table 1-1: Unemployment Insurance Eligibility, by Type of Determination and Outcome, United States, fiscal year 1998
Table 1-2: Monetary Determinations in the Five Pilot States and the United States, fiscal year 1998
Table 1-3: Separation Determinations in the Five Pilot States and the United States, fiscal year 1998
Table 1-4: Nonseparation Determinations in the Five Pilot States and the United States, fiscal year 1998
Table 1-5: Summary of Denial Case Error Rates by Type, 1987 Denials Pilot Project Table 1-6: Erroneous Denials for which Agency Was Responsible, by Type, 1987 Denials Pilot Project
Table 1-7: Agency Actions in Erroneous Denials, by Type, 1987 Denials Pilot Project
Table 2-1: General Characteristics of States Participating in the DCA Pilot Project, 1997
Table 2-2: UI Eligibility Criteria in States Participating in the DCA Pilot Project, 1997
Table 3-1: Informal or "Counter" Monetary Denials and Formal Monetary Denials of Unemployment Insurance Benefits, West Virginia, by Week, April and May 1998
Table 4-1: Denial Case Error Rates by State and Type
Table 4-2: Accuracy of Monetary Denials by State and Reason for Denial
Table 4-3: Accuracy of Separation Denials by State and Reason for Denial
Table 4-4: Accuracy of Nonseparation Denials by State and Reason for Denial
Table 4-5: Responsibility for Erroneous Denials, by State and Type of Denial
Table 4-6: Summary of Actions on Erroneous Denials, by State and Type of Denial
Table 4-7: Point at which Error Was First Detected in the Investigation, by State and Type of Denial
Table 4-8: Prior Agency Action on Erroneous Denials, by State and Type of Denial
Table 4-9: Prior Employer Action on Erroneous Denials, by State and Type of Denial
Table 4-10A: Monetary Denial Case Error Rates, Unadjusted and Adjusted for Appeals, Redeterminations, and Agency Resolution of Issues, by State
Table 4-10B: Separation Denial Case Error Rates, Unadjusted and Adjusted for Appeals, Redeterminations, and Agency Resolution of Issues, by State
Table 4-10A: Nonseparation Denial Case Error Rates, Unadjusted and Adjusted for Appeals, Redeterminations, and Agency Resolution of Issues, by State
Table 4-11: Combined Wage Claims and Denial Case Error Rates, by Type of Investigation and State

Table 4-12: Response Rates and Methods Used to Obtain Information in DCA Investigations, by Type of Investigation and State
Table 4-13: Denial Case Error Rates by Whether Information Was Obtained from the Claimant, Type of Investigation and State
Table 4-14: Crosstabulations of Separation Denial Accuracy by Modified QPI Pass/Fail
Table 4-15: Crosstabulations of Nonseparation Denial Accuracy by Modified QPI Pass/Fail
Table 4-16: Appeals by Finding of DCA Investigation and by Result of QPI Assessment: Separation Determinations
Table 4-17: Appeals by Finding of DCA Investigation and by Result of QPI Assessment: Nonseparation Determinations
Table 4-18: Components of QPI as a Predictor of Denied Claims Accuracy: Regression Analysis
Table 4-19: Relationships between Characteristics of Claimants and the Likelihood of Proper Denial: Regression Analysis
Table 6-1: Staff Hours Spent per Investigation of Denied Claims Accuracy, by Type of Denial, Five DCA Pilot Project States
Table 7-1: Summary of Denial Case Error Rates by Type, Unadjusted and Adjusted for Appeals, Redeterminations, and Agency Resolution, 1997-98 and 1987 Denials Pilot Projects
Table 7-2: Erroneous Denials for which Agency Was Responsible, by Type, 1997-98 and 1987 Denials Pilot Projects
Table 7-3: Crosstabulations of Separation and Nonseparation Denial Accuracy by Modified QPI Pass/Fail, All Five Pilot States

1.1. Development of the Denied Claims Accuracy Pilot Project

In 1984, the Unemployment Insurance Service of the U.S. Department of Labor developed a Benefits Quality Control (BQC) program. (This program is currently known as Benefits Accuracy Measurement; however, throughout this report, the original designation of "BQC" will be used.) In October, 1987, the "core" BQC program, which measures payment errors (particularly overpayments) became mandatory in 52 of the 53 Unemployment Insurance jurisdictions (all but the Virgin Islands). Originally, the Unemployment Insurance Service envisioned expanding the BQC program – after appropriate pilot testing – into aspects of the UI program other than the accuracy of intrastate paid claims. In particular, the agenda for quality control efforts included pilot tests for the accuracy of benefit denials, the accuracy of interstate payments, the accuracy of tax collections, and the use of telephone, fax, and the mail to perform quality control.

The General Accounting Office, the Office of Inspector General, organized labor, and advocates of claimants strongly urged the Department to expand BQC and implement a program that would estimate the accuracy of UI claims that were denied – in other words, a "Denials Quality Control" or "Denied Claims Accuracy" subprogram. This was viewed as important in order to maintain balance in the way the Department treats claimant issues. During fiscal year 1987, in a DOL-State pilot, five states investigated the accuracy of claims that were formally (that is, officially) denied for monetary reasons, reasons of separation, and reasons of weekly eligibility.

Although the 1987 pilot suggested that a substantial number of denials were erroneous and were not corrected by the workings of the redetermination and appeal processes, other priorities and resource limitations precluded incorporating denials into the framework of the BQC program. Rather than proceed with measuring the accuracy of denied claims, the Unemployment Insurance Service conducted the Revenue Quality Control and Performance Measurement Review initiatives. In 1993, however, the Department of Labor agreed with the Vice President's National Performance Review that the BQC program needed to be reexamined. According to the National Performance Review issue paper, the Department, as part of its reexamination of BQC, needed to consider whether BQC should "continue to keep its existing focus on paid claims ... or include measurement of decisions to ... deny claims."

The BQC reexamination was undertaken by a joint Federal-State Performance Enhancement Work Group and occurred within the context of an overall review and restructuring of UI performance measurement. The Performance Enhancement Work Group first proposed a new approach to unifying all UI performance measurement, and within this system it proposed a benefit accuracy measurement program considerably smaller than BQC that offered states more flexibility to use telephone, fax, and mail to verify information. The Performance Enhancement Work Group also recommended that the accuracy of denials be assessed.

There has been continuing interest in assessing the accuracy of denied claims for other reasons as well. In its 1994 performance review of the Unemployment Insurance Service's Consolidated Financial Statements, the Department of Labor's Office of Inspector General recommended that the Unemployment Insurance Service "initiate quality control programs to measure the accuracy of denied initial claim determinations which should be quantified and reported as underpayments in the financial overview." In a March 23, 1995, letter to Deputy Secretary of Labor Thomas Glynn, the United Auto Workers recommended that "benefits quality control should be modified to measure and report wrongful denials of UI benefits." Also, in a March 1995 meeting with Department of Labor officials, representatives of the National Employment Law Project said they considered the lack of a means for measuring the accuracy of denials to be a definite deficiency.

Because of the time that had elapsed since the 1987 pilot, given the need to address various questions that were left unresolved after the earlier pilot, and at the urging of the Deputy Secretary of Labor, the Unemployment Insurance Service decided to conduct another pilot in order to test the measurement of denials accuracy before implementing such a program nationwide.

This report describes that pilot – known as the Denied Claims Accuracy Pilot Project. The plan of the report is as follows. The remainder of this chapter presents the context of the Denied Claims Accuracy Pilot Project by discussing the processes by which eligibility for UI benefits is determined and the ways in which claimants may be denied UI benefits. Chapter 1 also reviews basic concepts of accuracy measurement and reviews the conduct and results of the 1987 Denials Pilot Project.

Chapter 2 describes the Denied Claims Accuracy Pilot Project, its purposes, and its design. Chapter 3 presents findings and recommendations from site visits that were made by representatives of the UI Service (both the National Office and Regional Offices) and the contractor (PRAMM Consulting Group, Inc.). Chapter 4 presents a quantitative analysis of the findings of the DCA Pilot Project. Chapter 5 discusses whether it might be possible to estimate the dollar implications of erroneous denials – that is, to estimate how much would have been paid in benefits if the erroneous denials had been determined correctly. Chapter 6 is a brief discussion of the costs of conducting a DCA program. Chapter 7 summarizes the report and presents conclusions and recommendations for implementation of the DCA program on a national basis.

1.2. UI Benefits Activity and Denials

There are three ways in which UI benefits may be denied to workers. First, a potential claimant might conclude that he or she is not likely to be eligible for benefits and hence never contact the UI agency. This might be called a "self denial." Second, a claimant might make an informal inquiry, usually about monetary eligibility (see below), and be advised that he or she is not eligible or not likely to be eligible. As a result, the potential claimant never files a formal initial claim. This case has frequently been termed an "informal" or "counter" denial. Third, if a formal initial claim is filed, Ul eligibility is tested, more or less sequentially, at three levels, and there is the potential for a formal denial at each level. Note that (except in section 3.4) this report focuses only on formal denials. What follows is a description of the three formal criteria by which eligibility for benefits are tested.

1.2.1. Monetary Criteria. In order to be eligible for UI benefits, claimants must have earned enough wages or worked enough weeks (or both) during a "base period," which is usually the first four of the past five completed calendar quarters before the initial benefit claim was filed. If the work and pay records available to the UI agency indicate that the state's monetary conditions are not satisfied, a "monetary denial" of benefits remains in effect until enough additional wages to qualify are earned. (In some cases, the records available to the agency are incomplete, and the denial remains in effect until the missing work and pay records appear or are identified).

As can be seen in Table 1-1 (top panel), in fiscal year 1998, about 10.6 million determinations of monetary eligibility were made. Although not shown in the table, about 90 percent of these determinations were made on initial claims (officially designated "new initial claims"). The remainder were made on transitional claims – claims filed by claimants who remain unemployed when their original benefit year expires and who attempt to establish a new benefit year on the basis of earnings during the lag quarter. The lag quarter is the quarter immediately preceding the quarter in which the initial claim was filed and is not part of the traditional base period. (To be eligible for benefits, such a claimant must generally have had some earnings subsequent to the lag quarter as well.) In 1998, 1.2 million, or 11 percent, of the monetary determinations resulted in a determination of monetary ineligibility.

>1.2.2. Separation Criteria. To be eligible for UI benefits, a claimant must also have been separated from employment due to lack of work and through no fault of his or her own. This means that workers who quit a job voluntarily are generally ineligible for UI benefits, although a worker who had "good cause" for quitting may be eligible for benefits (the definition of "good cause" varies from state to state, as discussed in section 2.3 below). It also means that a worker who is discharged for misconduct will be ineligible for UI benefits. Depending on the state and the specific reason for failing the separation criterion, a separation denial can make a claimant ineligible for periods ranging from several weeks to indefinitely. In the case of an indefinite separation denial, a worker will generally be required to find reemployment and to earn "requalifying wages" before being eligible for benefits.

Separation determinations can be made on initial claims and reopened claims – that is, claims that have the potential to start a new series of UI benefit payments. (Reopened claims – officially, "new additional claims" – are claims made during an existing benefit year after an interruption in benefit receipt.) The second panel of Table 1-1 shows that, in 1998, there were about 15.4 million initial and reopened claims. UI agencies raised and adjudicated separation issues with respect to 3.4 million (or 22 percent) of these initial and reopened claims. In turn, 55 percent of the adjudications resulted in denial of benefits. Accordingly, 12 percent of all initial and reopened UI claims resulted in denial of benefits based on conditions of separation.

1.2.3. Nonseparation Criteria. Finally, a claimant who is monetarily eligible for UI benefits and who separated from employment for acceptable reasons must be able to work, available for work, and actively seeking work if required by State law or policy. An otherwise eligible claimant who is not "able, available, and actively seeking" work is denied UI benefits for reasons of "nonseparation." A nonseparation denial usually involves loss of eligibility for the claimed week, although a worker who refuses to accept suitable work be denied benefits for several weeks.

In 1998, there were about 131.6 million weekly claimant contacts that could have resulted in a nonseparation determination of weekly eligibility (see the bottom panel of Table 1-1). Of these, State agencies identified and formally adjudicated about 4.3 million issues (that is, 3.3 percent of the contact weeks were adjudicated). The adjudications resulted in about 2.4 million denials. Hence, about 1.8 percent of all weekly claimant contacts resulted in denial based on nonseparation conditions.

An issue of nomenclature should be raised at this point. Separation and nonseparation criteria for UI eligibility are sometimes referred to together as "nonmonetary" criteria. As a result, denials that occur because a worker's conditions of separation were unacceptable are sometimes referred to not simply as "separation" denials but rather as "nonmonetary-separation" or "nonmon-sep" denials. Similarly, denials that occur because a worker was not "able, available, and actively seeking" work are sometimes referred to not simply as "nonseparation" denials but rather as "nonmonetarynonseparation" or "nonmon-nonsep" denials. Throughout this report, we use the shorter designations – "separation" denials and "nonseparation" denials – to refer to the two types of nonmonetary denial, understanding that both separation and nonseparation denials result from applying nonmonetary criteria for eligibility.

1.3. Administrative Data on UI Determinations and Denials

As discussed in the previous section, State UI agencies make millions of determinations each year regarding monetary and nonmonetary eligibility of claimants. This section provides further detail on UI benefit activities and denials. In particular, we present information reported by the states on monetary determinations (Table 1-2), separation determinations (Table 1-3), and nonseparations (Table 1-4). Data are shown separately for each of the five states that participated in the Denied Claim Accuracy Pilot Project (Nebraska, New Jersey, South Carolina, West Virginia, and Wisconsin), along with national totals. Our purpose is to compare UI benefit activities and denials in each of the five pilot states (and in the five pilot states combined) with the national experience.

1.3.1. Monetary Issues. Reports on monetary determinations and nonmonetary determinations (ETA 218 and ETA 207, respectively) are submitted quarterly by the states. The time period in these reports that most closely approximates the period when the pilots were actively collecting data was from October 1997 through September 1998, i.e., federal fiscal year 1998. Although actual data collection in the pilot states typically commenced slightly earlier, in mid-September 1997, and ended in late-August or early-September 1998, the timing of the data collection closely matched fiscal year 1998.

Nationwide during fiscal year 1998, over 11 percent of monetary determinations concluded that the claimant had insufficient wage credits for monetary eligibility. Table 1-2 shows there were 10.6 million monetary determinations and 1.2 million denials for a monetary denial rate of 0.114. The five pilot states combined made a total of 0.94 million monetary determinations of which 0.10 million had insufficient wage credits. Hence, the average monetary denial rate in the pilot states – 0.106 – was only slightly lower than the national average of 0.114.

When the individual states are examined in Table 1-2, however, the range of monetary denial rates is observed to be quite wide. Nebraska’s denial rate of 0.029 was only about one quarter of the national average. West Virginia and Wisconsin also had denial rates well below the national average – 0.065 and 0.069 respectively. In contrast, New Jersey and South Carolina had monetary denial rates that exceeded the national average during this period – 0.142 and 0.131 respectively.

Inferences about incorrect monetary denials cannot be made from the program data in Table 1-2. However, if the case error rates from the denials pilot are representative of national experience, the overall rate of erroneous monetary denials probably falls in the 0.13-0.19 range. The average case error rate was 0.16 across the five pilot states (144 of 901 monetary determinations). Applying the average case error rate of 0.16 to the 1.2 million determinations of insufficient wage credits would imply that more than 190,000 erroneous monetary denials (out of 10.6 million determinations) were made throughout the United States during fiscal year 1998. This is nearly 2 percent of all monetary determinations.

1.3.2. Separation Issues. Table 1-3 summarizes separation denials in fiscal year 1998. Unlike monetary determinations, which occur only when a claimant establishes a new benefit year, separation determinations are associated with both initial and reopened claims, the latter occurring after an interruption in benefit receipt during an established benefit year. Panel A of Table 1-3 displays initial and reopened claims for each state, for the five pilot states combined and the U.S. totals for fiscal year 1999. (In fact, what is shown as "initial and reopened claims" in Table 1-3 is the sum of (a) new initial claims times the fraction monetarily eligible, (b) interstate liable claims, and (c) reopened claims. The figures come from the ETA 218 and ETA 207 reports.) Two issues, voluntary quits and misconduct, account for nearly all separation determinations in the states. Therefore, Table 1-3 displays totals plus detail for these two issues. Panel B displays two types of separation activity measure: determination rates (determinations as a proportion of initial and reopened UI claims) and denial rates (denials as a proportion of initial and reopened UI claims). Both the determination rate and the denial rate affect the total number of separation denials. That is, a high volume of denials can be obtained by various combinations of a high determination rate and a high denial rate per determination.

The first feature that is clear in Table 1-3 is Nebraska’s unusually high determination rates for both voluntary quits and misconduct. In fact, Nebraska has the highest separation determination rate of the 53 UI jurisdictions because it adjudicates every separation that may be relevant to monetary eligibility. Nationwide, the average separation determination rate was 0.22. In the other four pilot states, the separation determination rate ranged between 0.134 (South Carolina) and 0.238 (Wisconsin). In Nebraska, however, the proportion was 0.947; that is, there were nearly as many separation determinations as there were initial and reopened UI claims. While determinations on misconduct issues were also frequent in Nebraska (more than twice the national average), the determination rate for voluntary quits was more than six times the national average (0.679 versus 0.096).

As noted, separation determination rates in the pilot states other than Nebraska are not far from the national averages. Nationally (and in the four pilot states other than Nebraska) there is a somewhat higher determination rate for misconduct issues than for voluntary quits. Nationwide, the average was 0.118 for misconduct and 0.096 for voluntary quits in fiscal year 1998. For all five pilot states (including Nebraska), the corresponding averages were 0.121 and 0.102, both just above the national average.

Separation denial rates are noticeably higher for voluntary quits than for misconduct. The national average denial rates in Table 1-3 were respectively 0.737 and 0.407. Compared with these averages, the averages from the pilot states were slightly higher: 0.802 for voluntary quits and 0.445 for misconduct. Nebraska, New Jersey, South Carolina and West Virginia had above average denial rates for voluntary quits (all above 0.850) while South Carolina also had a very high denial rate for misconduct (0.758). Because denial rates are much higher for voluntary quits, total denials resulting from this issue exceed misconduct denials (1.09 million versus 0.74 million) despite the fact that states made more misconduct determinations than voluntary quit determinations (1.83 million versus 1.48 million).

When the number of separation denials is related back to initial and reopened UI claims, Nebraska again is the outlier. Nationwide and in the four pilot states other than Nebraska, separation denials as a proportion of initial and reopened claims ranged between 0.107 and 0.120. In Nebraska, however, the corresponding proportion was 0.785 (these figures are not displayed in Table 1-3). The national proportion of 0.120 means there were 1.86 million denials due to separation issues in fiscal year 1999. Data from the five pilot states indicate that 8.7 percent of denials on separation issues were improper (87 of 1,006). Applying this case error rate to the 1.86 million separation denials of fiscal year 1998 would imply that, nationwide, there were over 160,000 improper denials on separation issues for this period.

1.3.3. Nonseparation Issues. Table 1-4 summarizes nonseparation determinations with totals and details for four different types of nonseparation issues. Because a nonmonetary determination could potentially occur for any week in which benefits are claimed, we sum continued weeks claimed and initial claims and refer to this sum as claimant contacts (that is, the sum of weeks claimed). We then define the nonseparation determination rate as the number of nonseparation determinations per 10 claimant contacts. (Since nonseparation determinations are relatively rare in the case of initial claims, it can be argued that initial claims should be excluded from the measure of claimant contacts. Excluding initial claims from claimant contacts would reduce claimant contacts nationally from 131.6 million to about 115 million, and would raise the national nonseparation determination rate from .329 [that is, 3.3 percent] to 0.374 [3.7 percent]. Thus, excluding initial claims from claimant contacts would change only slightly the nonseparation determination rates shown in Table 1-4.) The nonseparation denial rate, in turn, is the number of nonseparation denials per nonseparation determination. In fiscal year 1998, there were almost 132 million claimant contacts in all UI programs. For this same period there were more than 4.3 million determinations on nonseparation issues and almost 2.4 million nonseparation denials.

The meaning of nonseparation denials should be stressed. Most refer only to the week of the determination and not to the entire remaining period of potential eligibility. This contrasts with separation determinations, where denials usually apply to the entire spell under consideration. It follows that a denial on, say, an able and available issue will pertain just to the week in which the issue has been raised. Denials on nonseparation issues typically have less severe consequences for the claimant than denials on separation issues. (This argument does not apply to all nonseparation denials. If there is a refusal of suitable work – a less common nonseparation issue – the denial can apply to the entire remaining period of entitlement.)

The four nonseparation issues identified in Table 1-4 account for most nonseparation determinations. Combined, they totaled 4.12 million, or 95 percent of all determinations on nonseparation issues. The issue "other" is a catchall that includes alien applicants, athletes, and covered employees of school systems.

In the pilot states taken together, these four issues are generally less common than in the rest of the United States. Whereas the five pilot states accounted for 0.085 of claimant contacts, they accounted for lower shares of determinations on able and available, disqualifying (and deductible) income, and "other" issues. They had an above average representation only among nonseparation determinations related to reporting requirements (0.096). They had especially low representation among determinations relating to disqualifying income (0.024).

Within the five pilot states, overall determination rates are above average in Nebraska and Wisconsin. In the other three states, the overall determination rates are half or less of the national average. Nebraska has high determination rates for reporting requirements and for disqualifying income. Wisconsin has high determination rates for reporting requirements and "other" nonseparation issues.

In the United States overall, nonseparation denials occur in over half (0.545) of the determinations that are made. Overall and for three of the detailed determination categories, the average denial rate in the pilot states exceeded the national average. Denial rates are especially high in Nebraska. West Virginia was the only pilot state where the average nonseparation denial rate was less than the national average. Results from the pilot project suggest the case error rate data on nonseparation denials to be 0.150. In fiscal year 1998, there were a total of 2.36 million denials on nonseparation issues. Applying the case error rate from the pilot project to these national totals would imply an estimated total of 350,000 erroneous denials on nonseparation issues during fiscal year 1998.

Combining the rough estimates made here from all three types of erroneous denials would yield a total estimate of about 680,000 erroneous denials of UI claims during fiscal year 1998. This is the sum of 170,000 involving monetary determinations, 160,000 involving separation issues, and 350,000 involving nonseparation issues. This total gives an indication of the large potential for erroneous decisions affecting worker entitlements to UI benefits.

1.4. Measurement of the Accuracy of UI Denials

To a large extent, self denials and informal denials are beyond the reach of the UI system (but see section 3.4 below). Accordingly, the Department of Labor has regarded the extent and correctness of self denials and informal denials to be a matter for research, not for operational measurement. At present, the UI system does not assess the accuracy, per se, of denied claims, although as noted in section 1.1, the Department has recognized the importance and feasibility of measuring the accuracy of denied claims. The quality of most nonmonetary determinations, including denials, is regularly appraised through the Quality Performance Indicator Assessment (discussed next), but monetary denials are not appraised in any way. At present, the UI system has two mechanisms that could be used to measure the accuracy of all or some denied claims.

1.4.1. The Quality Performance Indicator (QPI) Assessment of Nonmonetary Determinations. Since the 1970s, Ul agencies have assessed the "quality" of nonmonetary adjudications by reviewing a sample of adjudications using the Quality Performance Indicator (QPI), a part of the Quality Appraisal program. About 60 percent of the determinations reviewed under QPI during fiscal year 1998 were denials, 34 percent were eligible claims, and information was not available for 6 percent of the cases. The QPI review basically assesses the quality of the processes of applying the separation and nonseparation criteria. The QPI was not designed to determine whether the process reached the fully correct decision given State law and policy, nor are the results tabulated to determine the share that were accurate. The QPI notion of accuracy is limited to determining whether the state (a) took proper steps to obtain information, (b) offered the opportunity for rebuttal, (c) correctly identified an issue, and (d) correctly applied appropriate law. The accuracy of the information used to make the determination is not considered in the QPI process. Neither is missing information due to late response or no response from one or another party considered.

1.4.2. Benefits Quality Control (BQC). The BQC program, implemented under regulation in October 1987, investigates the accuracy of UI benefit payments and yields estimates of UI benefit overpayments. It does not measure the accuracy of denials. The BQC program samples only payments (specifically, weeks paid), and accurately estimates the number of overpayments and dollars overpaid. However, it underestimates underpayments because it samples only weeks in which UI benefits were paid – it does not sample weeks that should have been paid but were not. By thoroughly reviewing and field-verifying all information relating to the accuracy of payment decisions, BQC investigators determine whether UI staff had the necessary and correct information to work with and whether information obtained by UI staff was used properly. Accordingly, BQC provides an assessment of whether decisions were made with full information and in accordance with law and policy. Moreover, BQC assesses both monetary determinations and nonmonetary (that is, separation and nonseparation) determinations. (Recall that QPI applies only to nonmonetary determinations.) In the case of separation and nonseparation determinations, the BQC review ascertains whether all issues were correctly identified by UI staff and whether identified issues were correctly handled. It also ensures that all appropriate information is marshaled for the adjudications. Hence, the BQC review often obtains information not received previously by the agency or not submitted timely to the agency.

The "BQC methodology" involves drawing samples of paid claims. As already outlined, the decision to pay a weekly benefit follows a sequence of three decisions. Reviewing the correctness of the payment requires BQC to verify all prior decisions on which the decision to pay the particular week claimed depends. As a result, BQC investigates "up the chain" of UI decisions. In 1994, BQC reviewed (through its samples) the universe of 120.4 million weeks paid. In addition, it reviewed the 10.4 million monetary decisions and 13.3 million separation decisions that had to be positive in order for those 120.4 million paid weeks to be correct. In 1994, the universe of decisions investigated by BQC sampling amounted to approximately 97 percent of all decisions made in intrastate Ul, UCFE, and UCX cases. (Interstate payments are not investigated in the BQC program.) As noted, several groups have stressed the need for balance in BQC by using its approach to assess the accuracy of denied claims as well as paid claims. This concern for balance led to the 1987 pilot testing of several approaches (including the existing BQC approach outlined here) soon after payments (or "core" BQC) became operational.

1.5. The 1987 Denials Pilot Project

During fiscal year 1987, with the technical assistance of Applied Management Sciences, Inc., five states pilot tested three different approaches to assessing the accuracy of formally denied claims using the BQC field-verification approach. The five states were Louisiana, Pennsylvania, Iowa, South Carolina and Washington. (This section relies on Belle and Casey 1988.)

• Approach 1: Separate cross-sectional weekly samples of denied claims, with BQC maintained. One state defined separate universes of the three types of denial decisions (monetary, separation, and nonseparation). It drew and investigated weekly samples of each. This state's BQC program remained unchanged and continued to measure the accuracy of paid claims in its usual way. This approach was adopted for the 1997-98 DCA Pilot Project.
  
  
• Approach 2: Separate cross-sectional weekly samples of positive and negative monetary, separation, and nonseparation decisions. In one state, the process of determining eligibility was disaggregated into its three stages or levels of decisions: monetary, separation, and weekly nonseparationeligibility. At each level, samples of both positive and negative decisions were drawn and investigated (as in BQC) to determine their accuracy. This model offered an alternative to the existing BQC design.
  
  
• Approach 3: Longitudinal tracking. Three states (Iowa, South Carolina, and Washington) tested a longitudinal approach, which, like Approach 2, involved a redesign of BQC. Each week, a sample of initial claims was drawn and added to a "tracking file." The experience of this cohort was monitored; all denials were investigated as they occurred, beginning with monetary denials. In addition, a sample of the weeks paid to the claimants remaining in each cohort was investigated each week, providing an alternative to BQC's method of selecting payments for review.

Although the five states used three different methods for selecting samples of denials, once the samples were selected, all investigated the cases in the same way, following as much as possible the in-person BQC methodology that was current at the time (there is now greater reliance on telephone, fax, and mail in conducting investigations). In addition to reviewing all pertinent agency records, each investigation involved an interview with the claimant and contacts with the parties necessary to ascertain the facts on which the denial decision was based.

1.5.2. Summary of Findings. The 1987 pilot investigated the correctness of denied claims before redetermination or appeal, and also noted whether those denials were ultimately reversed by the appeal and redetermination process. Table 1-5 displays the percentages of the decisions to deny benefits that the 1987 pilot found to be in error (that is, should have approved as eligible UI claims) both before and after appeal and redetermination.

The 1987 pilot suggested that, on average, existing appeal or redetermination processes reverse one-fifth to two-fifths of erroneous denials (depending on the state). Nevertheless, between one-tenth and one-sixth of denials remain erroneous. The pilot itself yielded only case error rates. Because claimants who are denied benefits have no observable claim experience, the dollar impact of erroneous denials – benefits lost by claimants – can be estimated or imputed only through some form of statistical modeling (see Chapter 5 for further discussion). The contractor did attempt this in the 1987 pilot, but the number of assumptions required to make such projections makes them of questionable value for guiding decisions.

In addition to investigating the correctness of denials, states in the 1987 pilot applied the usual BQC action, cause, and responsibility codes to denial errors. As Table 1-6 shows, the UI agencies were attributed either partial or total responsibility (the contractor's report did not differentiate between these) for about one-quarter of the erroneous monetary denials and for about three-quarters of erroneous nonmonetary (separation and nonseparation) denials.

Finally, the 1987 pilot characterized the agencies' actions on erroneous denials in one of five ways: (a) the source of the issue was undetectable with existing procedures, (b) the issue was already being resolved by the agency, (c) the wrong action was taken by the agency, (d) there was insufficient follow-through by the agency, and (e) the issue was not detected because the agency did not follow procedures. The percentages of erroneous denials for which each of these agency actions was taken are shown in Table 1-7.

The range associated with each of the actions taken is quite large, in part because the estimates are based on errors detected, the absolute number of which was frequently quite small. However, on average, Table 1-7 suggests that existing procedures were able to detect about three-quarters of all types of erroneous denials. In roughly one-quarter to one-half of all cases, the agency was already resolving the error. For nearly one-third of the erroneous separation and nonseparation denials, the agency had the necessary information but took the wrong action. (The agency took the wrong action in relatively few monetary cases.) For the remaining one-sixth (roughly) of all erroneous denials, the agency did not follow through on information it had or missed the issue because it did not follow its procedures.

2.1. Reasons for a New Pilot Project Several of the factors impelling the Department of Labor to measure the accuracy of denied claims have been outlined in section 1.1. Based on the experience of the 1987 pilot and further discussion, the Department decided in 1995 to measure denied claims accuracy by drawing three separate (cross-sectional) samples of denial decisions on a weekly basis – one each for monetary, separation, and nonseparation denials – and investigating each decision using the BQC methodology. This was one of the three alternative sampling designs (Approach 1) tested in the 1987 pilot (see section 1.5 above), and it was chosen as the most satisfactory sampling design because, both conceptually and practically, it closely resembles the sampling used to conduct the existing BQC program. As a result, it involves no fundamental redesign of the sampling procedures used for BQC.

Rather that attempt immediate nationwide implementation, however, there were three reasons for conducting a second pilot. First, nearly ten years had elapsed since the 1987 pilot, and it was unknown whether conditions had changed substantially since then. For example, changes in determination rates or denial rates could change the resources needed by State agencies to measure denied claims accuracy. If so, then those changed conditions would need to be taken into account in the design of a nationwide program.

Second, even if conditions were similar, the experience of a few pilot states would be highly useful to the other states in implementing the denied claims accuracy program. Third, several questions were left unresolved after the 1987 pilot, and additional information was needed to address these questions and guide implementation of a national measurement effort. These questions are discussed next.

2.2. Questions to Be Addressed and Issues to Be Resolved

The 1997-98 Denied Claims Accuracy Pilot Project had the goal of addressing the following four principal questions.

1. What are the resource implications of investigating denials accuracy? Answering this question requires answers to the following:

• How large should be the samples of denied claims that are investigated? Larger samples yield more precise estimates but require more resources. There is a tradeoff between the precision of the estimates and the resources needed to obtain them.
  
  
• How long do investigations of denied claims take? The 1987 pilot concluded that, on average, a denials investigation according to the BQC in-person protocol required about 60 percent as much time as a payment investigation. Since that time, however, the BQC protocol has changed. In 1993, an "alternative methodology" was adopted, and in January 1997, the Department allowed states complete flexibility to use telephone, mail, and fax to carry out investigations. How long a denials investigation takes will depend importantly on how the investigations are conducted – using the present mix of in-person and phone, mail,and fax contacts; using all phone, mail, and fax; or relying more or less entirely on in-person methods. Another determinant will be the extent of the denials investigation itself.
  
  
• In addition to changes in the way BQC investigations are conducted (mentioned above), it has since been recommended that the BQC approach should be used only to investigate the accuracy of monetary denials. The rationale for this recommendation stems from existence of the Quality Performance Indicator (QPI) review of nonmonetary determinations (both separation and nonseparation denials). This recommendation would have states use the QPI method and instrument to review samples of separation and nonseparation denials, rather than adopt the BQC approach (as has been done in the 1997-98 DCA Pilot). If this recommendation were accepted, it would have considerable resource implications because it involves only a review of existing records. It would not involve recontacting the parties to the adjudication, as does the BQC approach.
  
  
• Since the BQC approach is to be used, what should be the scope of the investigation for nonmonetary denials? That is, should a separation investigation focus only on the circumstances of the separation, or should it include a review of monetary eligibility as well? Should investigation of a nonseparation denial be limited to that particular issue, or should it include monetary and/or separation eligibility as well? (Under BQC, which was followed in general for the 1997-98 DCA Pilot Project, investigation of a separation denial does involve investigating prior monetary eligibility, and investigation of a nonseparation denial involves investigating prior monetary and separation eligibility. It should be noted, however, that under core BQC, a denial cannot be the starting point of an investigation; rather, a denial arises only when BQC reverses an original positive determination during an investigation.)

2. In sampling and in conducting the investigations, what allowances should be made for the normal processes of redeterminations and appeals? In other words, at what time should samples of denials be drawn, and at what time should they be investigated?

3. What modifications to the Data Collection Instrument that is used in BQC are needed for the Denied Claims Accuracy program? A basic assumption guiding the DCA Pilot Project was that the measures of interest are rates of correct and incorrect denial decisions of the three main types. (That is, the DCA Pilot Project did not attempt to estimate or project dollar impacts of erroneous denials.) In order to estimate erroneous denial rates, however, the states needed to determine what information would need to be gathered in the course of the denials investigation. Accordingly, decisions needed to be made about which elements to include and which to drop from the existing BQC Data Collection Instrument. Decisions also needed to be made about adding any new elements. Only the experience of states that were actively investigating denials and estimating denial rates could guide these decisions.

4. What is the appropriate means, and level of detail, for obtaining information on the cost of investigating the various types of denial issues?

2.3. States in the Pilot and Their Characteristics

As already noted, the pilot involved five participating states – Nebraska, New Jersey, South Carolina, West Virginia, and Wisconsin. These states were selected from volunteers responding to a system-wide announcement that requested applicant states to indicate the following:

• the name and qualifications of their proposed candidate for pilot coordinator;
• their ability to identify the populations of monetary, separation, and nonseparation denial actions from automated records;
• their ability to build transaction files and implement COBOL programs, and maintain both;
• their interest and capability to write BQC-type (COBOL) sample selection programs;
• their commitment to making UI-qualified investigators available for the pilot; and
• whether their UI law has features that might affect the difficulty or extent of denials; for example, the existence of an alternative base-year law.

In selecting the five states, the Department of Labor attempted to achieve a reasonable balance among considerations of geography, state size, and features of the UI law.

Some general characteristics of the five pilot states are displayed in Table 2-1. The pilot states represent a broad range of states by size of population and labor force, recent population growth rate, geographic size, population density, average pay, and labor market tightness (as measured by the unemployment rate). One of the pilot states, New Jersey, is among the most populous states (ranking ninth in population), and another, Wisconsin is a medium-sized state (ranking eighteenth). South Carolina's population has grown slightly faster than the U.S. average in the 1990s, whereas the other four states have grown more slowly than the U.S. average (with West Virginia and New Jersey being well below the U.S. average in their growth rates). Geographically, Nebraska is the largest of the pilot states and has very low population density, whereas New Jersey is the most densely populated state in the country. Except for New Jersey, which has relatively high average annual pay, the pilot states are near the U.S. average in annual pay. Finally, West Virginia's labor market is slack (an unemployment rate of 6.9 percent in 1997) compared with the national average, whereas Nebraska and Wisconsin have labor markets that are significantly tighter than the national average (unemployment rates of 2.6 and 3.7 percent).

For the purposes of the pilot, the diversity of UI eligibility criteria among the five pilot states is also important. Table 2-2 gives a summary of the monetary, separation, and nonseparation eligibility criteria in the five pilot states. The table and following discussion are based on the U.S. Department of Labor's Comparison of State Unemployment Insurance Laws (1997), Advisory Council on Unemployment Insurance (1995, Chapters 7 and 8 ), Nicholson (1997), Anderson (1997), and the laws of each of the five pilot states.

2.3.1. Monetary Eligibility Criteria. Table 2-2 characterizes the monetary eligibility criteria of the states along four dimensions: (a) the type of formula used to determine eligibility, (b) the minimum base period earnings required for eligibility, (c) the type of earnings distribution requirement, and (d) whether there are any restrictions on the eligibility of seasonal workers.

All four types of formula for determining monetary eligibility are represented among the five pilot states. Nebraska and West Virginia are among the 6 states that use the socalled "flat" formula, under which a claimant must have earned some specified minimum dollar amount during the base period. (In addition, both require that base period earnings be earned in more than one quarter.) New Jersey requires that a claimant have worked at least 20 weeks (at specified minimum weekly earnings) during the base period. (New Jersey is one of 7 states that require a worker to have worked a minimum number of weeks or hours in the base period.) South Carolina is one of 24 states using a "multiple of highquarter wages" formula. South Carolina's formula requires a worker to have earned at least $600 in the high quarter of the base period (that is, the quarter of the base period in which earnings were highest) and to have earned at least 1.5 times the high-quarter earnings (that is, at least $900) in the entire base period. Finally, Wisconsin is one of 14 states using a "multiple of weekly benefit amount" formula. Wisconsin' s formula requires first that a worker have earned at least $1,325 in the high quarter of the base period, which would qualify the claimant for a weekly benefit amount of $53. The worker must then have earned at least 30 times that calculated weekly benefit amount (that is, at least $1,590) during the entire base period to be monetarily eligible.

The second row under monetary eligibility criteria in Table 2-2 gives the minimum base period earnings that a claimant would need in order to qualify for benefits, as computed by the U.S. Department of Labor (Comparison of State Unemployment Insurance Laws, 1996, Table 301). The third row shows each state's distribution requirement. For example, Nebraska requires the claimant to have earned at least $400 outside the high quarter of the base period (the high quarter is the quarter of the base period in which earnings were highest). Similarly, the other five pilot states all have some requirement that the claimant's base period earnings were not concentrated in a single quarter. New Jersey, South Carolina, and West Virginia each require that a claimant have earnings in at least two quarters of the base period. Wisconsin requires the claimant to have earned at least 8 times his or her weekly benefit amount outside the high quarter of the base period.

Finally, the fourth row under monetary eligibility criteria in Table 2-2 shows that West Virginia and Wisconsin have special provisions that restrict the eligibility of seasonal workers. In general, wages earned by workers in an industry defined as seasonal can be used to establish UI eligibility only for unemployment during periods when the worker is usually employed in his or her seasonal job.

2.3.2. Separation Criteria. The middle panel of Table 2-2 summarizes the two main aspects of the separation criteria that a claimant must meet in each of the pilot states. The first is how a state handles the UI eligibility of a worker who voluntarily quit his or her job. Such a worker will be eligible for UI benefits only if he or she has "good cause" for quitting, and the definition of good cause differs from state to state. Table 2-2 shows that in New Jersey, West Virginia, and Wisconsin, the definition of "good cause" is restricted to issues directly related to work or the employer, although West Virginia and Wisconsin define certain specific reasons for voluntary leaving that are permitted. (These include such reasons as illness of the claimant and leaving to accept another job that does not materialize. The number of such specific reasons is shown in the "inclusions" row of Table 2-2.) Nebraska and South Carolina, on the other hand, allow "good cause" to include good personal reasons as well as reasons that are directly related to employment.

Table 2-2 also shows the length of time a claimant who quits voluntarily will be disqualified from receiving UI benefits. In Nebraska, a worker who quits voluntarily is disqualified for 7 to 10 weeks. In the other pilot states, a worker who quits voluntarily is disqualified for the duration of his or her current spell of unemployment. In order to requalify for benefits, the worker must then earn some minimum amount, specified as a multiple of the weekly benefit amount in New Jersey, South Carolina, and Wisconsin. In West Virginia, the worker must work at least 30 days to requalify.

The second aspect of the separation criteria summarized in Table 2-2 pertains to misconduct. In most states, a worker who is discharged for misconduct will be ineligible for UI benefits for some specified number of weeks. In the pilot states, the duration of disqualification ranges from 5 weeks in New Jersey to up to 26 weeks in South Carolina. In addition, Table 2-2 shows that all of the pilot states except New Jersey reduce the benefits that a worker receives at the end of a disqualification period stemming from misconduct. Finally, all of the pilot states except Wisconsin impose one or another additional penalty on a worker who was discharged for gross misconduct.

2.3.3. Nonseparation Criteria. Essentially, the nonseparation criteria for continuing eligibility specify that an individual must be able to work, available for work, and (usually) actively seeking work. The bottom panel of Table 2-2 summarizes the main features of the nonseparation criteria that a claimant must meet in each of the pilot states.

As noted by Anderson (1997) all the states require that a worker be "available" for work in some sense, although some states, such as South Carolina and West Virginia require only that a worker be available to work in his or her usual occupation. Nebraska, New Jersey and Wisconsin, in contrast, require that a worker be available for any work. (Several states require that a worker be available for "suitable" work, although none of these was in the DCA Pilot Project.)

The other main feature of nonseparation eligibility criteria that is shown in Table 2-2 is the handling of workers who refuse suitable work. In all of the pilot states, a worker who refuses suitable work is disqualified from receiving benefits for at least some time, and in two of the states – South Carolina and Wisconsin – such a worker is disqualified for the duration of his or her unemployment spell. In addition, Nebraska and West Virginia reduce the benefits that a worker receives at the end of a disqualification period stemming from refusal of suitable work.

The DCA Pilot Project was developed with the following policy decisions in mind:

• The Department of Labor, following the recommendation of the State-Federal group that developed the Ul Performs design, is committed to developing an approach to measuring denials accuracy;
  
  
• There is a need to measure the accuracy of denied claims in the same way as paid claims; that is, there is need to measure the extent to which decisions to deny UI benefits are being made in accordance with fully-informed application of State law and policy;
  
  
• The recommendation to use the QPI instrument for reviewing nonmonetary denials accuracy is a reasonable one. Accordingly, it makes sense to ascertain whether a QPI review would be a satisfactory alternative to (or substitute for) the more extensive (and more expensive) full BQC investigation;
  
  
• Data on denied claims will be verified in the same way as data on paid claims. This means that the same "full flexibility" in using phone, mail, and fax will be allowed investigators determining the accuracy of denied claims as is allowed in verifying paid claims.

These decisions imply that the DCA Pilot Project was an operational pilot, not a feasibility study or benefit-cost study. The only benefit-cost aspect of the pilot concerned the cost-effectiveness tradeoff regarding use of the QPI vs. the BQC methodology in assessing the accuracy of nonmonetary denials.

It follows that the goal of the DCA Pilot Project was to achieve the following objectives:

• Assess the current range of denial errors and establish desirable sample sizes;
  
  
• Determine how much the BQC field-verification methodology adds to the QPI methodology in assessing the accuracy of nonmonetary denials;
  
  
• Determine the cost of measuring denials by both the BQC and QPI methodologies;
  
  
• Identify the principal reasons for denial errors and ensure that these reasons are 32 coded so that all states can identify and eliminate or mitigate them;
  
  
• Identify obstacles to measuring denials errors consistently.

2.5. Design of the DCA Pilot Project

2.5.1. Overview. The pilot involved five participating states – Nebraska, New Jersey, South Carolina, West Virginia, and Wisconsin – selected from volunteers responding to a system-wide announcement. Each pilot state designated a coordinator who assisted the National Office (NO), the Regional Office (RO), and contractor staff (PRAMM Consulting Group, Inc.) in refining the design and outline materials. The National Office, in collaboration with the states, produced the operational instructions resulting from a series of annual meetings, starting in 1995, that brought together all the participants.

Department of Labor staff drew up the specifications for the universes of denials ("transactions files") from which samples were to be drawn. National Office technical support staff designed the sample selection programs and database programs as variations of the programs already written for BQC and RQC applications. Programming staff in two of the pilot states, South Carolina and Wisconsin, wrote the COBOL program used to edit the populations and select the samples. In addition, programming staff in each state installed and maintained the programs. After training the participating State staff (drawn mainly from existing BQC ranks) and completion of programming, states began sampling and investigating denied claims during September 1997, and continued sampling and investigating over the following year.

Federal and contractor staff monitored each state during the project, and a site visit was made to each state (except for South Carolina) for the purposes of monitoring the progress of the project. After completion of the investigations, the states completed case data collection instruments, which the Department downloaded to a central database in the National Office. The Department provided copies of the data to the contractor for the evaluation.

In November 1997, staff of the pilot states, the National and Regional Office Federal staff, and contractor staff met in Charleston, South Carolina, to discuss operational issues identified in the first months of the pilot. In November 1998, there was a debriefing meeting to discuss preliminary findings of the pilot and to help shape the final report and its recommendations. States will be asked to review and comment on the draft evaluation report.

2.5.2. Implementation of the Design. Implementation of the pilot involved the following steps:

1. Sampling Cases for Investigation. Starting in week 36 of 1997 (early September 1997), each participating state selected weekly samples of monetary, separation, and nonseparation denials for investigation. The goal was to sample four of each type of denial each week for investigation, and this goal was achieved in all five states by the second week of the pilot. The data elements that were included are indicated in Appendix 1 (the Data Collection Instrument).

2. Investigations. The pilot coordinator/supervisor assigned the sampled denials for investigation.

Monetary denials were investigated using a BQC-like protocol, which involved review of all pertinent agency records, an interview with the claimant and contacts with base-period employers to ascertain the correct wages, hours of work, weeks of work, etc., as prescribed by State law.

Nonmonetary denials (both separations and nonseparations) were reviewed as follows: (a) The DCA-trained investigator conducted a BQC-type review of the case, involving a claimant interview and appropriate contacts with employers and/or third parties to determine the eligibility decision that would accord with a fully-informed application of State law and policy. The field-verification data were coded into a data record maintained on the state's Sun computer. (b) A different investigator performed a QPI appraisal of all separation and nonseparation denials that were drawn for the pilot project. This investigator reviewed all pertinent and available agency records and scored the quality of the denial decision using the existing Quality Performance Indicator (QPI) review instrument. The QPI scoring elements were coded into a data record maintained on the state's Sun computer.

3. Collection of Time Data. Investigators and supervisors were expected to record time required to complete investigations and review cases. In general, each pilot state assigned a project code for the DCA pilot, and State staff recorded time spent performing DCA investigations using the state's time-charge system. Time-recording was kept to an absolute minimum.

4. Case Reviews. As now required under the BQC program, the supervisor was responsible for reviewing all cases before they were considered final. In addition, a sample of cases was reviewed by Federal or PRAMM monitors during site visits.

5. Data Pickup and Integrity. The Department of Labor picked up the data electronically on a periodic basis for its own use and for transmission to the contractor. The contractor also contacted State staff occasionally in the course of using the data.

2.5.3. Duration of the DCA Pilot Project and Level of State Effort. The pilot involved sampling over approximately a one-year period beginning week 36 of 1997 (early September 1997) and ending between week 33 and 36 of 1998 (late August to early September 1998). South Carolina and Wisconsin completed sampling in week 33 of 1998, New Jersey and West Virginia completed sampling in week 35, and Nebraska completed sampling in week 36. It was initially expected that approximately 3 months would be needed after the last sample was drawn to complete verifications, coding, and case review. (In fact, the data on which the results in Chapter 4 are based were current as of early November 1998 and were not yet complete.) The State pilot coordinators were available for refining the design of the pilot, training, and comment from the execution of the pilot agreement (September 1996) through the debriefing meeting in November 1998 and subsequent review of the contractor's draft evaluation report.

At the beginning of the DCA Pilot Project, a decision was made to provide each state with resources for two DCA Pilot Project investigators. Based on the 1987 Denials Quality Control pilot project, DOL and the states estimated that two investigators would be enough to investigate 200 monetary, 200 separation, and 200 nonseparation denials over a period of a year. Based on this estimate, the participating states agreed to draw and investigate 200 of each type of denial. However, the initial estimate turned out to be optimistic; in fact, the states found that two investigators could handle only about 150 of each type of denial over the course of a year.

Resources for the DCA Pilot Project were provided to states by two means. First, through the new UI Performs Grants Allocation, the Department of Labor provided resources to each participating pilot state for two DCA Pilot Project investigator positions. Each state provided UI-qualified investigators for the project, usually experienced BQC investigators, and covered travel and other costs involved in investigations. Second, resources for the pilot coordinator (including travel) were provided through a cooperative agreement between the Department of Labor and the states. In addition, each state was provided with $15,000 through the pilot agreement to cover programming costs, which included developing the transaction files and installing and maintaining the sampling and database software. South Carolina and Wisconsin received additional funding to cover the cost of writing sample selection programs according to DOL specifications. These programs were shared with the other DCA Pilot Project states and were provided to all states for the purposes of sampling denials and bringing the BQC software into Y2K compliance.

2.6. Timing of the Pilot Project

A kickoff meeting, which was attended by representatives of the five pilot states, the Regional Offices, the National Office, and the contractor, was held in Washington, DC, in October, 1996. Several decisions were reached at that meeting, including a division of tasks between South Carolina and Wisconsin in producing the COBOL program that would be needed to sample denied claims during the pilot. The Division of Information Technology produced basic database software during summer and early fall of 1997. Sampling for the pilot began in September 1997 and ran through August or September 1998 (depending on the state). A mid-project meeting of all pilot participants, hosted by the South Carolina agency, was held in Charleston, South Carolina, November 12-13, 1997. During the spring and summer of 1998, representatives of the Department of Labor (both National Office and Regional Offices) and the contractor staff visited West Virginia, Nebraska, Wisconsin, and New Jersey. The states' investigations were completed by early November 1998, and a close-out meeting of all pilot participants, hosted by the New Jersey agency, was held in Atlantic City, New Jersey, November 18-20, 1998.

3.1. Conduct of the Site Visits

During the year of sampling and investigations for the DCA Pilot Project (that is, between September 1997 and August 1998), representatives of the National Office, Regional Offices, and the contractor (PRAMM Consulting Group, Inc.) visited four of the five participating states. These site visits had three main purposes: (a) to inform the staffs of National Office, Regional Offices, and the contractor of the procedures being followed by each state, (b) to allow each participating state to advise the National Office, Regional Offices, and the contractor of problems and concerns, (c) to help ensure the integrity of the DCA Pilot Project and its outcomes, and (d) to ensure the usefulness of the DCA Pilot Project to national implementation of DCA measurement.

No site visit was made to South Carolina, although a Mid-Project Meeting was held in Charleston, South Carolina, on November 13 and 14, 1997, shortly after the start of sampling and investigations for the pilot. The meeting brought together representatives of each participating state, the National Office, the Regional Offices, and the contractor, with the goals of discussing findings, operational issues, and any problems at an early stage of the pilot. As noted in the Mid-Project Meeting Summary (Clark and Skrable 1997), the meeting resulted in agreement on the handling of several operational problems that arose early in the pilot.

South Carolina was a participant in the 1987 pilot and as a result played a leading role in the 1997-98 DCA Pilot Project. It was clear from discussions with the South Carolina staff at the Mid-Project Meeting in Charleston that sampling and investigations were going smoothly in South Carolina. As a result, South Carolina was given a somewhat lower priority for a site visit than the other four states. The South Carolina staff would clearly have preferred that a site visit be made to Columbia, so that they could express any concerns that they had and provide direct evidence to the National Office, Regional Office, and contractor that their implementation of the pilot was sound. In the end, however, the contractor, in consultation with the National Office staff, decided that an additional visit to South Carolina would not be essential, given the clear evidence that South Carolina's operations were running smoothly.

This chapter reviews the activities of the monitors and State Employment Security Agency staff during each of the site visits and draws some general conclusions from those visits.

3.2. New Jersey Site Visit

On April 21 and 22, 1998, Andy Spisak of the National Office visited the New Jersey Department of Labor to discuss with agency staff their experience in conducting the DCA Pilot Project (Spisak 1998). On the morning of April 21, he met with Charles Salmon (Chief, Benefit Payments), Tom Hynes (BQC Supervisor), Jim Fink (DCA Project Officer), and the investigative staff – Phyllis Clinebell, Carolyn Merrill, Kelly Sacks, Madeleine Sekulich, and Lucy Webb. The meeting was a free-form discussion of the project management's and investigator's experiences with and observations concerning the DCA Pilot Project. The main points raised in the meeting follow.

1. Staffing and Time Needed for Investigation. New Jersey found that assigning two investigators to DCA was inadequate. A total of five BQC investigators had participated in the investigations of the sampled denied claims. The agency estimated that 9 to 11 hours were required to complete a DCA case, in contrast to about 13 hours for the investigation of a paid claim using the BQC methodology.

Although the use of telephone, fax, and mail to collect data were expected to save time (compared with an in-person investigation), the savings often did not materialize due to call-backs necessitated by non-response, difficulty in securing cooperation over the telephone, and ignored or returned mail.

2. Alternative Base Period Issues. New Jersey law allows UI claimants to qualify for benefits under several alternative base period options. Initial claims are often ruled ineligible based on minimum qualifying wages of $2,020 or 20 weeks worked. Claimants with less than $2,020 or 20 weeks worked are classified as "indeterminate" pending collection of additional base period wage information. Claimants who do not quality under the initial test may qualify when wages from the lag quarter and current quarter are added. Because of the alternative base period provision, some ineligible claims that entered the DCA monetary samples had been determined eligible by the time the DCA investigation took place. The implications for national implementation of DCA affect the timing of the sampling for monetary denials.

3. "Pro Forma" Filings. An estimated 15 to 20 percent of the monetary denials in New Jersey resulted from applicants for means-tested welfare who are required to apply for UI, even though they have earned no wages in the base period. In addition, some nonseparation denials result from claimants who wish to establish a disability claim under Social Security or Supplemental Security Income, and who file a UI claim with the expectation that they will be adjudicated unable to work due to disability or illness. (In one case that Spisak reviewed, the claimant actually returned an uncashed check to the local UI office with the statement that she was physically unable to work and was not entitled to UI benefits. She refused to participate in the DCA review.)

4. Training. The BQC and DCA staff agreed that, although the transition from investigating paid claims to investigating denied claims was relatively easy, training for both paid and denied claims would be very helpful, especially regarding nonmonetary (separation and nonseparation) adjudications. Also, staff turnover since the last comprehensive training for BQC investigators reinforces the need for training new investigators.

5. Interstate Claims. The investigators agreed that investigating denied interstate claims was no more difficult than intrastate claims and were in some respects easier, since documentation could be obtained from a central unit that processes interstate claims. A large proportion of the interstate workload for New Jersey consists of claims from Puerto Rico filed by seasonal agricultural workers.

6. Quality Performance Indicator. The New Jersey staff felt that the results of QPI scoring of the separation and nonseparation denials sampled was largely consistent with the results of the benefit quality scoring of nonmonetary determinations reported on the ETA 9056 quarterly report. The sense of the group was that continuing QPI as part of the national implementation of DCA would be redundant.

7. Case Documentation. Difficulty in obtaining case documentation from the local offices (an issue raised at the Charleston Mid-Project meeting in November, 1997) was eased by having the case investigators pick up agency records at the local offices (rather than requesting records be sent from the local offices). Using several investigators permitted coverage throughout the state.

8. Data Collection Instrument and Coding. The agency recommended that the Error Amount field be expanded from three to four digits to accommodate DCA underpayments that sometimes exceed $999.

In addition to meeting with the BQC and DCA staff, Spisak reviewed approximately 4 to 6 cases of each type of denial, including a mix of decisions determined to be correct and in error by the DCA staff. Spisak concluded that the overall quality of the investigative case work was excellent, particularly with respect to the thorough documentation of the cases. Each case included a concise and clear narrative summary of the case prepared by the investigator. In several cases the claimant could not be contacted, as evidenced by certified letters which were returned as undeliverable, and investigators' notes of attempted telephone contacts. In some cases, employers could not be contacted, for example, in the case of seasonal employers at the beach resorts which were closed by the time an attempted contact was made.

9. Multiple Issue Cases. Some claims involve multiple issues. Although the issue that was selected for DCA investigation may have been decided in error, the claimant might still be ineligible due to some other issue or issues. For example, in one New Jersey case that was reviewed, the DCA investigation determined that the claimant should not have been denied benefits due to failure to report for an Eligibility Review Interview. However, the investigation also discovered an issue of adequate work search for which the claimant would have been denied benefits.

In the BQC paid claims database, the data element "Amount That Claimant Should Have Been Paid" was established to capture more accurately the net effect of multiple issues. Consideration should be given to establishing a similar element for DCA cases. It should be made clear that DCA investigators are not required to search for collateral issues. However, multiple issues that are discovered in the claimant's file or through the claimant interview should be taken into consideration in coding the case. A data element which reflects the effects of multiple issues may allow managers and analysts to examine the financial implications of an erroneously decided issue.

On the afternoon of April 22, Spisak had a final meeting with Mike Malloy (New Jersey UI Director), Charles Salmon, and Jim Fink. During this meeting, discussion focused on the points summarized above.

3.3. West Virginia Site Visit

On June 11 and 12, 1998, Terry Clark and Julius Greene (of the National Office), Frank Wannamaker (of the Philadelphia Regional Office), and Wayne Vroman, Steve Woodbury, and Woody Wright (all of the contractor) visited the West Virginia Bureau of Employment Programs in Charleston to review the progress of the DCA Pilot Project there (Clark 1998). The group met with Dennis Redden, the project chief for the DCA Pilot Project, and Charlotte Grey of his staff. In addition, several other members of Mr. Redden's staff met with the group at appropriate times during the visit.

Dennis Redden provided a well-structured overview of the eligibility review and denial process as it exists in West Virginia, and reviewed several actual cases to give a sense of content and process. The group concluded that the cases were well done, and the documentation very complete. The group also verified that the QPI review was performed by QPI staff, separately from the DCA investigation. Finally, the group noted that Mr. Redden's staff in Charleston reviews every determination issued by the agency for general accuracy and returns those that are unacceptable to the originator for correction prior to initial release. As such, it is not surprising that the agency has high quality nonmonetary determinations across the board.

The following issues were discussed in some depth during the meetings.

1. Staffing and Time Needed for Investigation. The West Virginia staff expressed the view that insufficient staff time had been allowed for DCA investigation. This was especially true given that the QPI component was still in place, though it was noted that the QPI component probably would not continue after the pilot is complete. The West Virginia staff stated that handling a case load of 600 denials per year would require three positions (not two, as were allocated for the DCA Pilot Project). Dennis Redden's findings indicated that the review itself takes roughly 3 hours per case, with data entry requiring between 15 and 30 minutes, and supervisory review another 30 minutes.

2. Response Rates. Difficulty in obtaining responses from claimants was an issue because there is no real incentive for a denied claimant to cooperate in the investigation. West Virginia had found (as expected) that if the claimant had filed an appeal, he or she does respond to the DCA investigation. West Virginia investigators had also found that sometimes a claimant's story would change upon review – since the claimant knew that what was said the first time resulted in benefits being denied. The West Virginia staff call this the "second time around" syndrome.

Even so, West Virginia reported a response rate to the DCA inquiries of over 80 percent. The process involved mailing a questionnaire upon case selection, inviting response, and following up 2 weeks later by telephone if no response had been received. It would appear that this process was successful for West Virginia. The staff felt that is was easier to get a response in a nonseparation investigation than in a monetary or separation investigation because nonseparation denials are temporary. The expectation that additional benefits will be paid in the current benefit year in the case of a nonseparation denial creates an incentive for the claimant to cooperate in the DCA investigation.

3. Time Lapse Standards for Case Completion. The BQC program has set standards for the percentages of cases completed within 60 and 90 days. The group discussed completion targets in view of the different workload burden of DCA. Since DCA is designed to review only a portion of the data that the BQC program requires, it was felt that a shorter target for completion might be appropriate.

4. Timing of Sampling Denials. In the pilot design, it was decided to pull monetary denials immediately after the denial was issued. This created a problem for West Virginia because automatic denials – that is, denials that are issued before Federal or out-of-state wages can be added to a claimant's record – are sampled for DCA investigation. Often, these cases are eligible claims that would correct themselves in a few days when the missing (but already requested wages) are added to the claimant's record.

The West Virginia staff suggested that a slight delay – perhaps 2 weeks – in the selection process would solve this problem. The group agreed that this issue is not unique to West Virginia and needs to be resolved before nationwide implementation.

5. Interstate Claims. The experience of DCA investigators in West Virginia suggested that Interstate Claims could be included in the DCA investigations without difficulty. This was in large part because states are now allowed to use the telephone, mail, and fax to collect data and carry out investigations, rather than relying exclusively on inperson

methods.

7. Study of Informal or "Counter" Denials. Of particular interest was West Virginia's attempt to gather data about the incidence of self denials. The process of claimants selecting themselves out of the system before officially filing a claim for benefits occurs, presumably, in all states, but it is difficult to get information about the process because, by its nature, the process leaves no record to review. As noted in Chapter 1, denials that occur after only an informal inquiry into monetary eligibility (and without a formal claim being filed) are referred to as "informal" or "counter" denials.

In West Virginia, each informal inquiry into the system ("Would I be eligible?") creates a record that can be monitored manually. (The West Virginia agency refers to these as "pseudo-monetary" determinations.) West Virginia volunteered to monitor informal or "counter" claims for nine weeks, and created a program that precluded double entries if more than one inquiry was made. The program also edits out entries for which a formal claim is subsequently (that week) filed. This effort provided what appears to be the first data available on informal denials.

Table 3-1 displays the findings that West Virginia had accumulated after 9 weeks. The data suggest that, during a nine-week period in April and May, 1998, there were 752 informal denials and 551 formal monetary denials of UI benefits in West Virginia. However, none of the informal denials would be selected for DCA review, since only claimants who file a formal claim and are denied benefits become part of the population of denials from which samples are drawn.

While this special study gives an idea of the number of individuals who may be experiencing informal benefit denial, it does not yield data on the accuracy of those informal denials. West Virginia offered to investigate the accuracy of informal denials using the DCA Pilot Program methodology if the Department of Labor would continue to fund the two DCA Pilot Program positions during the year between the end of the Pilot and nationwide implementation of DCA. The group agreed that this would be an excellent opportunity to obtain data that would otherwise be unavailable. However, West Virginia later concluded that significant modification of their data base would be necessary to conduct investigations of informal denials.

8. Appellate Finality. A troublesome issue with the DCA investigative process is that the finality of a state's decision means that DCA investigative staff is unable to correct some instances of denial error. When discovery of an error works against a claimant, as it usually does in a BQC investigation, finality of a decision seems more acceptable, since finality implies that the claimant is not harmed by the outcome of the investigation. (That is, as a result of finality, the erroneous payment stands even though is was discovered to be erroneous.) In DCA, however, West Virginia has found that some erroneous denials can not be reversed because of finality. These are denials that, based on DCA investigation, are erroneous, should be reversed, and that would make the claimant eligible for payment if they were reversed. During the meetings, the group discussed the case of a woman who had been terminated (not for cause) and who had been told by the terminating employer to tell the UI office that she had quit voluntarily because it would look better on her record. The DCA investigation concluded that the woman should have been eligible for benefits, since the employer admitted giving the incorrect advice (that is, to say she had quit), which in turn had resulted in a denial of benefits. But the denial was final and could not be reversed as a result of the DCA review. The group concluded that the states may need to consider legislative remedies to allow limited jurisdiction when DCA is implemented nationwide.

3.4. Nebraska Site Visit

On June 29 and 30, 1998, Burman Skrable (of the National Office) and Steve Woodbury (of the contractor) visited the Nebraska Department of Labor offices in Lincoln to discuss Nebraska's experience in conducting the DCA Pilot Project. On the morning of June 29, they had a short meeting with Don Gammill, UI program evaluation administrator, and Ron Joyce, head of the BQC Unit, to discuss the general progress of the DCA Pilot Project in Nebraska. They then spent the remainder of the morning with Ron Joyce, who gave them a thorough review of the pilot's implementation in Nebraska. During the afternoon of June 29 and the morning of June 30, Skrable and Woodbury reviewed approximately 20 cases of each type of denial, randomly selected from the DCA files of the BQC unit. They found that the cases were well-documented and easy to follow. Ron Joyce made himself available to answer questions about Nebraska law and practice, and the process was very instructive. Skrable and Woodbury concluded that the quality of the investigations was excellent.

During the site visit, Ron Joyce raised the following issues, most of which he has since elaborated on in his "Summary Report" (Joyce 1998).

1. Staffing and Time Needed for Investigation. The design of the pilot called for two investigators to be assigned and dedicated to the DCA Pilot Project. Originally, the Nebraska staff had expected that separation and nonseparation investigations, since they tend to be "issue-specific," would be completed more quickly than monetary investigations, which tend to be more involved and take longer to complete due to the need to obtain wage records. These expectations turned out to be incorrect: separation and nonseparation investigations took nearly as long as monetary investigations, and a backlog of incomplete cases soon appeared. To help solve the backlog, the DCA case load was distributed among all BQC program staff. This did not solve the basic problem that two DCA investigators were insufficient to handle the case load, but the overload was distributed among five investigators rather than between just two. The clear conclusion in Nebraska, as in other states, is that 600 DCA cases annually are more than two investigators can handle.

The Nebraska staff has suggested that mandating the same DCA sample size for large and small states alike may put a relatively large burden on small states, particularly in regard to meeting timeliness standards (Joyce 1998).

2. Response Rates. Nebraska investigators experienced little difficulty in obtaining information for the investigation from employers. Obtaining information from denied claimants, however, was more difficult. Nebraska staff has estimated that its response rate from claimants was 74 percent, even with repeated attempts to reach the claimant by telephone.

Nonresponse delayed the completion of investigations because investigators were required to apply the standards established under BQC, which include second notification and subsequent mailing of questionnaires.

3. Timing of Sampling Denials. In the DCA Pilot Project, samples of denied claims were drawn at the end of the most recent production run, meaning that records were selected before claimants had been notified of the reason for their denial. This proved to be a problem in Nebraska because claimants could not be contacted until they had been duly notified. The result was further delays for case completion. Nebraska staff have suggested a procedural change that would solve this problem: Download DCA samples and distribute them for investigation only after the successive sample has been run. This would provide all parties adequate time to receive formal notification, and would also accommodate the potential filing of appeals before being contacted by BQC/DCA staff. A further sampling issue raised by Nebraska staff concerned monetary denials that involve combined wage claims. Specifically, Nebraska staff suggested that a combined wage claim should not be considered monetarily ineligible until the State Employment Security Agency has formally reached that conclusion. This would avoid drawing cases that, although they appear to be monetary denials at the time they are drawn, turn out to be eligible for benefits.

4. Programming. Nebraska was able to adapt the COBOL II sampling program without difficulty. A few sampling errors were found early in the pilot, and the problems were corrected. In general, programming and sampling appear to have gone quite smoothly in Nebraska.

5. Data Collection Instrument and Coding. Nebraska staff has noted that the Data Collection Instrument for the DCA Pilot Program was derived from the existing BQC Data Collection Instrument. As a result, coding the findings of investigations was a problem in certain cases and could be simplified in others. One problem, for example, was that the DCI would accept only three digits for a monetary denial error, which means that the potential dollar error could not be reflected accurately.

The main simplification suggested by Nebraska staff is that the DCI should provide the option to stamp any field (in other words, increase the number of defaults). For example, the DCA has a series of earnings-related fields that do not apply to nonseparation denials, and hence could default to zero. Nebraska staff estimate that there are as many as 26 such fields. Another example would be the "before audit" fields, which do not apply to monetary denials, and could also default to zero. A review of the DCI should lead to a determination of which coding elements are necessary.

6. Dollar Implications of Erroneous Denials. Finally, the Nebraska staff has been especially concerned about the difficulties associated with estimating the dollar implications of erroneous denials. As discussed further in Chapter 5, estimating the dollar impact of erroneous denials is inherently difficult and speculative, and Nebraska's law and practice raise additional issues that are likely to make such estimates rather unconvincing (see Chapter 5 and Joyce 1998).

3.5. Wisconsin Site Visit

On August 3 and 4, 1998, Jack Estes (of the Chicago Regional Office) and Steve Woodbury and Woody Wright (of the contractor) visited the Milwaukee offices of the Unemployment Insurance Division, Wisconsin Department of Workforce Development, to learn about Wisconsin's implementation of the DCA Pilot Project. Most of their time was spent meeting with John Mand and Chet Frederick of the Wisconsin Department. The visitors also met and talked with the DCA investigators in the Milwaukee office. During the morning of August 4, Estes, Woodbury, and Wright reviewed 4 or 5 DCA cases representing each type of denial, and found them to be well-documented. John Mand was available to answer questions about Wisconsin law and practice. The visiting team concluded that the quality of the investigations was excellent.

During the site visit, John Mand and Chet Frederick raised the following issues, many of which are discussed at greater length in Mand's "Denials Closeout Conference Notes" (Mand 1998).

1. Staffing and Time Needed for Investigation. As was true in other states, Wisconsin found that two DCA investigators could not handle a case load of 600 denials (200 denials of each type) per year. Rather, it appeared that 3 investigator positions would need to be allocated to manage such a case load. Wisconsin estimates that 2 investigator positions would be able to support an annual sample of 450 cases (150 of each type of

denial). An allocation for supervision and support staff would also need to be included.

2. Time Lapse Standards for Case Completion. Wisconsin favored timeliness standards for Denials, similar to the standards established for BQC cases. However, there was a concern that the higher level of nonresponse among claimants in the DCA Pilot Project may create greater problems in meeting time-lapse standards in the denied claims accuracy program than in the existing BQC (for which response rates are higher). For BQC cases, Wisconsin has experienced response rates around 98 percent. In contrast, Wisconsin's claimant response rate during the DCA Pilot Project was about 78 percent. Because of this, Wisconsin "reopened" a higher proportion of DCA cases than of BQC cases, as claimants eventually responded (for example, when they later establish eligibility). Under BQC, reopening a case (for example, to recode the claimant's prior "missing" data elements) resets the timeliness clock, occasionally losing the 60 or 90 day timeliness that had previously been met on the case. With DCA, the higher proportion of cases "reopened" would make it unlikely that BQC timeliness standards could be met. Two possible approaches would be (a) to set more lenient standards, or (b) add a "reopen code" (one that does not reset the timeliness clock) for the purpose of recoding a case due to a claimant's later response.

3. Timing of Sampling Denials. Other states felt that problems could be avoided by sampling denials (particularly monetary denials) one to two weeks later than was done during the DCA Pilot. Wisconsin staff, however, did not share this view. Rather, Wisconsin staff favored sampling denials immediately after each week's production has been completed, with no lag in sampling. This is the existing procedure with BQC, and was the procedure for the DCA Pilot Project. Wisconsin staff suggest that, with no delay in sampling, they were often in contact with the parties to the denial decision within the initial 14-day appeal period. This gave investigators the opportunity to answer questions, explain the decision and its effect, and advise the parties as to their appeal rights should they choose to exercise them.

4. Data Collection Instrument and Coding. Wisconsin staff had a variety of suggestions for improving the DCI. Some of these suggestions concerned the attempt, which is embodied in the DCI, to estimate the dollar impact of erroneous denials. Wisconsin staff agreed with other states on the difficulties inherent in attempting such estimates.

In addition, Wisconsin agreed with other states that many fields in the DCI should be programmed to default to prespecified values for each type of denial investigation. For example, in nonseparation cases, a series of possible data elements should be coded as "not applicable."

5. Programming. Wisconsin reported very little difficulty with programming, and the DCA sample selection program performed well. There was one sampling error during the first week of sampling, which came from an invalid nonmonetary issue family. The problem was corrected by removing that issue family from the programming. The invalid case was replaced by a new case from the last batch sampled.

6. Quality Performance Indicator. The views of Wisconsin staff on QPI are similar to the views of agency staff in other states. The essence of the view is that QPI merely reviews the information that already exists in a case file and ensures that proper processes were followed in making a determination. A DCA investigation, on the other hand, attempts to verify the facts that are in a case file (which are sometimes found to be incorrect) and to secure missing information and additional information (for example, statements from the claimant or employer that may not be in the file, rebuttals, and other information not previously obtained). The DCA investigation then determines the correctness of a denial in light of the verified and more complete set of information that is then available. The QPI process is viewed as a good indicator of the "quality" of an investigation, but an inadequate indicator of whether a decision is proper.

7. Value of the Investigating the Accuracy of Denials Claims. In his memo summarizing Wisconsin's experience with the DCA Pilot, John Mand states that, "the Denials review has brought to light a number of problems, either in process or procedure, that were not easily recognizable, if at all, from the BQC review. This has much to do with the fact that we're looking at a claim and/or case from different perspectives. A payment case has often overcome some of these problems to get there, while the denials case has found some of these problems to be fatal to ever establishing entitlement/benefit payment. In concert, I think the BQC and Denials reviews give us a much better overview of the UI program, than either one can individually."

8. Denials Forms. Wisconsin staff noted that, because of the short preparation time in implementing the Denials Pilot, they had adapted their forms and procedures directly from BQC. They believed that these could be refined to be more specific to investigating Denied Claims Accuracy. In particular, they were consolidating their three denial coding "worksheets" into one worksheet, grouping together data elements that are common to all types, those that are specific to one type, and following the input flow of the entry screens and the coding manual.

It appears that most of the Pilot states used three types of Claimant Questionnaires, adapted from BQC, for their monetary, separation and nonseparation denials. Wisconsin's staff has been debating the merits of a single questionnaire adapted from BQC, compared with three separate questionnaires, one for each type of denial investigation.

3.6. Summary of Findings from the Site Visits

This section brings together the observations that were gathered during the site visits and summarizes the states' comments and concerns about the DCA Pilot Project. With only minor exceptions, the states' observations agree with each other. The states' comments were in general made with an eye to nationwide implementation of DCA measurement. Accordingly, this summary attempts to identify the issues that require consideration before nationwide implementation of the DCA program.

1. Staffing and Time Needed for Investigation. The pilot states reported that the time required to complete DCA cases was only somewhat less than BQC paid claims investigations. As a result, they suggest that the samples sizes used in the DCA Pilot Project will need to be reduced for national implementation. Samples of 200 of each type of denial (600 total) were drawn for the DCA Pilot Project, based on an estimated two staff years to complete the cases. The consensus of the pilot states, however, was that 3 staff years would be needed to handle that case load. Sample sizes will have to reflect resource requirements and resource availability.

A related issue is the relative sizes of the DCA and paid claims samples. Although in the pilot, only 200 cases of each denial type were samples, the aggregate DCA samples were 600 cases annually, compared with 480 paid claims in the "core" BQC program. In addition, the ten states with the smallest UI workload currently sample only 360 BQC cases per year. It may be necessary to reconsider DCA sample sizes for the ten smallest states in light of the current BQC sample differential.

2. Response Rates. The pilot states all report lower response rates from claimants in their DCA investigations than they experience in their core BQC investigations. The reason is clearly that denied claimants – especially those denied for monetary and separation reasons – have little incentive to cooperate with the agency. As a result, claimant response rates to the DCA investigations were in the range of 74 to 81 percent (except in South Carolina, where they were 97 percent or higher). This compares with claimant response rates over 95 percent for core BQC investigations. (Note that the response rates discussed in this section are based on discussions that took place during the site visits. More complete and detailed data on claimant response rates are discussed in section 4.5 below.)

3. Time Lapse Standards for Case Completion. Unlike the core BQC program, there were no standards for timely completion of cases in the DCA Pilot Project. The pilot states reported that difficulty in obtaining responses from claimants was the main cause of delays in completing cases. The pilot states do favor timeliness standards, recognizing that standards for DCA may need to be structured somewhat differently than existing BQC timeliness standards. Wisconsin suggested either setting more lenient standards or adding a "reopen code" (one that does not reset the timeliness clock) for the purpose of recoding a case due to a claimant's later response.

4. Timing of Sampling of Denials. With the exception of Wisconsin, the pilot states agree that the sampling of monetary denials should be delayed for two weeks (ten work days) from the date that the claim was filed. Such a delay would avoid including in the samples to be investigated claims that were initially denied but that will be redetermined based on confirmation of additional wages for Combined Wage Claims (intrastate claims that combine earnings from one or more employers from another state in order to establish monetary eligibility or to yield the highest possible benefit amount for the claimant), interstate claims, and (in some states) alternative base year qualification.

5. Interstate Claims. The pilot states reported no greater difficulties with investigations involving interstate claims than with those involving intrastate claims. Indeed, in some cases, the pilot states found interstate claims easier to handle because documentation could be obtained from a central unit that processes interstate claims.

6. Quality Performance Indicator. During the site visits, staffs of the pilot states generally indicated that they prefer for intensive investigation of denied claims (as in the DCA Pilot Project) to the Quality Performance Indicator rating the nonmonetarily denials. Their reasons for this preference varied, but in general they believe that the QPI rating, which only scores the "quality" of the process that led to a nonmonetary denial, can be misinterpreted as an indicator of whether denials are correct or incorrect. But the QPI is emphatically not such an indicator. The pilot states, in other words, want to measure the accuracy of denials through intensive investigations, since such a process will result in a measure of the accuracy of denied claims that is what it appears to be.

7. Programming. The pilot states reported only minor (and easily resolved) start-up problems in implementing the COBOL sampling program and getting the DCA Pilot Project running.

8. Data Collection Instrument and Coding. Most of the pilot states had suggestions for changes in the Data Collection Instrument. Many of the suggested changes pertained to allowing fields in the DCI to be programmed to default (or to be "stamped") to prespecified values for each type of denial investigation. Another frequently noted problem was that the Data Collection Instrument accepts only three digits for a monetary denial error. Finally, some states have specific issues that must be accommodated in the Data Collection Instrument. For example, because New Jersey receives many "pro forma" filings, a separate action code should be established for UI claims filed as a requirement for establishing a welfare, disability, Supplemental Security Income, or other social program claim. This would allow a state and the Department of Labor to identify and remove these cases fro