Prepared for:
Division of Performance Review
Office of Workforce Security
Employment and Training Administration
U.S. Department of Labor
200 Constitution Avenue, N.W. – Room S-4231
Washington, D.C. 20210

Submitted by:
Stephen Woodbury
W.E. Upjohn Institute, 300 South Westnedge Avenue, Kalamazoo, MI 49007
(616/343-5541, woodbury@we.upjohninst.org) and
Department of Economics, Michigan State University, East Lansing, MI 48824
(517/355-4587, woodbur2@msu.edu)
and
Wayne Vroman
The Urban Institute, 2100 M Street, NW, Washington, DC 20037
(202/261-5573, wvroman@urban.ui.org)

This report was prepared under Contract No. M-5538-5-00-97-30. The authors are grateful to Burman Skrable and Andrew Spisak of the Office of Performance Review for extensive discussion and comments, and to Leland Teal, Layne Waters, Bob Branham, and Doug Potter of the South Carolina Department of Employment Security for their cooperation and efforts in making available the supplemental data used in chapter 5.

Executive Summary
List of Tables
Introduction
1. QPI versus DCA: The Value-Added of DCA
1.1. "Undetectable" Errors Discovered by DCA Investigations
1.2. Point of Detection of Erroneous Denials and the QPI
1.3. What If a Claimant Changes His or Her Story for the DCA Investigation?
1.4. Implications of QPI and DCA for States' Procedures
1.5. Summary and Conclusions
2. Can the QPI Review Mimic the Findings of DCA Investigations?
2.1. Lowering the QPI Pass/Fail Threshold
2.2. Eliminating Apparently Misleading Elements from the QPI
2.3. Discriminant Analysis of the QPI and Denied Claims Accuracy
Appendix to Chapter 2: Nonlinear Transformation of Least Squares Coefficients to Obtain Weights of the Discriminant Function
3. Erroneous Denials Compared with Overpayments
3.1. Erroneous Denials and Payment Errors
3.2. Why the Differences between Overpayment and Denial Error Rates?
3.3. Claimant Characteristics and Denial Probabilities
4. Benefits Lost Due to Erroneous Denials
4.1. Issues in Estimating Lost Benefits
4.2. Three Approaches to Estimating Lost Benefits<br> 4.3. Description of Penalties
4.4. National Estimates
5. Regression-Based Estimates of Benefits Received by Erroneously Denied Claimants
5.1. Description of the Data and Samples
5.2. Modeling Benefits Received and Benefit Duration
5.3. Expected Benefits and Benefit Duration of Erroneously Denied Claimants
5.4. Implications for Benefits Lost Due to Erroneous Denials
5.5. Summary and Conclusion
References

This follow-up report addresses several questions and issues surrounding the accuracy of denied claims for Unemployment Insurance (UI) that either were not addressed, or were addressed only in a preliminary way, in the Denied Claims Accuracy Pilot Project Final Report (Woodbury and Vroman 1999). Chapter 1 explores further the relationship between the findings of the Denied Claims Accuracy (DCA) intensive field audit and the scoring of denied cases under the Quality Performance Indicator (QPI) review. In each of the five pilot states, approximately 100 Separation and Nonseparation denials were subjected to both the DCA intensive field audit and the QPI review. These parallel reviews provide a rich source of information for examining the correlation between the DCA findings and QPI scoring.

Chapter 1 reaches three main conclusions. First, the great majority of separation and nonseparation errors result from agency error or some kind. Second, erroneous separation and nonseparation denials that passed QPI were more likely to involve an issue that the agency could not detect, indicating that the DCA tends to pick up errors that the QPI misses. That is, the QPI tends to miss a significant subset of problems in the separation and nonseparation determinations processes. In particular, the findings suggest that that the QPI cannot detect errors that require new information, especially the type of information that would be obtained from interviews with claimants or third parties. Third, in addition to giving an upward-biased picture of the extent to which the determinations process is flawed (as discussed in the May 1999 Denied Claims Accuracy Pilot Project Final Report), the QPI understates the extent to which incorrect action is a problem and overstates the extent to which inadequate information is a problem. That is, although the outcomes of the QPI review and the DCA investigation point to the same main problems with the determination process, an administrator using the QPI alone would have difficulty correctly allocating resources to improving the quality of decisions made by adjudicators relative to improving the information on which decisions are based.

Chapter 2 attempts to modify the QPI so as to mimic the results obtained by DCA investigations. The question addressed is whether the relatively inexpensive QPI review could substitute for the relatively expensive DCA audit. Several methods of modifying and adjusting the QPI are examined, including discriminant analysis. We conclude that, although the QPI could be modified so as to reduce the number and proportion of proper denials that fail, it cannot be modified so as to reduce the number or proportion of erroneous denials that pass. Under the best circumstances QPI is capable of classifying at most 41 percent of erroneous nonmonetary denials as failing.

Chapter 3 examines two related questions. First, we compare the denial error rates found by the DCA pilot project with the overpayment error rates found under the BAM program in the same five pilot states during the same time period as the DCA. The main finding is that total overpayment rates are sharply lower than erroneous denial rates. For example, in the five states taken together, the total overpayment rate on monetary determinations was 0.6 percent, compared with a monetary denial error rate of 16 percent. That is, the monetary denial error rate was nearly 27 times the total overpayment rate on monetary determinations. Also, total overpayment errors result mainly from lack of information rather than from human error. This contrasts with the situation for erroneous denials, where errors of judgment appear to play a larger role.

Second, chapter 3 examines whether the characteristics of claimants are related to the probabilities of erroneous denial or total overpayment. The results suggest four main conclusions: (a) Claimants whose earnings history puts them near the threshold of benefit eligibility are more likely to receive an erroneous monetary denial than are other claimants. (b) Adjudicators may incorrectly use a relatively weak earnings history as an indicator that a claimant does not meet the separation or nonseparation conditions for eligibility. (c) Claimants who are at the maximum potential duration, but whose WBA is below the maximum, are more likely than others to receive a total overpayment. (d) Individual characteristics such as race and gender may play a role in erroneous denials and total overpayments, although the evidence is not strong on this point.

Chapter 4 examines the benefits lost to claimants due to erroneous denials. As discussed in the May 1999 final report, a variety of problems arise in estimating the benefit losses due to erroneous denials. Four of these are reviewed and discussed in section 4.1: self-correction of initial administrative errors, the interconnectedness of error corrections, estimating the cost per case, and aggregation issues. Section 4.2 discusses three methods of estimating the dollar costs of denied claims: (a) a key week approach, (b) a benefit year approach, which is used in this report, and (c) a hybrid approach. Section 4.3 describes the penalties associated with each of the three types of denials. Finally section 4.4 presents estimates of the benefits lost by claimants due to erroneous denials. The estimates suggest that, overall, about $625 million in benefits were erroneously denied during fiscal year 1998, amounting to just over 3 percent of total regular UI benefit payments. Of this total, about $240 million were erroneously denied due to incorrect monetary determinations, about $230 million were erroneously denied due to incorrect separation determinations, and about $150 million were erroneously denied due to incorrect nonseparation determinations.

Chapter 5 outlines and implements a regression strategy for estimating the benefits that erroneously denied UI claimants would have received during the full benefit year, had they been correctly determined eligible. The main practical barrier to such an approach is lack of data on the full benefit-year experience of a sample of eligible claimants. This was overcome with the cooperation of the South Carolina Quality Control Division, which provided data on the full benefit-year payments made to its BAM sample during the period of the DCA Pilot Project. The supplemental data from South Carolina allows estimation of models of benefits received that serve as the basis for imputing the benefits (and weeks of benefits) that would have been received by erroneously denied claimants if they had not been denied.

Imputations based on the estimated models (see section 5.3) suggest that the benefits lost by erroneously denied claimants (as a percentage of the benefits received by a typical correctly determined claimant) amount to just under 80 percent for erroneous monetary denials and about 55 percent for erroneous separation denials. (The weeks of benefits lost as a percentage of the weeks of benefits received by a typical correctly determined claimant amount to 91 percent for erroneous monetary denials and 67 to 68 percent for erroneous separation denials.) These findings imply that the total lost benefits due to erroneous denials amount to about $565 million in fiscal year 1998, or about 3.1 percent of total regular UI benefit payments (section 5.4). Of this total, about $220 million were erroneously denied due to incorrect monetary determinations, about $190 million were erroneously denied due to incorrect separation determinations, and about $155 million were erroneously denied due to incorrect nonseparation determinations. These estimates are only slightly less than the estimate of lost benefits developed in chapter 4.

List of Tables
Table 1-1: Prior agency action on erroneous denials, by state and type of denial
Table 1-2: Prior agency action on erroneous denials, by type of denial and whether erroneous denials passed QPI
Table 1-3: Error detection point on erroneous denials, by type of denial and whether erroneous denials passed QPI
Table 1-4: Prior agency action on erroneous denials by detection point, erroneous separation and nonseparation denials
Table 1-5: Percentages of erroneous denials that failed each element of the Quality Performance Indicator review, by type of denial
Table 1-6: Percentages of "undetectable" erroneous denials that failed each element of the Quality Performance Indicator review, by type of denial
Table 2-1: Crosstabulations of separation denial accuracy by alternative QPI pass/fail scores
Table 2-2: Crosstabulations of nonseparation denial accuracy by alternative QPI pass/fail scores
Table 2-3: Components of QPI as a predictor of denied claims accuracy
Table 2-4: Crosstabulations of separation denial accuracy by QPI-based discriminant function pass/fail scores
Table 2-5: Crosstabulations of nonseparation denial accuracy by QPI-based discriminant function pass/fail scores
Table 3-1: Error rates on denied claims (DCA pilot) and paid claims (BAM) in DCA pilot states, September 1997 through August 1998
Table 3-2: Prior agency action on claims "totally overpaid" (BAM) in DCA pilot states, September 1997 through August 1998
Table 3-3: Prior agency action on erroneous denials (DCA pilot) and claims "totally overpaid" (BAM) in all five DCA pilot states, September 1997 through August 1998
Table 3-4: Relationships between claimant characteristics and the unconditional probability of (a) erroneous monetary denial, (b) erroneous separation denial, (c) erroneous nonseparation denial, and (d) total overpayment: regression analysis
Table 4-1: Denial error rates, unadjusted and adjusted for "self-corrections," by pilot project and issue
Table 4-2: Framework for viewing the duration of erroneous denials
Table 4-3: Penalties for separation issues
Table 4-4: Determinations and denials for the United States, and imputations of erroneous denials for the United States based on DCA pilot project, 1998
Table 4-5: Estimates of benefits lost due to erroneous denials, United States, fiscal year 1998
Table 4-6: Mean nonseparation penalty periods, imputations for the five pilot states and the United States, 1998
Table 5-1: Characteristics of UI claimants correctly paid and erroneously denied for monetary, separation, and nonseparation reasons, South Carolina, 1997-98
Table 5-2: Models of benefits paid and number of weeks paid during the benefit year, various groups of UI claimants, South Carolina, 1997-98
Table 5-3: Observed and expected benefits paid and number of weeks paid during the benefit year, various groups of UI claimants, South Carolina, 1997-98
Table 5-4: Imputed benefits (and weeks of benefits) lost by erroneously denied claimants as a percentage of benefits received by correctly determined claimants
Table 5-5: Estimates of benefits lost due to erroneous denials, United States, fiscal year 1998

This follow-up report addresses several questions and issues surrounding the accuracy of denied claims for Unemployment Insurance (UI) that either were not addressed, or were addressed only in a preliminary way, in the Denied Claims Accuracy Pilot Project Final Report (May 1999). The Office of Performance Review requested the contractor to pursue further research on these questions, and this report describes the findings of that follow-up work.

The first two chapters of this follow-up explore further the relationship between the findings of the Denied Claims Accuracy (DCA) intensive field audit and the scoring of denied cases under the Quality Performance Indicator (QPI) review. In each of the five pilot states, approximately 100 Separation and Nonseparation denials were subjected to both the DCA intensive field audit and the QPI review. These parallel reviews provide a rich source of information for examining the correlation between the DCA findings and QPI scoring. Chapter 1 offers a descriptive discussion of the valueadded of the DCA investigation and concludes that the DCA provides new information without which a substantial number of separation and nonseparation denial errors could not be detected or diagnosed. That is, a desk audit like the QPI review misses such errors because it relies on information on file with the agency. Chapter 2 attempts to modify the QPI so as to mimic the results obtained by DCA investigations. The question addressed here is whether the relatively inexpensive QPI review could substitute for the relatively expensive DCA audit. (The conclusion is that it could not.)

Chapter 3 examines two related questions. Sections 3.1 and 3.2 compare the denial error rates found by the DCA pilot project with the overpayment error rates found under the BAM program in the same five pilot states during the same time period as the DCA. The main purpose is to examine differences between denial and overpayment error rates and to understand what factors account for the differences. Section 3.3 examines whether the characteristics of claimants are related to the probabilities of erroneous denial or total overpayment. The DCA Pilot data are pooled with BAM data in order to estimate four models: one each for the probability of erroneous monetary, separation, and nonseparation denial, and one for the probability of overpayment.

Chapters 4 and 5 examine the benefits lost by claimants due to erroneous denials. As discussed in the May 1999 final report, a variety of problems arise in estimating the dollar impact of erroneous denials. Chapters 4 and 5 of this follow-up report addresses the various problems and offers some estimates that are based on plausible assumptions.

A main goal of the DCA Pilot Project was to compare the results of a comprehensive field investigation (the Denied Claims Accuracy audit) with the Quality Performance Indicator (QPI) assessment of nonmonetary determinations. Section 4.6 of the Denied Claims Accuracy Pilot Project Final Report (May 1999) developed some basic comparisons of the DCA findings and the QPI assessment, and this chapter pursues the topic further. As will be seen, the DCA codings of prior agency action and point of detection of erroneous denials provide a way of diagnosing each erroneous denial and relating the circumstances surrounding each back to the QPI review.

Section 1.1 uses DCA data on prior agency action to address the following questions: What percentage of erroneous denials did the DCA investigation determine the agency did not detect or cannot detect? What does the DCA investigation show about the effectiveness of the QPI in detecting problems in the separation and nonseparation determinations processes?

Section 1.2 uses DCA data on the point of error detection in the DCA investigation to examine further the causes of denials that were found improper by the DCA investigation. In particular, we compare the error detection point in erroneous denials that passed the QPI with the error detection point in erroneous denials that failed the QPI, with an eye to understanding differences between DCA and QPI in the types of error that each can detect. One goal is to understand better why DCA and QPI give such different results.

Section 1.3 is a brief digression on whether claimants who give a different story to the DCA investigator than they gave in the original fact-finding could pose a problem for the DCA method.

Section 1.4 attempts to pull together what has been learned about the implications of the QPI and the DCA for modifications that the states could make, either to improve error detection or to prevent denial errors in the first place. The section discusses what the QPI review and the DCA determination each can reveal about errors in separation and nonseparation denials.

Chapter 1 has two related goals. The first is to understand better the relationship between the DCA findings and the QPI review. Given that the QPI is a process-oriented review, whereas the DCA field audit is an outcome-oriented determination, how do these two reviews supplement and complement one another? The second goal of chapter 1 is to evaluate whether the DCA could lead states to take actions other than those the QPI review would indicate. The DCA is more expensive than the QPI. What is the value-added of the DCA? Is there a reasonable expectation that state agencies would act on the additional information offered by the DCA?

1.1. "Undetectable" Errors Discovered by DCA Investigations

An important question about including separation and nonseparation determinations in the DCA program is whether the DCA investigation will yield additional information that contributes to the improvement of each state's program. In this section, we examine (a) the state agency actions that occurred before the DCA investigation and (b) the point in the DCA investigation at which an error was discovered. Cross-tabulating these two variables provides insights into the value of the DCA investigation. Two questions in particular are addressed: What is the percentage of erroneous denials that DCA determined the agency did not detect or cannot detect (and which the QPI review would not detect)? What does the QPI review show about state agency efforts to obtain information on such cases?

1.1.1. Prior Agency Action on Erroneous Denials. For each erroneous denial, DCA investigators classified the state agency action that occurred before the DCA investigation into one of six categories:

• Could not detect issue: Official procedures had been followed and forms had been fully completed but the error issue could not be detected by normal procedures.
• Was already resolving issue: The agency was in the process of resolving the error issue and took correct action before the DCA investigation was completed, or the agency had correctly resolved the error issue before the sample was selected.
• Took incorrect action: The agency identified the error issue before the sample was selected but took incorrect action.
• Did not identify issue: The agency had enough documentation to identify that there was an error issue but did not resolve the error issue.
• Did not follow procedures: Official procedures had not been followed or official forms had not been properly completed by the agency, making it impossible to detect the error issue.

The first category (could not detect issue) is especially important because it indicates that the agency could not uncover the error issue even though it followed its normal procedures. The error was discovered only through the DCA field investigation. It follows that one or more aspects of the agency's existing procedures should be reviewed with an eye to making changes that would result in more complete fact finding.

Similarly, the second category (was already resolving issue) suggests that the agency needs to review its procedures to see whether improvements could be made that would speed collection of information needed to make a fully informed decision. This category also alerts the agency to incomplete or inaccurate initial decisions that require additional work by the agency for a correct determination.

In contrast, the last three categories indicate that the agency had enough information to identify and resolve the error issue but either failed to do so, took incorrect action, or did not follow its own procedures. Knowing this is clearly useful to managers and administrators in identifying aspects of program operations that require correction.

Table 1-1 shows the actions that were being taken (or had already been taken) by each state agency for each type of denial at the time of the DCA investigation. Although the focus is on separation and nonseparation denials, Table 1-1 also includes monetary denials so that comparisons can be drawn across all three type of denials. (The figures on erroneous monetary denials will be referred to again in chapter 3.)

The first panel of Table 1-1 shows that prior agency actions on monetary denials vary significantly among the five pilot states. In three states (Nebraska, South Carolina, and West Virginia), the error-causing issue could not have been detected through normal procedures for over half of the erroneous monetary denials. Also, in New Jersey, South Carolina, and Wisconsin, from 30 to 50 percent of the erroneous monetary denials were in the process of being corrected by the agency. Finally, in 15 to 55 percent of erroneous monetary denials, the agency (a) identified the issue but took incorrect action, (b) had adequate documentation to identify the issue but did not do so, or (c) did not follow official procedures. In sum, the first panel of Table 1-1 suggests that, although existing agency procedures would have resolved about 30 percent of the erroneous monetary denials, nearly 40 percent of the errors could not have been detected under existing procedures, and the remaining 30 percent of monetary denial errors result from incorrect action or failure to identify the issue or to follow procedures.

The second panel of Table 1-1 shows the actions taken by states on erroneous separation denials. There are three main findings. First, the error-causing issue could not have been detected through normal procedures for about 20 percent of the erroneous separation denials overall. Second, only 8 percent (7 out of 86) of erroneous separation denials were in the process of being corrected by the agency at the time of the DCA investigation. Overall, erroneous separation denials appear less likely to be corrected by agency actions than erroneous monetary denials. Third, in all five pilot states, the agency took incorrect action, did not identify the issue, or did not follow procedures in 60 percent of more of the erroneous separation denials.

Similarly, the findings on erroneous nonseparation denials show that nearly 70 percent of the erroneous nonseparation denials involved the agency taking an incorrect action, not identifying an issue, or not following official procedures. Only 22 percent of the erroneous nonseparation denials were undetectable. As with erroneous separation denials, relatively few erroneous nonseparation denials were in the process of being corrected by the agency (under 9 percent).

In sum, the findings in Table 1-1 suggest that the great majority of erroneous separation and nonseparation denials result from agency error of some kind. This is in sharp contrast to monetary denial errors, of which only 30 percent result from agency error.

1.1.2. Prior Agency Action and the QPI Review. By means of the prior agency action code, the DCA investigation provides program managers with information that can be used to improve the system of determining UI eligibility. Can the QPI review provide the same or similar information?

Table 1-2 shows, for the five pilot states combined, the prior agency action taken by state agencies in separation and nonseparation denials, and for each prior action shows the percentage of improper denials that passed the QPI review. The purpose of the table is to suggest the extent to which the QPI review could identify the error issues revealed by the DCA investigation, and whether the QPI's potential effectiveness varies by prior agency action.

As already noted, for the majority of erroneous separation and nonseparation denials, state agencies took incorrect action, did not identify an issue, or did not follow agency procedures (see the "total" column in Table 1-2). However, the mix of prior agency action differs between denial errors that passed the QPI review and those that failed the QPI review. The erroneous denials that passed QPI were more likely to involve an issue that the agency could not detect. For example, whereas 11 percent of the separation denial errors that failed QPI involved an "undetectable" error, about 35 percent of the separation denial errors that passed QPI involved an "undetectable" error. Similarly, about 14 percent of the nonseparation errors that failed QPI involved an "undetectable" error, whereas 32 percent of the nonseparation errors that passed QPI involved an "undetectable" error. This is clear evidence that the DCA investigation tends to "catch" errors that the QPI cannot detect. Stated differently, the DCA investigation has the clear potential to uncover and add information to the process of improving the system's performance.

1.2. Point of Detection of Erroneous Denials and the QPI

The DCA investigation also gives information on the method by which each case error was discovered — the error detection point. As can be seen in Table 1-3, four detection points are coded in the Data Collection Instrument: verification of wages or separation, the claimant interview, through a third party, and UI records. Table 1-3 suggests that UI records were the most common way of detecting erroneous separation and nonseparation denials: 45 percent of separation errors were detected through UI records, and nearly 63 percent of nonseparation errors were detected though UI records. The claimant interview also played a significant role in detecting both separation and nonseparation errors, and verification of wages and separation played a significant role in detecting separation errors.

Table 1-3 also breaks down the error detection point by whether the case passed or failed the QPI review. This breakdown suggests that the error detection point for a case that fails the QPI review is most likely to be either verification of wages/separation or UI records. (The column percentages for these two error detection points exceed the overall row percentages for failing QPI in both the separation and nonseparation denials.) This suggests again the value and importance of the DCA field audit in detecting errors. By its nature, the QPI review tends to pick up errors that can be detected by examining agency records that are on hand — although it is still true that the QPI passes over 25 percent of erroneous separation denials that were detected from UI records and over 40 percent of erroneous nonseparation denials that were detected from UI records. (These are cases, in general, where the agency had adequate information but took the wrong action.) The QPI, however, cannot detect errors that require new information from, for example, claimant interviews and third parties.

1.3. What If a Claimant Changes His or Her Story for the DCA Investigation?

A concern that has been raised about the DCA investigation is that a claimant could give a different story to the DCA investigator than was given in the original factfinding. If this were to occur, the argument goes, then the DCA investigator could conclude that the determination was originally incorrect, but this would unfairly impugn the original investigation because it was the claimant's testimony that originally misled the determination.

One way to appraise whether this argument should be a serious concern is to look at data on point of detection jointly with data on prior agency action. This is done in Table 1-4. If a claimant changes his or her story, then the DCA point of detection should be "claimant interview," and the prior agency action should be "not detectable." In Table 1-4, it can be seen that only 6 out of the 86 erroneous separation denials (or 7 percent), and 15 out of the 149 nonseparation denials (or 10 percent), fit this description.

Although these figures suggest that "story-switching" is not a major problem, the number of erroneous denials that fit this description may give an imperfect idea of the extent to which "story-changing" could be a problem. First, the claimant interview could be the detection point in erroneous denials that were coded as "undetectable" even if the claimant did not change his or her story. If so, then the number of cases that fit the above description overstates the number of "story-switchers." On the other hand, a claimant could switch his or her story without the claimant interview being the detection point. But in this latter case, the change in the story would be a moot point because the error was found by some other means.

The conclusion is that the figures given above (7 percent for separation denials and 10 percent for nonseparation denials) are upper-bound estimates of the extent to which story-switching might be responsible for error detection. Based on the evidence, then, it would be difficult to conclude that "story-switching" is a serious problem.

1.4. Implications of QPI and DCA for States' Procedures

Most of the evidence to this point has suggested that the DCA has the potential to offer information that the QPI could not provide. However, the QPI also has the potential to serve as a tool in detecting errors and diagnosing problems with a state UI system. As discussed in greater detail in the next chapter, the QPI scores each case along six lines:

• the adequacy of information obtained from the claimant
• the adequacy of information obtained from the employer
• the adequacy of information obtained from others
• whether the adjudicator provided the opportunity for rebuttal to the applicable parties
• whether the nonmonetary determination met the provisions of state law and/or policy
• the adequacy of the written determination

Accordingly, it should be useful to examine denials that DCA found to be in error in light of what the QPI review found, with an eye to understanding what the QPI would suggest to an administrator about problems in eligibility determination.

Table 1-5 displays, for the separation and nonseparation denials that DCA found to be in error, tabulations of the QPI findings on each of the six aspects of a case listed above. The first column shows that the QPI review failed two-thirds of the separation errors for not meeting the provisions of law and policy. (Under the QPI scoring system, a determination that fails law and policy fails the QPI; therefore, the same two-thirds of these denials, all of which were erroneous, failed the QPI — see the bottom row.) Also, the QPI review indicated that for over one-third of the separation error cases, claimant information and/or employer information was inadequate or missing, and the written determination was inadequate or wrong.

The second column shows that the QPI review failed over half (54.5 percent) of the nonseparation errors for not meeting the provisions of law and policy. (The same 54.5 percent of these erroneous denials failed the QPI — see again the bottom row.) Also, the QPI review indicated that for nearly 30 percent of the nonseparation error cases, claimant information was inadequate or missing, and for nearly 40 percent of the cases, the written determination was inadequate or wrong.

Looking at the QPI results in Table 1-5, one would conclude that there are two main problems with the separation and nonseparation determination processes: incorrect application of the state's law and policy, and inadequate or missing information (from the claimant in both separation and nonseparation denials, and especially from employers in separation denials). In effect, these problems are similar to the main problems identified by the DCA in its investigation: incorrect actions taken by the agency, and inability to detect issues (usually due to incomplete information). Based on such findings, one could perhaps defend the QPI and say that it identifies the same problems as are identified by the DCA.

The difficulty in such an argument is that the QPI offers no benchmark for identifying the actual extent and nature of determination outcomes that are erroneous. As already noted, overall, the QPI gives the impression that there are far more denial errors than actually occur (based on the DCA). Further, the QPI tends to miss denial errors that involve inadequate or incorrect information on file with the agency. This last point can be seen in Table 1-6, which shows the same information as Table 1-5 but for separation and nonseparation denials that were found to be "undetectable" as well as erroneous by the DCA investigation. Of the erroneous separation denials that were undetectable, the QPI review failed only 40 percent (as opposed to failing two-thirds of all erroneous separation denials). Similarly, of the erroneous nonseparation denials that were undetectable, the QPI review failed only 34 percent (as opposed to failing nearly 55 percent of all erroneous nonseparation denials). This suggests again that the QPI review has a difficult time with the undetectable cases — the cases for which agency procedures are relatively weak or for which information is inadequate.

It follows that, even though the outcomes of the QPI review and the DCA investigation point to the same main problems with the separation and nonseparation determination processes, the QPI is a rougher gauge of the extent of the problems. First, the QPI gives an upward-biased picture of the extent to which the determinations process is flawed. Second, the QPI overstates the extent to which incorrect action is a problem and understates the extent to which inadequate information is a problem. It follows that an administrator using the QPI alone would have trouble knowing the appropriate quantity of additional resources to devote to a problem, or of knowing when to stop increasing the resources devoted to a problem. Because the DCA is designed with the purpose of estimating the accuracy of denied claims, it seems natural to use the QPI in conjunction with DCA to gain a rounded picture of the accuracy and quality of nonmonetary determinations.

1.5. Summary and Conclusions

This chapter has focused on the potential value-added of the DCA investigation, relative to the existing QPI review, in helping states administer the UI program. In particular, it has attempted to draw out features of the DCA that would give states an advantage in detecting errors in separation and nonseparation denials. Section 1.1 used DCA data on prior agency action to examine the causes of denial errors and the extent to which the QPI could detect denial errors. That section drew two main conclusions. First, the great majority of separation and nonseparation errors result from agency error of some kind. Second, erroneous separation and nonseparation denials that passed QPI were more likely to involve an issue that the agency could not detect, indicating that the DCA tends to pick up errors that the QPI misses. That is, the QPI tends to miss a significant subset of problems in the separation and nonseparation determinations processes.

Section 1.2 used DCA data on point of error detection to examine further the causes of denial errors. Comparison of the error detection point in erroneous denials that passed the QPI with the error detection point in erroneous denials that failed the QPI confirmed the reasonable suspicion that the QPI cannot detect errors that require new information, especially the type of information that would be obtained from interviews with claimants or third parties.

Section 1.4 attempted to bring together the various analyses of the DCA and QPI and draw out their implications for practice. The main questions addressed are: What can the QPI review suggest about procedures that the states need to modify either to improve error detection or (preferably) to prevent denial errors in the first place? Can the DCA determination reveal additional information that the QPI cannot offer? In particular, are there gaps in the QPI approach that can be filled only with an intensive field audit like the DCA?

Section 1.4 highlighted two significant drawbacks of the QPI. First, the QPI gives an upward-biased picture of the extent to which the determinations process is flawed, as discussed in the Denied Claims Accuracy Pilot Project Final Report (May 1999). Second, the QPI understates the extent to which incorrect action is a problem and overstates the extent to which inadequate information is a problem. Although the outcomes of the QPI review and the DCA investigation point to the same main problems with the separation and nonseparation determination processes, an administrator using the QPI alone would have difficulty correctly allocating resources to improving the quality of decisions made by adjudicators relative to improving the information on which decisions are based. It seems natural to use the QPI and the DCA together to gain a rounded picture of the accuracy and quality of nonmonetary determinations.

2.1. Lowering the QPI Pass/Fail Threshold

The overall QPI score is calculated as the sum of scores on six components: adequacy of claimant information, adequacy of employer information, adequacy of other information, provision of opportunity for rebuttal, whether the determination meets the provision of state law and policy, and adequacy of the written determination. Accordingly, the overall QPI score (denoted simply as QPI) is calculated as: QPI = clmtinfo + empinfo + othinfo + rebutprv + lawpol + writdet, where:

clmtinfo = the case's score on the adequacy of information obtained from the claimant (0 if not available or missing, 5 if inadequate, and 10 if adequate or not applicable);

empinfo = the case's score on the adequacy of information obtained from the employer (0 if not available or missing, 5 if inadequate, and 10 if adequate or not applicable);

othinfo = the case's score on the adequacy of information obtained from others (0 if not available or missing, 5 if inadequate, and 10 if adequate or not applicable);

rebutprv = the case's score on whether the adjudicator provided the opportunity for rebuttal to the applicable parties (0 if opportunity not provided, 10 if opportunity provided or not applicable);

lawpol = the case's score on whether the non monetary determination met the provisions of state law and/or policy (0 if denial determination met provisions of state law and policy, 30 if determination questionable; 50 if determination met provisions of state law and policy);

writdet = the case's score on the adequacy of the written determination (0 if completely wrong, 5 if not adequate, 10 if adequate).

The overall QPI score cannot exceed 100.

The same six components are also used to calculate a "modified pass-fail" QPI score. To obtain the modified pass-fail score, the law and policy (lawpol) component is linked to the first four components in the following way. If a case receives fewer than 10 points for adequacy of claimant information (clmtinfo), adequacy of employer information (empinfo), adequacy of other information (othinfo), or provision of opportunity for rebuttal (rebutprv), then the score on law and policy cannot exceed 30 (that is, whether the case meets the provisions of state law and policy are at best "questionable"). For example, if a case scores 5 (inadequate) or 0 (not available or missing) on adequacy of claimant information, then the score on law and policy could be at most 30, and the sum of the six components would be at most 75. Only when the first four components receive scores of 10 is the law and policy component scored independently.

A QPI summary score of 80 or less is considered failing. Table 2-1 (panel A) shows a cross-tabulation of the accuracy of separation denials by whether the denial determination passed or failed QPI. (Panel A of Table 2-1 is repeated from Table 4-14 of the Final Report.) Of the 902 separation denial cases that were both investigated by DCA and had QPI appraisals, 603 were determined proper denials by DCA and passed QPI. Also, 54 denial cases were determined improper denials by DCA and failed QPI. But 218 cases that were proper denials (as determined by DCA) failed QPI, and 27 cases that were improper denials (as determined by DCA) passed QPI.

Table 2-2 (panel A) shows cross-tabulations of the accuracy of nonseparation denials by whether the denial determination passed or failed QPI. The findings are similar to those for separation denials: Of the 895 nonseparation denial cases that were both investigated by DCA and had QPI appraisals, 607 were determined proper denials by DCA and passed QPI. Also, 78 nonseparation denial cases were determined improper denials by DCA and failed QPI. But 145 cases that were proper denials (as determined by DCA) failed QPI, and 65 cases that were improper denials (as determined by DCA) passed QPI.

Two main conclusions follow from panels A of Tables 2-1 and 2-2. First, the QPI is only weakly correlated with the findings of the DCA investigations. A high proportion of erroneous denials pass QPI and a high proportion of proper denials fail QPI. Second, the QPI gives an excessively negative view of the extent to which denials are erroneous.

It is natural to ask whether either of these problems could be solved or mitigated simply by lowering the QPI pass/fail threshold — that is, by considering denial determinations with a QPI score of less than 80 to be passing. Panel B of Table 2-1 shows that, if the QPI pass/fail threshold were lowered to 65, only 14 percent of the separation denial determinations would fail QPI — much closer to the 9 percent error rate found for separation denials by the DCA investigations. However, lowering the QPI threshold also raises the number and proportion of erroneous separation denials that pass QPI. With a threshold of 80, one-third of the erroneous separation denials pass QPI, whereas with a threshold of 65, nearly two-thirds (63 percent) of erroneous separation denials pass QPI.

Panel B of Table 2-2 tells a similar story for nonseparation denials. If the QPI threshold were lowered to 70 for nonseparation determinations, only 17 percent of the nonseparation denial determinations would fail QPI — very close to the 16 percent error rate found for nonseparation denials by the DCA investigations. But again, lowering the QPI threshold raises the number and proportion of erroneous denials that pass QPI. With a threshold of 80, 45 percent of the erroneous nonseparation denials pass QPI, whereas with a threshold of 70, 59 percent of erroneous nonseparation denials pass QPI.

We conclude that lowering the QPI pass/fail threshold would not be a satisfactory way of modifying the QPI so as to improve its performance and a measure of the accuracy of non monetary denials. By correcting one problem with the QPI — the excessively negative impression the QPI gives of denial determinations — one would simply increase the number and proportion of erroneous denials that pass QPI.

2.2. Eliminating Apparently Misleading Elements from the QPI

The findings shown in panels A of Tables 2-1 and 2-2 show that, in addition to giving an overly negative view of denial determinations, the QPI is only weakly correlated with the findings of DCA investigations. It is also natural to ask whether the QPI could be modified in some way so as to improve its correlation with the DCA findings. For example, if some components of the QPI are highly correlated with the DCA findings, then perhaps these components could be used (and the others eliminated) so as to improve the QPI as a performance indicator.

Results presented in the Denied Claims Accuracy Pilot Project Final Report (pp. 89-91 and Table 4-18) show that three of the six components of the QPI — claimant information, employer information, and rebuttal opportunity provided — tend to be negatively related to the outcome of the DCA investigation. These basic results are repeated in Table 2-3, which shows probit regressions of the DCA outcome (proper or erroneous denial) on the individual components of the QPI. (A separate equation is estimated for separation and nonseparation denials. Table 2-3 differs somewhat from Table 4-18 the Final Report because is based on probit analysis, with coefficients transformed so that they can be interpreted as discrete linear changes).

A positive coefficient in Table 2-3 indicates that a high score on the QPI component is positively related to a correct determination, and conversely. The main conclusion from Table 2-3 is that separation denials that received a high score on the claimant information or rebuttal opportunity provided components of the QPI were actually more likely to be found in error by the DCA investigation. Also, nonseparation denials that received a high score on the claimant information component of the QPI (and, to some extent, the employer-information and rebuttal-opportunity-provided components) were more likely to be found in error by the DCA investigation. These conclusions are strongest for the adequacy of claimant information, whose coefficient is negative and significant at the 5-percent level in both the separation and nonseparation equations.

The results in Table 2-3 suggest that the QPI would perform better if some of its components — those that are negatively correlated with the findings of the DCA investigation — were eliminated. A simple method of testing whether the QPI could be modified to give results that more closely resemble the DCA results is to compute a modified QPI score that relies only on the three components of the QPI scoring that are positively correlated with the outcome of the DCA investigation: other information, law and policy, and the written determination.

Deleting the three components of the QPI that appear to be negatively correlated with the outcome of the DCA investigation yields the following modified QPI score (QPI_m):

QPI_m = (othinfo + lawpol + writdet) (10/7),

Panel C of Tables 2-1 and 2-2 shows crosstabs of this modified QPI with findings of the DCA investigations. With a pass/fail threshold of 80 (as with the conventional QPI) the modified QPI gives results that are essentially the same as the results from the conventional QPI — compare panels A and C in Table 2-1 for separation denials, and compare panels A and C in Table 2-2 for nonseparation denials. That is, eliminating the components of the QPI that are negatively related to the findings of the DCA investigations does not improve the performance of the QPI.

Would choosing a different threshold for the modified QPI help? Panel D of Tables 2-1 and 2-2 crosstabulates the DCA outcomes against the modified QPI but this time using a QPI pass/fail threshold of 65. The results in panel D of Table 2-1 (for separation denials) are quite similar to those in panel C (which used a threshold of 70 with the conventional QPI). And similarly, the results in panel D of Table 2-2 (for nonseparation denials) are quite similar to those in panel C (which used a threshold of 65 with the conventional QPI). Lowering the modified QPI pass/fail threshold can bring the total proportion of denials that fail QPI closer to the proportion of denials that are in error, but only by increasing the number and proportion of erroneous denials that pass QPI. (Recall that this was the outcome when the threshold was lowered for the conventional QPI — again, compare panels A and B in Tables 2-1 and 2-2.)

The conclusion is that a simple attempt to modify the QPI — eliminating components of the QPI that are negatively correlated with the findings of the DCA investigation — does not improve the performance of the QPI.

2.3. Discriminant Analysis of the QPI and Denied Claims Accuracy

An alternative method of examining whether the QPI scoring can be modified to mimic (or predict) the findings of the DCA investigation is to compute a discriminant function based on the QPI's components. The goal here is to form a linear combination of the components of the QPI such that the linear combination can be used to discriminate between correct and erroneous denials. This could be accomplished using a canned discriminant analysis package; however, because discriminant analysis is a variant of standard regression analysis, it is straightforward to develop the approach in a regression context. (This also has the advantage of making clear what is being done.)

2.3.1. The Discriminant Function and Assignment of Scores.

Consider using the sample of cases for which both DCA investigations were conducted and QPI scorings were performed to estimate the following regression model :

In (1), DCA is a zero–one indicator of whether the denial was correct or erroneous (1 if correct, 0 if erroneous), clmtinfo, empinfo, othinfo, rebutprv, lawpol, and writdet are the components of the QPI as already defined, the b's are regression coefficients to be estimated, and e is a random error term. Applying ordinary least squares to (1) yields estimates of the b's, which can be used as weights of a linear discriminant function: where L denotes the discriminant function and the b's are least-squares estimates of the b's in equation (1). These b's are weights that maximize the ability to discriminate between correct and erroneous denials. [Some practitioners advocate using a nonlinear transformation of the estimated coefficients in the discriminant analysis, as described in the appendix to this chapter. However, the results will be essentially similar to those described here.]

Once the weights of the discriminant function have been obtained, equation (2) can be used to compute a score for each denial case. This is done by substituting the values of the QPI components of each case into the discriminant function. Each case will have its own score, which can be referred to as Li. Cases with higher Li scores should be more likely to have been correct denials than cases with lower Li scores.

Next, a cut-off score, L*, must be chosen so that each case can be classified as either a "correct" denial or an "erroneous" denial based in its score. Cases with Li greater than the cut-off are classified as "correct," whereas those with Li less than the cut-off are classified as "erroneous." There are several possible ways of choosing a cut-off, but the simplest is to choose L* so that the proportions of cases assigned to "correct" and "erroneous" status based on the scores (Li) are the same as the proportions of correct and erroneous cases based on the DCA investigations. We follow this procedure below, but also check the sensitivity of the results to selection of the cut-off by examining a lower cut-off, as was done in Tables 2-1 and 2-2.

The final step is to compare the correspondence between the assignments based on the discriminant analysis with the findings of the DCA investigations. This can be done with the same type of 2-by-2 matrix that was used in Tables 2-1 and 2-2. To summarize, discriminant analysis of the QPI scoring proceeds in the following steps:

• Regress a zero–one indicator of whether a denial was correct or erroneous on the appropriately coded components of the QPI — equation (1).
• Use the estimated coefficients of equation (1) as weights in a discriminant function — equation (2).
• Obtain a score for each denied case by substituting the values of the QPI components of each case into the discriminant function.
• Select a cut-off score.
• Classify cases with scores above the cut-off as "correct" denials, and classify cases with scores below the cut-off as "erroneous" denials.
• Compare the outcomes obtained from classifying cases according to the discriminant analysis with the findings of the DCA investigation.

2.3.2. Results. Table 2-4 displays the results of the procedure outlined above for separation denials, and Table 2-5 does the same for nonseparation denials. The regression estimates that underlie the discriminant function scores used in Tables 2-4 and 2-5 are similar to those shown in Table 2-3, and hence are not displayed. (They differ from the estimates in Table 2-3 only because they are estimated by ordinary least squares and include, in addition to the six QPI components already discussed, a seventh — whether appeal information was provided to the claimant).

Panel A of Table 2-4 crosstabulates the discriminant function pass/fail score based on all QPI components (with a pass/fail threshold of 0.75) against the findings of the DCA investigation. (A cut-off of 0.75 assigns roughly the same proportions of separation denials to "pass" and "fail" status as the proportions of correct and erroneous cases found in the DCA investigations.) Panel B of Table 2-4 repeats panel B of Table 2-1 and is the proper comparison for Panel A of Table 2-4 because it shows the crosstabulation of QPI pass/fail scores (with a pass/fail threshold of 65) against the findings of the DCA investigation. (Recall that with a QPI pass/fail threshold of 65, only 14 percent of the separation denial determinations fail QPI, which is as close as we were able to bring the conventional QPI to the 9 percent error rate found for separation denials by the DCA investigations.)

Comparing the results in panels A and B of Table 2-4 suggests that the discriminant function approach does succeed in reducing the number and proportion of proper separation denials that QPI fails. However, the discriminant analysis does not significantly alter the number or proportion of erroneous denials that QPI passes (the first rows of panels A and B of Table 2-4 are essentially similar.) This suggests again that the DCA investigations uncover information about cases that are essential to the propriety of the case and that cannot be discovered through a simple QPI case review.

Panels A and B of Table 2-5 are analogous to panels A and B of Table 2-4, but for nonseparation denials. Panel A crosstabulates the discriminant function pass/fail score based on all QPI components (with a pass/fail threshold of 0.71) against the findings of the DCA investigation. A cut-off of 0.71 assigns roughly the same proportions of nonseparation denials to "pass" and "fail" status as the proportions of correct and erroneous cases found in the DCA investigations. Panel B of Table 2-5 repeats panel B of Table 2-2 for comparison because it shows the crosstabulation of QPI pass/fail scores (with a pass/fail threshold of 70) against the findings of the DCA investigation. (With a QPI pass/fail threshold of 70, 17 percent of the nonseparation denials fail QPI, which is close the 16 percent error rate found for separation denials by the DCA investigations.)

The conclusions for nonseparation denials are similar to those for separation denials. The discriminant function approach reduces somewhat the number and proportion of proper nonseparation denials that QPI fails (although far less than was true for separation denials; compare the second rows of panels A and B of Table 2-5.) However, the discriminant analysis does not significantly alter the number or proportion of erroneous denials that QPI passes (the first rows of panels A and B of Table 2-4 are essentially the same.) Again, the DCA investigations appear to uncover important information about cases that cannot be discovered through a QPI review.

Recall that up to three of the components of the discriminant function used to obtain the scores in panel A of Tables 2-4 and 2-5 are negatively related to the accuracy of the denial — adequacy of claimant information, adequacy of employer information, and provision of rebuttal opportunity. Accordingly, it might make sense to drop those components from the discriminant function. Doing so will not improve the performance of the scores that are based on the discriminant function (in fact, just the opposite). But it seems difficult to justify inclusion of components in a discriminant function that, although they should in principle be positively related to the outcome in question, are negatively related to the outcome.

Panels C of Tables 2-4 and 2-5 display crosstabulations the discriminant function pass/fail scores based only on adequacy of other information, whether the determination met provisions of state law and policy, and adequacy of the written determination. (Cutoffs of 0.71 assign roughly the same proportions of separation denials to "pass" and "fail" status as the proportions of correct and erroneous cases found in the DCA investigations.) Panels D of Tables 2-4 and 2-5 repeat panel D of Tables 2-1 and 2-2 for comparison.

Comparison of the results in panels C and D of Tables 2-4 and 2-5 suggests that dropping the components of QPI that are negatively related to the accuracy of denials yields a discriminant function (in panels C) that gives essentially similar results as a QPI modified to drop those components (panels D). Also, the modified discriminant function performs less well (or no better) than does the discriminant function based on all QPI components (panels A). This latter result is not surprising because the modified discriminant function omits information that is correlated with the probability of correct denial (albeit in a perverse way, as discussed above).

To summarize, the results shown in Tables 2-4 and 2-5 suggest that using a discriminant analysis of QPI components to construct case scores can result in some improvement of the QPI's performance. In particular, the approach can reduce the number and proportion of proper denials that fail QPI. However, discriminant analysis cannot reduce significantly the number or proportion of erroneous denials that QPI passes. Under the best circumstances — that is, using a discriminant analysis that takes maximum advantage of the information contained in the components of the QPI — QPI is capable of classifying at most 41 percent of erroneous nonmonetary denials as failing. Because the goal of both Quality Assurance and Quality Control is to uncover and diagnose errors in the system so that they can be corrected, the ability of QPI to identify correctly only about 40 percent of all erroneous nonmonetary denials is a clear shortcoming that suggests again the importance of conducting intensive field investigations like those carried out during the DCA pilot project.

Appendix to Chapter 2: Nonlinear Transformation of Least Squares Coefficients to Obtain Weights of the Discriminant Function

Rather than use the estimated coefficients of equation (1) directly to obtain a discriminant function, some practitioners advocate using the following nonlinear transformation of the coefficients (b's) in equation (1) to obtain the weights (b's) in equation (2). (See, for example, Kleinbaum, Kupper, and Muller 1988, pp. 566–572.) Use the squared multiple correlation coefficient (R2) obtained in estimating equation (1) to compute the Mahalanobis generalized measure of the distance between the correctly and erroneously denied samples (D2):

where b'_i denotes the estimated coefficient of the i'th right-hand-side variable in equation (1).

Once the weights of the discriminant function have been obtained, equation (2) can be used to compute a score for each denial case, as described in the text.

State Unemployment Insurance agencies make decisions about the eligibility and amount of UI benefits to be paid (if any) to each UI claimant. Errors can be of two types. First, a worker who is in fact eligible may be erroneously denied benefits. Such errors (rejecting an eligible claimant) have been the main concern of the DCA pilot project. Second, a claimant who is ineligible for UI benefits may be found eligible for and receive benefits (or, relatedly, a claimant may receive a higher benefit amount than he or she is eligible for). This latter type of error (accepting an ineligible claimant) has been investigated since the late-1980s under the Benefit Accuracy Measurement program (BAM, previously known as Benefits Quality Control, or BQC).

Section 3.1 compares the denial error rates found by the DCA pilot project with the overpayment error rates found in the same five pilot states during the same time period under the BAM program. The main purpose is to examine differences between denial and payment error rates. Section 3.2 attempts to account for the differences between denial and payment error rates. For example, are there differences between denial and payment errors in undetectable issues or incorrect decisions made by the agency?

3.1. Erroneous Denials and Payment Errors

The top panel of Table 3-1 reviews the DCA Pilot data on erroneous denials from September 1997 through August 1998. In the five pilot states taken together, 16 percent of monetary denials were erroneous, nearly 9 percent of separation denials were erroneous, and 15 percent of nonseparation denials were erroneous.

The middle and bottom panels of Table 3-1 display the payment error rates (and the frequencies from which the rates are derived) in the five pilot states during the same time period. These figures come from BAM records. Two kinds of overpayment rate are shown. The middle panel shows "total overpayments," which are positive payments made to UI claimants who the BAM investigation determined should have received a payment of zero in the key week. The bottom panel shows the sum of total and partial overpayments, which are total overpayments plus payments that are too large, according to the BAM investigation.

Note that, in computing the overpayment error rates, the total number of BAM cases is used as a base (that is, the denominator of the overpayment error rates is the total number of BAM cases). The reason is that BAM has a single sampling frame of paid cases and investigates all eligibility issues for each payment (because each payment was issued after monetary, separation, and nonseparation eligibility had been determined). The universe consists of actions that affirmed the claimant's eligibility under all three criteria. This differs from the DCA pilot, in which there was a separate sampling frame for each type of denial.

The total overpayment measure is a closer analog to erroneous denials than is the sum of total and partial overpayments. The reason is that denial entails no payment when the correct payment was positive. The obverse of denial is total overpayment — a positive when no payment should have been made. Also, many partial overpayments involve only a small sum (even a dollar or two). Accordingly, most of the comparisons below are between erroneous denials and total overpayments.

The middle panel of Table 3-1 shows that total overpayment rates are sharply lower than erroneous denial rates. In the five states taken together, the total overpayment rate on monetary determinations was 0.6 percent, compared with a monetary denial error rate of 16 percent. That is, the monetary denial error rate was nearly 27 times the total overpayment rate on monetary determinations.

Similarly, the total overpayment rate on separation and nonseparation determinations exceeded the separation and nonseparation denial rates, although the differences are less striking than for monetary determinations. The total overpayment rate on separation determinations was 1.2 percent, compared with a separation denial error rate of almost 9 percent. The total overpayment rate on nonseparation determinations was 5.6 percent, compared with a nonseparation denial error rate of 15 percent.

The gap between total overpayment rates and denial error rates holds for all five pilot states and for all three types of claim, with just two exceptions. In South Carolina and West Virginia, the nonseparation denial error rate is similar to the total overpayment rate on nonseparation determinations. South Carolina has the highest overpayment rate on nonseparation determinations, reflecting what appears to be a complicated treatment of disqualifying income. West Virginia, on the other hand, has by far the lowest rate of erroneous nonseparation denials, and a rate of overpayments on nonseparation determinations that is close to the average of the five pilot states. With these two exceptions, the behavior of the five pilot states is consistent with a far stronger aversion to overpayment than to erroneous denial. This aversion to making overpayments relative to erroneously denying benefits is consistent with keeping benefit payments down.

Although the comparison between erroneous denials and the sum of total and partial overpayments is less striking, it leads to the same general conclusion: State agencies are far more wary of making overpayments than of incorrectly denying benefits. Note that under 10 percent of all overpayments on monetary determinations (that is, the sum of total and partial overpayments) are total overpayments. (The total overpayment rate is 0.6 percent, whereas the total and partial overpayment rate is 6.9 percent.). But nearly all overpayments on separation determinations are total overpayments. (The total overpayment rate is 1.2 percent, whereas the total and partial overpayment rate is 1.3 percent.). Roughly 60 percent of all overpayments on nonseparation determinations are total overpayments. (The total overpayment rate is 5.6 percent, whereas the total and partial overpayment rate is 9.4 percent.).

3.2. Why the Differences between Overpayment and Denial Error Rates?

The main finding of the preceding section is that total overpayment rates are sharply lower than erroneous denial rates. This suggests that agencies place a higher priority on avoiding overpayments to ineligible claimants than on improperly denying benefits to eligible claimants.

However, the findings require further explanation. In particular, what mechanism underlies the low frequency of overpayments, relative to erroneous denials? Do overpayments involve more situations that are difficult to detect, and do erroneous denials involve a greater tendency to misapply adequate information (for example, by taking an incorrect action, failing to identify an issue, or not following procedures)?

Table 3-2 takes a first step in addressing these questions by tabulating prior agency action of total overpayments by state and type of determination. For total overpayments on monetary, separation, and nonseparation determinations, the table shows the percentage (and number) of cases that could not be detected, were already being resolved, and on which adequate information was misapplied (through an incorrect action, failure to identify an issue, or by not following procedures). (These categories and their interpretation are described in more detail in chapter 1.) The table displays figures for each of the five pilot states individually and for the five states aggregated. Note that Table 3-2 has the same form as Table 1-1, which displays the same type of data for erroneous monetary, separation, and nonseparation denials.

Comparison of the findings is facilitated by Table 3-3, which shows prior agency action on erroneous denials (from Table 1-1) and on total overpayments (from Table 3- 2) for the five DCA pilot states aggregated. Table 3-3 gives the striking impression that a large majority of total overpayments involve errors that could not be detected by normal agency procedures. Over three-quarters of total overpayments on monetary and nonseparation determinations were "undetectable," and about 60 percent of total overpayments on nonseparation determinations were "undetectable." The percentages of erroneous denials that were undetectable are far lower: 39 percent for erroneous monetary denials, 20 percent for erroneous separation denials, and 22 percent for erroneous nonseparation denials.

It follows that erroneous denials are far more likely to result from errors of judgment — that is, misapplication of adequate information — than is the case for total overpayments. Table 3-3 suggests that this is the case. Roughly 30 percent of erroneous monetary denials result from incorrect agency action, failure to identify an issue, or failure to follow procedures (that is, misapplication of adequate information); this contrasts with 14 percent of total overpayments on monetary determinations resulting from these causes. Similarly, over 70 percent of erroneous separation denials result from misapplication of adequate information, in contrast to 36 percent of total overpayments on separation determinations. And nearly 70 percent of erroneous nonseparation denials result from misapplication of adequate information, in contrast to about 21 percent of total overpayments on nonseparation determinations.

Overall, the findings of this and the previous section suggest that agencies have quite low total overpayment rates, especially on monetary and separation determinations. Also, the total overpayment errors that do occur tend to be caused not by human error — rather, they are difficult to detect. This latter is not an unexpected finding: Claimants have an interest in providing any pertinent information indicating that they are eligible for benefits and in concealing information suggesting that they are ineligible. As a result, agencies can be expected to have more information in the case of an erroneous denial than in the case of an overpayment. (However, the inference that agencies can be expected to have more information in the case of erroneous denials makes it surprising that overpayment rates are so much lower than erroneous denial rates.)

In contrast, rates of erroneous denials are far higher than rates of total overpayment. Moreover, erroneous denials are far more likely to result from errors of judgment than are total overpayments. Whereas most total overpayments can be considered "undetectable," most separation and nonseparation denial errors result from misapplying adequate information.

3.3. Claimant Characteristics and Denial Probabilities

• The claimant may be properly determined eligible and paid the right benefit amount;
• The claimant may be properly denied benefits;
• The claimant may be erroneously denied benefits;
• The claimant may be incorrectly determined eligible for benefits and paid benefits for which he or she in ineligible (total overpayment), or may be correctly determined eligible but paid more or less than the correct benefit amount (partial overpayment or partial underpayment) .

The third possibility is the type of error that is the focus of a DCA investigation, and the fourth possibility is the type of error that is the focus of the BAM program.

This section models these unconditional probabilities using ordinary least squares (OLS) equations that control for the characteristics of each claimant. The goal is to examine the role, if any, that demographic and other characteristics of claimants play in the determinations process and its outcome. (More sophisticated modeling techniques are available, and may be appropriate, for examining these issues. In particular, because the determinations process is one that has multiple possible outcomes, multinomial logit would be an appropriate technique. Accordingly, the analysis presented here should be considered preliminary.)

Four models are estimated. In the first, the sample of eligible claimants from BAM is pooled with the sample of claimants from the DCA pilot project who were eligible (as discovered by the DCA investigation) but who had received an erroneous monetary determination. (Claimants who received a total overpayment are dropped from the BAM sample because they are ineligible. Claimants who received a partial overpayment are retained because, even though they received a partial overpayment, they were eligible.) The following model is then estimated:

where DM_i equals 1 if claimant i was denied benefits for monetary reasons, 0 otherwise, X₁ through X_K denote characteristics of the claimant and the claim, beta₀ through beta_K are linear coefficients to be estimated by OLS, and u_i is an error term assumed to be random. Note that, because the sample used in estimation includes only eligible claimants, the DM_i indicator denotes not just denial, but erroneous monetary denial. The comparison being made in this model is between (a) eligible claimants who were correctly determined eligible for benefits, and (b) eligible claimants who were erroneously denied benefits.

• age of the claimant in years
• an indicator for gender (0 if female, 1 if male),
• an indicator equal to 1 if the claimant is a U.S. citizen, 0 otherwise,
• five 0-1 indicators for ethnicity (African American, Hispanic, American Indian, Asian/Pacific Islander, and Caucasian),
• four 0-1 indicators for level of schooling (less than high school, high school graduate, some college, college degree),
• the ratio of the claimant's UI Weekly Benefit Amount (WBA) to the maximum WBA in the claimant's state,
• the potential duration of benefits (in weeks),
• four 0-1 indicators of the season in which the claim was filed (winter, spring, summer, fall), and
• five 0-1 indicators of the state in which the claim was filed (Nebraska, New Jersey, South Carolina, West Virginia, Wisconsin).

Similar models are estimated for separation and nonseparation denials. For the model of separation denials, the sample of eligible claimants from BAM is pooled with the sample of claimants from the DCA pilot project who were eligible (as determined by the DCA investigation) but who had received an erroneous separation determination. For the model of nonseparation denials, the sample of eligible claimants from BAM is pooled with the sample of claimants from the DCA pilot project who were eligible but had received an erroneous nonseparation determination. (In creating the samples used to estimate these models, claimants who received a total overpayment are dropped from the BAM sample because they are ineligible. However, claimants who received a partial overpayment are retained.)

A fourth model is estimated for total overpayment. For this model, a sample of ineligible claimants is constructed by pooling all properly denied claimants from the DCA pilot sample with all claimants who received a total overpayment from the BAM sample. The model estimated is then:

where OT_i equals 1 if claimant i received a total overpayment, 0 otherwise, X₁ through X_K denote characteristics of the claimant and the claim, g0 through gK are coefficients estimated by OLS, and ei is random error. The comparison being made in this model is between (a) ineligible claimants who were properly determined to be ineligible (and so received no benefits), and (b) ineligible claimants who were improperly determined to be eligible (and so received benefits they should not have received).

Table 3-4 displays the results of the models just described. In the first column, the sample of eligible UI claimants is used to estimate the probability of erroneous monetary denial. The results suggest that men and Hispanics who are monetarily eligible for benefits are significantly more likely to be denied (erroneously) than are others. Claimants whose correct WBA and potential duration of benefits are relatively high are significantly less likely to be erroneously denied for monetary reasons. This is not surprising — claimants whose correct WBA and potential duration are high are not the "borderline" cases that are likely to be error-prone. Finally, the probability of erroneous monetary denial appears to be lower in New Jersey than in the other four pilot states, other things equal.

The second column of Table 3-4 displays the results of estimating a model in which an indicator for erroneous separation denial sample is regressed on the same independent variables. (The sample of eligible UI claimants from BAM is pooled with the erroneous separation denials from the DCA pilot.) These results suggest that the older a claimant, the less likely he or she is to receive an erroneous separation denial. Claimants whose correct WBA and potential duration of benefits are relatively high are also less likely to receive an erroneous separation denial. (This result makes less sense in the case of separation denials than for monetary denials. Although it seems likely that conditions of separation are related to a worker's earnings history — low earnings workers are more likely to quit or be discharged for cause — it should be no more difficult to determine the conditions of separation for a lower-wage/lower-benefit worker than for a higher-wage/higher-benefit worker. The finding suggests that adjudicators may incorrectly use a relatively weak earnings history as an indication that a claimant does not meet the separation conditions for eligibility.) Finally, the probability of erroneous separation denial appears to be higher in Wisconsin than in the other four pilot states, other things equal.

In the third column of Table 3-4, the sample of eligible UI claimants is used to estimate the probability of erroneous nonseparation denial. The results suggest that blacks are significantly more likely to receive an erroneous nonseparation denial than are others. As is true of monetary and separation denials, claimants whose WBA and potential duration of benefits are relatively high are significantly less likely to receive an erroneous nonseparation denial. (The comments about this result in the context of separation denials also apply here.) Finally, the probability of erroneous nonseparation denial appears to be lower in New Jersey and West Virginia than in the other three pilot states, other things equal.

The right-most column of Table 3-4 displays the results of estimating a model in which an indicator of total overpayment is regressed on characteristics of the claimant and other characteristics of the claim. (For this regression, the sample of ineligible UI claimants from the DCA pilot is pooled with the BAM sample of ineligible claimants who received a total overpayment.) The results suggest that ineligible men and American Indians (of whom there are very few in this sample) are more likely than others to receive a total overpayment. Claimants whose correct potential duration is relatively long are also more likely to receive a total overpayment, but the claimant's WBA (relative to the state maximum WBA) has no discernible impact of the probability of total overpayment. That is, claimants who are at the maximum potential duration, but whose WBA is below the maximum, are more likely than others to receive a total overpayment. Finally, compared with Wisconsin, total overpayments are more frequent in South Carolina and less frequent in New Jersey and West Virginia.

Overall, the results in Table 3-4 suggest four main conclusions. First, and understandably, claimants whose earnings history puts them near the minimum threshold of benefit eligibility are more likely to receive an erroneous monetary denial than are other claimants. Second, it appears that adjudicators may incorrectly use a relatively weak earnings history as an indicator that a claimant does not meet the separation or nonseparation conditions for eligibility. Third, claimants who are at the maximum potential duration, but whose WBA is below the maximum, are more likely than others to receive a total overpayment. Fourth, there is some evidence in the data that individual characteristics such as race and gender may play a role in erroneous denials and total overpayments. These results are not strong; however, they should alert agencies to a potential problem and should be investigated in future studies of overpayments and erroneous denials.

Errors in determining eligibility have financial consequences for claimants and UI trust funds. This chapter combines information on error rates gathered in the denied claims pilot project (DCA) with other data to estimate the dollar impacts of erroneous denials.

Estimating the benefits not paid to claimants as a result of erroneous denials poses significant analytic problems. For erroneous denials related to monetary and separation determinations, there is no initial payment or associated payment stream because the administrative decisions found the claimant ineligible. The unobserved counterfactual — a payment stream covering a succession of weeks within a benefit year — never took place. The dollar cost to the claimant is the weekly benefit amount (or WBA, which can be calculated using correct information on base period earnings) times the unobserved number of weeks in benefit status. Estimating the unobserved weeks in benefit status presents a challenge that is addressed in this and the next chapter.

The chapter is divided into four sections. Section 4.1 identifies several important issues that need to be addressed in developing cost estimates. Section 4.2 describes three alternative approaches for making cost estimates. Section 4.3 reviews penalties associated with benefit denials and discusses duration in benefit status. Section 4.4 presents two sets of estimates of the benefits lost due to erroneous denials. Estimates based on the experiences of the pilot states are used to derive national totals.

4.1. Issues in Estimating Lost Benefits

Several issues must be addressed in deriving estimates of the benefits lost due to erroneous denials. Four are treated here: self-correction of initial administrative errors, the interconnectedness of error corrections, estimating the benefits lost per case, and aggregation issues.

4.1.1. Self-correction. Initial errors in denying benefits may be corrected by the normal operations of the UI system. Such "self-correction" was addressed in both pilot projects and both analyses showed self-corrections to be common, especially for monetary determinations. It would seem that estimates of lost benefits should include only cases where initial agency errors would not be corrected through routine administrative procedures.

An important issue (raised in chapter 3 of the Final Report) is the length of the interval between the claim date and the date for measuring the accuracy of monetary determinations. Determining the appropriate length of this interval is a key operational consideration for the eventual implementation of DCA measurement in the states. All but one pilot state recommended that the sampling of monetary denials be delayed for ten work days from the date that the claim was filed, in order to avoid including in the samples to be investigated claims that were initially denied but that will be redetermined in the normal course of the determination process.

It seems useful to distinguish between initial errors and final errors, where the latter constitute a smaller total due to agency self-corrections, employers' actions, and appeals. Table 4-1 summarizes the error rates (both unadjusted and adjusted for selfcorrections) for the two pilot projects. For both pilot projects the figures displayed are simple averages for the five pilot states. In both pilots, self-correction was larger for monetary issues and separation issues than for nonseparation issues. For the former pair of issues more than one-quarter of initial errors were "corrected" by the combined effects of agency procedures, employer actions, and appeals.

There is suggestion in Table 4-1 that overall denial error rates declined between the two periods and by more for monetary and separation issues than for nonseparation issues (where error rates may even have increased). However, because only one state participated in both pilot projects, conclusions regarding a possible decrease in error rates between the two time periods cannot be drawn.

One approach to estimating the benefits lost due to erroneous denials would be to recognize that errors, even when corrected, entail costs to the claimant in delayed payments. If benefits are eventually paid, however, these payments should be captured by BAM (because BAM covers all paid weeks). Using this logic, erroneous denials that are eventually corrected are already part of BAM, and to avoid double counting, they should not be included in the DCA estimates. Thus, it seems that the appropriate approach would examine just the forgone benefits associated with final ("adjusted") errors as displayed in the middle column in Table 4-1. Accordingly, the estimates of lost benefits presented below use only "final" error cases.

4.1.2. Interface Among the Errors. The process of claim, determination, and payment involves a sequence of administrative decisions. For a new initial claim, the sequence is roughly: (a) a monetary determination, (b) the possibility of a separation determination, and (c) the possibility of one or more nonseparation determinations. Receipt of benefits requires positive decisions on all three administrative decisions.

While the preceding sequence oversimplifies the actual process, it provides a useful framework for considering the effects of erroneous denials. If a claimant is erroneously denied on a monetary issue which is then corrected, there could still be a separation denial and/or nonseparation denial. In developing accurate estimates of the benefits lost due to erroneous denials, the interrelations implied by this sequence must be recognized.

This issue was addressed in the DCA Final Report (Woodbury and Vroman 1999, section 5.2.2). There, two situations were noted in which incorrect denials would not lead to payments: (a) an erroneous denial where the reason was incorrect but the decision was correct and (b) correction of an erroneous denial that would be followed by a correct denial at a later stage of the payment process (for example, correction of an erroneous monetary denial could be followed by a denial on a separation issue and/or a nonseparation issue).

The earlier report noted that denials that were correct but for the wrong reason (that is, the first of the above two situations) were observed for 3 percent of monetary denials, 15 percent of separation denials and 14 percent of nonseparation denials. Thus, the impact of these situations could be estimated directly and would be expected to have minor implications for the magnitude of lost benefits.

The second situation is more problematical due to the analytic framework of the DCA Pilot Project, which investigated a case only with respect to one of the three issues — monetary, separation, or nonseparation. Not all corrected monetary determinations would lead to a payment, and not all corrected separation determinations would lead to a payment (at least for a full period in benefit status). In the cost estimates presented in section 4.4, we attempt to take account of this difficulty in the design of the DCA Pilot; however, the corrections are only rough.

4.1.3. Benefits Lost per Case. The procedure for estimating benefits lost (that is, not paid) focuses on benefits lost per erroneously denied claim. For a claimant erroneously denied benefits, this loss is the product of estimated time in benefit status and the weekly benefit amount (WBA). Of the two elements that determine benefits lost per case, the WBA is known once the correct information on base period earnings has been obtained. (In both pilot projects, the WBA of those erroneously denied was lower than for those who received benefits. The differential was about 20 percent in the 1997-98 pilot.) What is not known is the time in benefit status. For individuals erroneously denied, benefit duration must be estimated.

One way to estimate benefit duration is to use statewide (or national) average duration for beneficiaries during the same period as that covered by the pilot project; that is, fiscal year 1998. This method has been used by Skrable (1999) and, with modifications discussed in section 4.3, is used in section 4.4 below.

As noted in chapter 5 of the DCA Final Report, an alternative way of estimating average benefit duration is to derive a statistical estimate of the average duration for each person erroneously denied benefits in the pilot states and then average the statistical estimates. Unfortunately, the data needed to derive such estimates are available in only one of the pilot states — South Carolina. In chapter 5, the South Carolina data are used to obtain such estimates. The implications of those estimates for lost benefits are also derived in chapter 5.

4.1.4. Aggregation Issues. After identifying an erroneous denial and estimating the resulting benefits lost, those losses must be aggregated to the universe of cases. Aggregation involves at least three issues. The first is to identify the universe of similar cases. Second, aggregation may entail summing over time periods; for example, from a week to a full calendar year. Third, there are questions of how to aggregate results from the five DCA pilot states to national totals.

To identify the universe of similar cases it seems appropriate to use data from administrative reports routinely submitted by the states. For monetary determinations the needed data are included in the ETA 218 reports. These data record the total number of monetary determinations and the number of claims with sufficient wage credits. Those with insufficient credits can serve as the universe for incorrect monetary denials. During fiscal year 1998 there were 10.78 million monetary determinations and 1.17 million findings of insufficient wage credits. Hence, 1.17 million is a universe count of denials on monetary issues.

Data submitted by states in ETA 207 reports for the same period show that there were 3.42 million determinations on separation issues and 1.86 million denials. Nonseparation determinations and denials from these same reports totaled 4.28 million and 2.39 million respectively in fiscal year 1998. Thus, universe counts for separation and nonseparation denials are 1.86 million and 2.39 million. The error rates estimated in the five pilot states can be applied to the national denial totals to estimate the national number of errors of each type.

Aggregation by time period must also be recognized. Quality control measurement systems operate using a key week concept. The findings from a key week need to be aggregated over time to estimate annual dollar losses due to payment errors. This is true of both denial errors and payment errors.

In order to assess the national consequences of denied claim errors, it is necessary to aggregate the results from the five pilot states to the U.S. as a whole. Two questions arise in connection with obtaining national totals from the experience of the five pilot states: First, are the average benefits lost per case in the pilot states representative of the national average? Second, are there other peculiarities of the pilot states — for example, the mix of denied cases across the three issues (monetary, separation, and nonseparation) — that need to be recognized in developing national estimates?

To illustrate one aspect of the problem, it is instructive to review data from the pilot states on the average weekly wage and the average weekly benefit. One potential element in aggregation would be to adjust for differences in the average weekly wage (hence WBA) between the pilot states and the U.S. average. In 1998, the national AWW for all covered employees was $610.43. The employment-weighted AWW for the five DCA states was $610.26, a deviation of only 0.03 percent from the national average. This virtual equality, however, did not translate into equality of WBAs. In calendar year 1998, the national average WBA was $202.29 whereas the average WBA (weighted by weeks compensated) for the five pilot states was $231.33, a difference of 14.4 percent. The difference reflects the importance of the high WBA in New Jersey and its share of weeks compensated in the weighting for the pilot states.

4.2. Three Approaches to Estimating Lost Benefits

The dollar costs of denied claims can be estimated in three ways: (1) the key week approach, (2) the benefit year approach, and (3) a hybrid approach. Each is described presently.

4.2.1. The Key Week Approach. This approach, which would copy the approach followed in BAM, would concentrate on benefits lost in a single week due to an erroneous denial. For both monetary and separation issue errors, the weekly benefit loss would be given by the WBA. For most nonseparation issue errors, the benefit loss would be the WBA, but this can be modified in some instances. A disqualifying or deductible income penalty would be assessed as denial of benefits for one or more weeks either with or without a reduction in the Maximum Benefit Amount (MBA). Penalties would vary according to the exact income source and according to state law. (Tables 410A and 410B of the "Comparison of State Unemployment Insurance Laws" show the type of penalty applied to pension benefits and other types of employee compensation subject to disqualifying and deductible income penalties.)

After the erroneous denials have been corrected, the interfaces between the issues would still need to be considered. Thus, as noted in section 4.1, not all corrected monetary denials would be followed by a payment because of possible denials on separation and nonseparation issues. (It is also possible for a claimant to remain monetarily ineligible even after a monetary determination error has been corrected — for example, the error may have been failure of an employer to report wages during the base period, but base period earnings may remain insufficient to meet the monetary eligibility criteria.)

A problem in following the key week approach is the absence of an observed payment stream for erroneous monetary and separation denials. Payments never started, so there is no stream of weeks compensated from which to draw a sample. Because there is no payment stream, one cannot sample from different points (weeks) in the stream as in BAM. Thus, a focus on key weeks (as in BAM), would make it impossible to impute lost benefits due to erroneous monetary and separation denials. (For nonseparation errors, there is no such problem because, as with BAM cases, nonseparation errors can occur during any individual week in benefit status, and hence can be sampled.)

After assigning a cost per key week to all erroneous denials (typically the WBA), there would remain the question of how to aggregate these costs to statewide annual totals. A direct approach would be to multiply total weekly costs by 52, which would convert a "week" into a year. Although simple, we are reluctant to propose this as a method for aggregating to statewide annual totals. Discussions with BAM professional staff in the pilot project states and at the national Office of Workforce Security suggest that no convincing aggregation procedure exists. These discussions lead us to conclude that errors on monetary and separation determinations are essentially tied to the case or person, not to a key week.

4.2.2. The Benefit Year Approach. This approach examines the consequences of erroneous administrative decisions within a framework where the time unit is the benefit year. Errors are modeled as having consequences that span several weeks. This approach to estimating lost benefits explicitly recognizes benefit duration and the associated stream of payments that did not occur.

Under this approach, the WBAs calculated for each of the three determination issues are the same as under the key week approach. The interfaces between the three issues also need to be explicitly treated. As noted in section 4.1, benefit duration could be modeled in different ways (the use of statewide average durations or projected durations based on a regression equation methodology are the main possibilities). The durational approach is appropriate even for nonseparation issues because many of these erroneous disqualifications have multi-week penalties (see section 4.3 below).

The universes for sampling the three types of cases are as follows:

• Monetary errors. The appropriate universe is new initial claims for benefits. If successful, these claims establish new benefit years. Thus, erroneous denials and denial rates should be measured relative to all of these determinations. As noted, population data on denials (finding insufficient wage credits) are available from ETA 218 reports for the same period.
• Separation errors. Except for separations caused by lack of work, most separations are adjudicated; that is, a determination is performed. Where the separation resulted from a voluntary quit or discharge for misconduct, the agency almost always conducts an adjudication. Error rates should be measured relative to the number of determinations on these issues for new (monetarily eligible) initial claims and additional initial claims (where monetary eligibility has already been established). Universe counts of these determinations are available from ETA 207 data. Unfortunately, the ETA 207 reports show determinations on new initial claims and additional claims combined, not separately. Thus, possible differences in determination rates and denial rates between the two types of initial claims cannot be estimated from these data.
• Nonseparation errors. These arise during the course of paying claims. Error rates should be applied to the universe of weeks claimed (following both new and additional initial claims) to estimate the total number of such errors. Determinations on nonseparation issues are reported in ETA 207 data while weeks claimed can be derived from ETA 5159 reports. As with separation issues, however, the nonseparation data on weeks claimed and the associated determinations and denials do not distinguish weeks associated with new initial claims from weeks associated with additional initial claims.

An implication of the preceding is that denials for separation issues can be expected to have shorter penalty periods than denials for monetary reasons because separation denials are applied to both new initial claims and additional claims. Penalty periods for nonseparation errors should be shortest because they may occur at any point in a benefit spell rather than at the start. (For further discussion of penalty periods, see section 4.3.)

4.2.3. A Hybrid Approach. The BAM (key week) approach to estimating lost benefits draws samples from the universe of paid claims. This can be applied directly to just one type of issue in DCA, nonseparation determinations. In erroneous denials involving monetary and separation determinations, there is no series of weeks in benefit status where key week sampling can be applied. The hybrid approach follows BAM in the treatment of nonseparation issues but a benefit year approach for the other two issues (monetary and separation determinations).

For all three issues, the determination of the WBA is the same as in the key week and benefit year approaches described above. The interfaces among the three types of denials would be treated in a similar manner under the hybrid approach. However, for both monetary and separation errors, a duration in benefit status would be assigned in arriving at the estimate of lost benefits. As noted above, the duration estimates could be taken from statewide averages (with adjustments as described below) or could be based on a regression methodology utilizing BAM micro data (see chapter 5). Because many of the procedures to be followed under the three approaches are identical, it would be interesting to know the sensitivity of findings to the choice of approach.

Denials lead to penalties that delay or reduce (sometimes to zero) the benefits paid to claimants. It is useful to review briefly the penalties for each type of denial. As will be seen, the penalties vary by issue and by state for separation and nonseparation determinations.

The loss to the individual claimant due to an erroneous denial, regardless of the issue, is the weekly benefit amount (WBA) times the weeks of UI benefits that would have been paid had a correct determination been made. There is little uncertainty surrounding the WBA but much uncertainty as to the duration in benefit status. For both separation and nonseparation determinations, the claimant will have satisfied the state’s monetary eligibility criteria so that the WBA is established. For monetary determinations, the WBA will also be known after correct information on the claimant’s base period earnings has been assembled.

For the key week and benefit year approaches to cost estimation discussed in section 4.2, it is necessary to estimate the length of time the claimant would have received benefits during the current benefit year, had the correct eligibility decision been made. Under the benefit year approach, duration must be estimated for all three types of errors while the hybrid approach requires duration estimates for monetary and separation denials. Only the key week approach could (potentially) make estimates without reference to benefit duration.

Although estimating duration is highly uncertain for individual recipients, there are systematic differences in benefit duration for the three types of denials. Table 4-2 presents a taxonomy to help illustrate differences in the durational consequences of the three types of erroneous denials.

The rows of Table 4-2 identify administrative activities for the three types of determinations (monetary, separation and nonseparation). For each type of determination, the table shows the universe of claims subject to the determination, the administrative decision (outcome), and the penalty for a denial. The columns identify three types of claims that need to be distinguished because of differing durational consequences: new initial claims, additional initial claims, and continued claims. (Note that the table simplifies by omitting interstate and transitional claims.) The body of Table 4-2 shows the interface between type of claim and type of determination. Monetary and separation determinations are applied to new initial claims. Separation determinations are also applied to additional initial claims (second and later claims in a given benefit year). Nonseparation determinations are applied only to continued claims.

A monetary determination is made for all new initial claims. The vast majority of new initial claims arise from a separation due to lack of work, a quit, or a discharge. (In what follows, we focus on discharges due to ordinary misconduct, not flagrant or aggravated misconduct.) For quits and discharges, the UI agency adjudicates the separation to determine eligibility. Adjudication occurs in roughly one-fourth of initial claims (that is, both new and additional initial claims, according to ETA 207 reports on the number of determinations and unpublished data counts of new spells).

A denial for separation reasons usually implies that benefits will not be received for the duration of the current spell of unemployment. (Exceptions arising from disqualifications for specific periods are discussed below.) A durational penalty also applies for a monetary determination when the claimant is found to have insufficient wage credits. However, on average, the duration of benefits associated with a monetary denial (that is, due to insufficient wage credits) tends to be longer than the duration associated with a separation denial. The reason is apparent in Table 4-2: many separation denials are applied to additional initial claims, in which claimants have already used up a substantial share of their MBA with an earlier claim (or claims).

4.3.1. Monetary denials. These are the most straightforward. When the claimant is found to have insufficient earnings in the base period, the penalty is complete exclusion from benefit status. The claimant does not establish a benefit year. If there is a later separation and a subsequent claim for benefits, the claimant may collect benefits following the later separation; however, for the denied claims project, this future event lies beyond the scope of analysis.

4.3.2. Separation denials. For all practical purposes there are two separation issues: voluntary quits and misconduct. In FY 1998 determinations on voluntary quits totaled 1.484 million and determinations of misconduct totaled 1.830 million. Combined, they accounted for 97.3 percent of all separation determinations for the year. (These totals come from ETA 207 quarterly reports.)

Penalties for separation issues are essentially of two kinds. The first is disqualification from receiving benefits for the duration of the current spell of unemployment — a "durational" disqualification. A claimant subject to such a durational disqualification must requalify for benefits (by working for a specified period of time and/or earning a specified amount) in order to be eligible for benefits in the event of a later involuntary job separation. The second type of penalty is delay of benefit receipt for a specified number of weeks, usually (but not always) with a corresponding reduction in the MBA. In general, a claimant subject to such a disqualification may wait out the specified period of disqualification, reopen the claim for benefits, and receive benefits.

Table 4-3 displays penalties for each of the five pilot states on the two separation issues. For a voluntary quit, four of the five pilot states disqualify the claimant for the duration of the current spell and impose a requalification requirement. Nebraska is the exception, delaying benefits for the week of the claim and 7 to 10 subsequent weeks and reducing the MBA by a corresponding amount. (More than one such penalty may be assessed in Nebraska if a claimant quit from more than one base period employer.)

Requalification requirements raise another issue in the estimation of lost benefits. A subsequent separation may have implications for an earlier erroneous denial. An erroneously denied claimant could lose benefits from a later separation if the separation occurs before the requalification period has been satisfied. Although such situations exist, they are probably rare, and we do not take account of them in the cost estimates presented.

In four of the five pilot states, the penalty for a misconduct discharge is delay of benefit payments along with reduction of the MBA. In New Jersey (the exception), benefits are delayed for the week of the claim plus the subsequent 5 weeks; the MBA is not reduced. In Nebraska and South Carolina, there is discretion in the number of weeks by which benefits are delayed and reduced (that is, the penalty is of variable duration). In Wisconsin, the claimant is disqualified for the duration of unemployment. In both West Virginia and Wisconsin, benefit reductions can be recovered by meeting a requalification requirement. This could be relevant for second and later separations that occur within a given benefit year.

The penalties on separation issues for the pilot states, as summarized in Table 4-3, need to be viewed in a national perspective. Of the 53 UI programs, 49 disqualify a claimant who quit voluntarily for the duration of unemployment. Thus, for voluntary quits, the penalties in the pilot states roughly reflect the national situation. However, in 40 of 53 UI programs, a claimant discharged for misconduct is disqualified for the duration of unemployment. (Of the pilot states, Wisconsin is the only one that disqualifies a discharged claimant for the duration of unemployment.) That is, the pilot states greatly understate the prevalence of durational disqualifications for misconduct. Aggregated to national totals, the benefits lost due to erroneous misconduct denials in the pilot states would understate the corresponding benefit lost nationwide.

Table 4-3 introduces complexities that need to be considered in analyzing the costs of erroneous denials. The main point of the information in Table 4-3 is that the penalties associated with individual disqualifications vary considerably by issue and state. To treat all penalties on separation denials as if they were durational would be a simplification of reality, especially for misconduct issues.

4.3.3. Nonseparation denials. The previous discussion about variation in disqualification penalties has even more force when applied to nonseparation disqualifications. The UI reporting system (ETA 207 reports) explicitly identifies five nonseparation issues: (a) able and available for work, (b) disqualifying or deductible income, (c) refusal of suitable work, (d) reporting requirements, and (e) profiling. In addition, there is a sixth catch-all category ("other"). During FY 1998 there were 4.3 million nonseparation determinations with the two largest categories being able and available (1.360 million) and disqualifying or deductible income (1.012 million). All of these disqualifications are applied to weeks of continued claims arising from both new and additional initial claims.

Most disqualifications for nonseparation violations are of two types [(U.S. Department of Labor, Comparison of State Unemployment Insurance Laws, tables 400 (able and available), 404 (refusal of suitable work), and 410A and 410B (disqualifying and deductible income)]. Able and available disqualifications are generally for the week of the violation with no reduction in the MBA. Thus, a claimant who exhausted benefits could collect the full MBA, even with the penalty (that is, there would only be a delay in receipt of benefits). Violation of reporting requirements, refusal of suitable work, and profiling violations generally have penalties that cover several weeks, often the remainder of the current unemployment spell. (The "other" category, because it is a catch-all, has more than a single potential disqualification depending on the issue.)

If the claimant exhausts benefits for which he or she is eligible, single week penalties that delay payments without reducing the MBA do not ultimately result in lost benefits. For claimants who do not exhaust their MBA, the penalty represents a one week loss of benefits.

As noted, many penalties for nonseparation issues apply for multiple weeks or for the remaining duration of the unemployment spell (for example, violation of reporting requirements or refusal of suitable work). The determination leading to the penalty can occur for any week in which benefits are claimed. These weeks may follow either a new initial claim or an additional initial claim. For the latter, the penalty will typically last for fewer weeks than if it followed a new initial claim.

Disqualifying or deductible income penalties either reduce benefits in weeks when the income is received (hence delaying benefit receipt) or reduce the MBA by the amount of the alternative income source, up to the full MBA. (In cases where benefits are reduced by less than the MBA, benefit payments are also delayed.) These penalties cover workers' compensation, wage continuation payments, severance pay, vacation pay, pension benefits. State approaches differ widely, and the penalties associated with disqualifying income accounted for about 25 percent of all nonseparation disqualifications in fiscal year 1998 (ETA 207 data).

National estimates of the benefits lost due to erroneous denials must make assumptions about three questions discussed in section 4.1: self-correction, the interface among the errors, and the benefits lost per case. The estimates presented in this report all use the benefit year approach discussed in section 4.2. We begin with a discussion of the estimates that are similar to those prepared by Skrable (1999), and then modify those estimates to check the sensitivity of his estimates to certain assumptions.

4.4.1. Number of erroneous denials. Skrable starts with an imputation of the number of erroneous monetary, separation, and nonseparation denials in the United States, based on UI financial data and the 1998 DCA results. Table 4-4 displays the calculations. During 1998, there were roughly 10.8 million monetary determinations, 15.9 million separation determinations, and 117.6 million nonseparation determinations in the United States (see the notes to Table 4-4 for sources). Of the monetary determinations, approximately 1.2 million (or 10.9 percent) resulted in denial of benefits. Of the separation determinations, 3.4 million (or 21.4 percent) were adjudicated, and 1.9 million (or 11.7 percent) resulted in denial. Of the nonseparation determinations, 4.3 million (or 3.6 percent) were adjudicated, and 2.4 million (or 2.0 percent) resulted in denial.

The 1998 DCA Pilot Project found that, in the five pilot states, 16.0 percent of the monetary denials, 8.7 percent of the separation denials, and 15.0 percent of the nonseparation denials were erroneous. The 1998 pilot also found that a significant proportion of erroneous denials were corrected either by the UI agencies or through the appeals process. After adjusting for these "self-corrections," the error rates are 11.2 percent for monetary denials, 6.4 percent for separation denials, and 12.9 percent for nonseparation denials.

Assuming that the adjusted error rates estimated in the five pilot states are representative of error rates throughout the United States, they can be applied to the number of denials in the United States to obtain imputations of the number of erroneous denials nationally. This is done in the bottom row of Table 4-4: the imputed number of erroneous monetary denials for the United States, 131,264, is calculated by multiplying the total number of denials in the U.S. by 0.112, and similarly for separation and nonseparation denials. [These figures differ somewhat from those in Skrable (1999), apparently due to a different accounting of self-corrections.]

Skrable (1999) further reduces the number of erroneous monetary denials because only about 75 percent of all correct monetary determinations result in a first payment. There are two reasons for this. First, many monetarily eligible claimants turn out to be ineligible for nonmonetary reasons. Second, not all monetarily eligible claimants receive a first payment; that is, they end their claim because they find a job quickly or drop out of the labor force. After accounting for these factors, the imputation is that roughly 99,000 claimants who would have received benefits were erroneously denied benefits for monetary reasons. [We assume that all erroneous separation and nonseparation denials would have resulted in benefits being paid. As a result, no similar reductions are made for erroneous separation or nonseparation denials.]

The imputed number of erroneous monetary, separation, and nonseparation denials that would have resulted in benefits being paid are shown both in the bottom row of Table 4-4 and in the first column of Table 4-5, which is used to display further development of the denial cost estimates.

For erroneous monetary denials, this relative WBA factor is 0.859; for erroneous separation denials, it is 0.866; and for erroneous nonseparation denials, it is 0.909 (see the column headed "relative WBA factor" in Table 4-5, panels A and B.) 4.4.3. Average duration of benefits received by erroneously denied claimants. Finally, for each type of denial, it is necessary to use an estimate of the average number of weeks of benefits that would have been received by erroneously denied claimants. Estimates of the number of weeks of benefits lost due to erroneous monetary, separation, and nonseparation determinations are discussed in turn.

First, the average number of weeks of benefits lost due to each erroneous monetary denial can be imputed as the average number of weeks compensated per first payment (that is, the sum of first and subsequent spells of benefit receipt within a given benefit year). This assumes that erroneously denied claimants are similar to correctly determined eligible claimants. In FY 1998, this average number of weeks compensated was 14.2 in the United States (ETA 5159 reports) and is used in both panels A and B of Table 4-5 as an imputation of the number of weeks of benefits lost due to an erroneous monetary denial.

Second, how many weeks of benefits were lost by the typical claimant who was denied for separation reasons? The average number of weeks compensated overstates this because separation determinations are made on additional initial claims (as well as on new initial claims), and additional initial claims result in shorter spells of benefit receipt. During 1998, 61.2 percent of all initial claims were new initial claims, and 38.8 percent were additional initial claims (ETA 5159 reports). In 1990- 1993, across fifteen states for which special survey data are available (Battelle Memorial Institute 1999, Table 7-1), the average benefit duration for first spells of unemployment was about twice the duration of subsequent spells (the mean first spell was 13.18 weeks long, and the mean subsequent spell was 6.16 weeks long).

These figures allow a rough imputation of the average number of weeks of benefits lost by a claimant denied for separation reasons. Suppose that the duration of subsequent spells in 1998 was still roughly half that of first spells, as found by Battelle Memorial Institute (1999, Table 7-1) for 1991-93. Then the average number of compensated weeks associated with an additional initial claim would be about 7.1 (half of 14.2). If all separation denials were for the duration of unemployment, the mean duration of a separation denial would be 11.4 weeks [= (0.612•14.2) + (0.388•7.1)]. (This also assumes that the rates of adjudication for reasons of separation are the same for new and additional claims.) The figure would be somewhat lower if account were taken of the fact that some separation denials are for fixed periods that are shorter than the duration of the spell. In any case, panel B of Table 4-5 uses 11.4 weeks as the imputed number of weeks of benefits lost due to an erroneous separation denial.

Third, how many weeks of benefits were lost by the typical claimant who was denied for nonseparation reasons? Earlier estimates of the loss in benefits from nonseparation disqualifications have assumed the penalty to be one week per disqualification (Belle and Casey 1988, Skrable 1999). This assumption is made in panel A of Table 4-5.

As discussed above, able and available disqualifications are for one week (the week of the violation). However, other nonseparation disqualifications, such as those for violating reporting requirements, refusal of suitable work, and (in many cases) disqualifying and deductible income, involve penalties that cover several weeks or the the remainder of the current unemployment spell. It seems important to use an estimate of the weeks of benefits lost due to a nonseparation denial that takes account of this fact.

Table 4-6 shows figures that have been used to derive such estimates. The first row shows the average number of weeks compensated per first payment in the five pilot states (individually and aggregated) and in the United States (ETA 5159 reports). The imputed duration of a multi-week nonseparation penalty (third row) is based on three assumptions: First, the average nonseparation penalty is assessed at the midpoint of a spell of benefit receipt (so that the average multi-week penalty cuts off half the compensated weeks of unemployment). Second, a nonseparation penalty is equally likely to be assessed on new initial and additional initial claims. Third, the average duration of a subsequent spell of unemployment is half that of a first spell (as assumed above). These second two assumptions can be summarized by a "duration adjustment" (DURADJ), which can be written as:

DURADJ = P + (1-P)(0.5)

where P denotes the proportion of all initial claims that are new initial claims (that is, first spells, based on U.S. Department of Labor 5159 reports), and 0.5 is the duration of the average subsequent spell relative to the average first spell. (Intuitively, the duration adjustment reduces the duration of a penalty assessed on a subsequent spell of unemployment by one-half and weights the number of penalties assessed on first and subsequent spells appropriately using P.) The imputed duration of an average multi-week penalty (third row) is then the product of one-half the average weeks of benefit receipt and the duration adjustment.

Finally, the imputed average duration of all nonseparation penalties is the weighted average of the durations of multi-week and one-week penalties (bottom row of Table 4-6). For example, 42.3 percent of all nonseparation penalties in the United States were multi-week, so the imputed average of all nonseparation penalties was 2.91 = [(5.51•0.423) + (1.