Misreporting of Government Transfers: How Important Are Survey Design and Geography?

Survey Data

The FoodAPS survey collects data from 27 states on all household food purchases, nutrition information, and health. The sample is stratified and weighted to yield nationally representative estimates for research on health and obesity, food insecurity, and food assistance. The survey particularly highlights SNAP receipt and recipients. This emphasis dictates the survey design, as FoodAPS uses two separate sampling frames: a SNAP recipient and a nonrecipient frame. The recipient frame was constructed from addresses of individuals receiving SNAP in February 2012 provided by the states. The nonrecipient frame was constructed by removing the recipient addresses from an address‐based sampling list constructed from the U.S. Postal Service Delivery Sequence file. Roughly 30% of addresses were sampled from the recipient frame; the remainder of the addresses were sampled from the nonrecipient frame.

To account for changes of receipt status after the construction of the sampling frame,22 Such changes may arise, because sampled households may enroll or leave SNAP or move between the construction of the sampling frame and the interview (April 2012–January 2013). In addition, five states did not provide the address data to construct the SNAP recipient frame, so there was only one sampling frame in these states. See the FoodAPS user guide (USDA ERS 2016a) for further detail.
FoodAPS asked about SNAP receipt in a screener interview conducted by telephone. The main point of the screener interviews was to mitigate nonresponse bias and determine eligibility of the household to participate in FoodAPS. Screener information on SNAP receipt and income was also used to define four strata or target groups. The first target group included all SNAP recipients. The other three target groups contained nonrecipients stratified by income relative to the poverty line. A sample of 4826 households was interviewed before and after a week of recording food acquisition between April 2012 and January 2013. Computer‐aided in‐person interviews were conducted with the main food shopper in the household. All information on SNAP receipt and most of the remaining information used in this study stems from the initial interview. Only a few covariates such as household income were collected in the final interview. See the Appendix for a brief summary of the interview procedure and content.

The first SNAP question in FoodAPS was “Do you or does anyone in your household receive benefits from the SNAP program?” If the respondent indicated yes, questions on the date of the last receipt of SNAP benefits, the amount received and whether the amount received was higher, lower, or equal to the usual amount received followed. Those who did not indicate current SNAP receipt were asked questions about SNAP receipt in the past including a question on the date of their last SNAP payment. Note that the initial question did not define a reference period but left the respondent some leeway in defining what constitutes “receiving benefits from the SNAP program.” To study survey error, we need to match the definitions in the survey question exactly, which is not possible if the question depends on the respondent's interpretation. Therefore, we cannot study the initial question on current SNAP receipt. Rather, we examine the following question on the date of the last payment, because the administrative data allow us to exactly match the information this question asks as we discuss later. The FoodAPS documentation provides further information on the survey design.

There are several survey design reasons why misreporting in FoodAPS may differ from that in the general economic surveys used in previous studies. Celhay et al. (2018b) provide a detailed discussion and empirical evidence on how survey design features (such as salience, reference periods, recall, stigma, and respondent cooperation) may affect survey error. Many of these sources of survey error differ between major household surveys and FoodAPS. SNAP receipt was a salient topic in FoodAPS. Respondents were asked about SNAP receipt both in the screener interview and early in the initial interview. The questions regarding SNAP receipt also differed from those in major household surveys. Most importantly, the survey asked about payments in the recent past. Thereby, FoodAPS had a much simpler reference period than most major household surveys that ask about receipt in the past year or even the past calendar year. Most large surveys ask just one question on receipt and possibly a follow‐up question on amounts received. The FoodAPS interview contained a separate section on nutrition assistance that included a sequence of questions on SNAP receipt, making the topic more salient. The salience of SNAP may not only have affected survey accuracy by altering respondent behavior, but also by affecting interviewers. Interviewers were aware of the importance of SNAP receipt status and had information on SNAP receipt status as reported in the screener interview.

Administrative Records and Data Linkage

The FoodAPS survey data were combined with administrative data on SNAP from two sources: administrative records on SNAP payments and SNAP electronic benefit (EBT) card transactions. The administrative records on SNAP payments, called the caseload data, were obtained from state SNAP agencies. The transaction data, referred to as the ALERT data, were provided by the U.S. Department of Agriculture (USDA) and contain a record for each payment with a SNAP EBT card. The FoodAPS documentation (USDA ERS 2016a,b) provides detailed information on the content of these files and how they were linked to the survey data.

Unfortunately, the administrative records differ between states. Six of the 27 participating states did not provide caseload data with payment dates. Therefore, only the ALERT data on SNAP use was linked to the FoodAPS survey. Consequently, no reliable administrative data on the date of the last payment is available in these states. Information on the timing of the last payment is crucial to the central goal of this study, which is to examine whether survey respondents accurately reported receipt of SNAP in the past month. Therefore, we excluded the six states for which this information is not available. Of the remaining 21 states, we exclude eight states that provided caseload and ALERT data, but the ALERT data did not contain linkable identifiers. Thus, rather than linking the combined caseload and ALERT data to the survey, the two data sources had to be linked to the survey separately. In addition to the probabilistic link of the caseload data summarized later, transactions from the ALERT data were linked to the survey data using a different probabilistic record linkage procedure described in the Appendix.33 See Fellegi and Sunter (1969), Copas and Hilton (1990), and Winkler (2014) for detailed discussions of probabilistic record linkage and the FoodAPS documentation for detail on the implementation.

Restricting the sample as summarized in Figure 1 involves a trade‐off between a large, representative sample and data accuracy. Excluding some states and households means that our sample is not representative of the entire United States and (due to stratification and reweighting) not necessarily representative of the population in the states in our sample. Nevertheless, rather than obtaining nationally representative estimates, our main goal is to study survey error and to what extent we can obtain nationally representative estimates in the presence of survey error that may vary across states. Consequently, we only analyze data from the 13 states that provided the most accurate and homogeneous data to minimize confounding survey error with variation in administrative data and linkage quality.44 For one of the 13 states, linkable identifiers were not provided for about half the state, because the state uses two different processing systems for two geographic regions. We only include the subsample with identifiers.
Courtemanche et al. (2018) and Kang and Moffitt (2018) use the data from all states and examine their accuracy and usefulness further. They also find differences in the linked administrative variable between the 13 states that provided caseload and ALERT data with unique IDs and the remaining states. Providing less information and linking the two administrative data sources separately likely induced additional error, so that the linked administrative variable is less accurate for the states we exclude. In addition, studying geographic variation in survey error requires the accuracy of the linked administrative variable not to vary between states. Courtemanche et al. (2018) show that variation between states also stems from differences in data quality or linkage rather than from differences in reporting only. For the sample we use, we have no evidence of systematic differences in the accuracy of the linked data between states. Consequently, these states provide us with the largest sample in which the accuracy of the linked data should be homogeneous enough to attribute variation in the differences between the administrative and survey variables to differences in survey error rather than variation in administrative data quality.

Even for the 13 states in our sample, the data do not contain identifiers to link the administrative data to the survey, so they were linked probabilistically as described in the FoodAPS documentation. The two data sources were linked directly, rather than linking both sources to a population register as done in many prior record linkage studies. Thus, we cannot determine the extent of or adjust for missed links, because we cannot distinguish survey records that did not link to the administrative data because they do not receive SNAP, from survey records that did not link due to missing or incorrect personal information. See Meyer and Mittag (2018) for further discussion and Meyer et al. (2018) for arguments why such missed links likely understate net differences between the data sources. Twenty‐one households in our sample did not provide consent to record linkage. We exclude these households from our sample throughout. For the remaining households, matching records in the caseload data were searched based on first name, last name, phone number, and street address (including apartment number). If a household was matched to a SNAP case, the date of the most recent SNAP payment according to the caseload data was added to the FoodAPS data.

Comparable Administrative and Reported Definitions of SNAP Receipt

Examining survey errors requires comparing survey responses to a measure of truth or a more accurate variable that measures the same concept. See for example, Groves and Lyberg (2010) for a discussion. Consequently, we need to define a measure of SNAP receipt based on the survey reports and construct an administrative variable that matches this definition as closely as possible.

As discussed earlier, the question on which the variable whether anyone in the household receives SNAP (SNAPNOWREPORT) is based does not indicate a reference period.55 The date of last receipt among those who answered this question affirmatively varies between households. Some report dates up to 6 months in the past. Thirty‐nine households report a date of last receipt more than 32 days ago, even though this means that they either recently became nonrecipients or received a more recent payment. This suggests that the interpretation of the question indeed varied between respondents. The FoodAPS household codebook (USDA ERS 2016b) provides further detail on this issue.
Consequently, we cannot match the definition of SNAPNOWREPORT in the administrative data. Instead, we define current receipt based on the reported date of the last payment. Our survey measure of current SNAP receipt is an indicator whether the reported date of last SNAP receipt occurred on the interview day or in the 32 days before the interview.66 We use 33 days to ensure that the time period cannot fall between two monthly payments. SNAP payments are dispersed monthly on fixed dates, so a period of 31 days should always include a payment to every recipient. We consider 33 days to make sure we do not miss any regular recipients, because they may have received a payment later on the interview day or because some states do not distribute SNAP payments on Sundays and thus vary payment dates by a day.
According to our survey variable, 691 out of 2257 survey households are SNAP recipients, which yields a (weighted) receipt rate of 11.7%. There are several other ways to define SNAP receipt. Table A1 provides an overview of both survey and administrative SNAP receipt definitions. Changing the time period of reported SNAP receipt to 31 or 33 days before the interview reclassifies less than five households. Neither change has a visible effect on the rates of reported receipt and survey error. On the other hand, our variable differs from SNAPNOWREPORT for 59 households. Fifty‐three are reclassified as recipients, which increases the estimated reported receipt rate by one percentage point.

To assess errors in the survey reports, we need to define a second SNAP receipt variable based on the linked administrative data that matches our survey definition of SNAP receipt. Both the caseload and ALERT data contain exact dates of payments and transactions, so we can match the definition of the survey variable in each of the two linked administrative data sources exactly. We consider a household to be a current SNAP recipient according to the administrative data if either the linked caseload or ALERT data indicate a payment or a transaction on the interview day or in the 32 days before the interview. As Table A1 shows, using a 31 or 33‐day period before the interview day does not affect the variable at all.

The survey question asks about the date of the last benefit receipt, which would ideally be captured by the SNAP payments from the caseload data alone. Even though this definition is less congruent with the survey question than one based on the caseload payments only, we use the additional information on transactions from the ALERT data to define administrative SNAP receipt for two reasons. First, the linked caseload variable is not error‐free and incorporating information from the ALERT data appears to correct some errors. Table A1 supports this point. There are 119 households with ALERT transactions, but no SNAP payment in the reference period. Transactions require prior payments, so these households likely received payments to their EBT card during the reference period that were not recorded in the linked caseload data due to errors. The only other explanation would be that these households used to receive SNAP, but currently do not, and spent SNAP benefits from prior months during the reference period.77 It is important to note that for our sample, the caseload and ALERT data were linked deterministically, so disagreement between the two sources cannot arise from mislinking one source, but not the other.
This explanation seems unlikely, because most SNAP benefits are spent within a few days of receipt and are rarely carried over to the next month. In addition, there are clearly too many households of this type to be explained by SNAP exit or interruptions.88 As Table A1 shows, recording these households as nonrecipients would lead to an implausibly low rate of SNAP receipt of 10.4% that would indicate substantial over‐reporting of 12%.
Consequently, the ALERT information on usage in the reference period indicates that the household also received SNAP benefits in the reference period that were missed by the caseload data or link. A second reason to consider these households to be administrative recipients is that we are mainly interested in whether households report receiving support from SNAP. Households who report SNAP receipt this month, because they used the EBT card even though there was no payment this month are a minor problem compared to the large share of households who do not report SNAP receipt. Therefore, we prefer to classify as correct reporters the few households that appear to misinterpret the survey question and report SNAP use rather than benefit receipt.

Defining administrative SNAP receipt based on the most recent administrative dates allows us to match the variable definition of the survey. Unfortunately, how the survey producers determined the most recent date is unclear and this process resulted in some dates being after the initial interview. FoodAPS only merged at most one payment date to each survey household. Thus, we do not know whether these households should be classified as recipients, because they also received SNAP before the interview, or as nonrecipients, because they enrolled in SNAP between the initial interview and the date recorded in the administrative data. A total of 262 administrative recipient households have dates after the initial interview according to at least one of the administrative data sources. It seems implausible that 12% of all interviewed households enroll in SNAP shortly after the interview. Table A2 tabulates the frequency of days passed between the interview and the recorded date, showing that most dates after the initial interview are shortly after the initial interview. This result supports statements by the survey producers that the linked dates after the interview likely arise from using an algorithm to pick the most recent date that may erroneously consider an administrative receipt date after the initial interview to be the most recent date if more days passed between the last date before the initial interview than between the initial interview and the first date after the initial interview.

Nonetheless, some of the households with administrative dates after the initial interview may have enrolled in SNAP after the initial interview and should therefore not be classified as administrative recipients. We combine information on receipt before the interview from several sources to identify households that are unlikely to have enrolled in SNAP between the initial interview and the recorded administrative date. For 145 of the 262 households with a date after the initial interview in either the caseload or ALERT data, the other administrative data source indicates that the household already received SNAP before the initial interview. Of the remaining households, 76 are from the SNAP recipient frame. Consequently, we know that someone at this address received SNAP in February and later in the interview year, which makes it very likely that the household also received SNAP at the time of the interview.99 This rule may misclassify a small number of households who either interrupted a SNAP spell at the time of the interview or moved into an address from the SNAP recipient frame, did not receive SNAP at the time of the interview, but enrolled in SNAP shortly after the interview date.
Finally, for 24 of the remaining observations, the administrative data indicate a payment or transaction within a week of the initial interview. It usually takes several days to process SNAP applications and make payments and hence for transactions to be possible. Thus, it appears more likely that these households were already recipients at the time of the initial interview than that they managed to enroll in SNAP and receive benefits in less than a week. We drop from the sample the remaining 17 households for which we cannot determine whether they already received SNAP at the time of the initial interview.1010 Classifying these 17 households as administrative recipients increases the false negative rate by 4.4 percentage points and thereby amplifies our results. Another alternative is to classify only households for which at least one of the administrative data sources indicates receipt in the 32 days before the initial interview as administrative recipients. This rule increases the false positive rate by 1.5 percentage points and reduces the false negative rate by 2.1 percentage points. Thus, most households reclassified as nonrecipients by this rule report receipt, making them likely recipients.

The households with administrative dates of SNAP receipt or use after the initial interview also lack a measure of time between last receipt and initial interview, a variable we use in a later analysis. We impute a receipt date prior to interview assuming that it occurred 30 days before the recorded date. We only use these imputed variables in our analyses of recall and document that our findings still hold without the imputed observations. Figure 2 shows that including these households with imputed administrative dates of last receipt also removes an anomaly in the frequency distribution of time since last administrative receipt. Interview dates were not aligned with SNAP disbursement dates, so one would expect the number of days between the two dates to be roughly uniformly distributed. The shaded bars in Figure 2 show the frequency of days as originally recorded, that is, without imputed days after the interview. The frequency of interviews between 23 and 30 days is much lower than one would expect based on a uniform distribution. The contour‐bars in Figure 2 also include dates after the initial interview replaced by the recorded date minus 30 days.1111 This rule only adds observations to the frequency plot, so that the contour‐bars always cover the shaded bars.
Including these imputed dates adds the “missing” interviews between 23 and 30 days, because most recorded dates after the interview are in the week after the interview as Table A2 shows. Imputing these dates thereby makes the frequency distribution closer to the uniform distribution we would expect, which again underlines that dates after the interview are likely driven by a coding problem that considered dates shortly after the interview to be the most recent date.

Finally, the FoodAPS data set provides a SNAP receipt variable, SNAPNOWHH, that is a combination of administrative and survey data on reported receipt. The main problem in using SNAPNOWHH to study survey error is the way it combines administrative and survey reported receipt, which mutes almost all overreporting. See the Appendix for further information.

Recall

Reference periods differ in length and complexity between the surveys in Table 1. The ACS asks about SNAP receipt in the 12 months before the interview. The CPS collects information on SNAP receipt and amounts received in the previous calendar year in spring of each year. The SIPP is a panel survey that asks about monthly SNAP receipt in the four months preceding the interview. Both the length and the complexity of the recall period are known to affect reporting accuracy. The last row of Table 1 already hints at recall playing a role in explaining the differences. It restricts the sample of Celhay et al. (2018a) to current administrative recipients, that is, households that receive SNAP in the month of interview (and in the reference period of the survey in the case of the CPS) according to the administrative data. The recall problem faced by these households is more similar to the one faced by SNAP recipients in FoodAPS in that both are asked to recall an event that happened in the past month.1212 They still differ in that FoodAPS respondents are asked to recall a date and that respondents of the other surveys are asked to recall any payments over a longer time period rather than only in the past month.
The rates at which current administrative recipients fail to report are indeed lower than in the overall population. The reporting rates of current administrative recipients in the ACS and SIPP are similar to the reporting rate of administrative recipients in FoodAPS—slightly lower in the SIPP (17%) and slightly higher in the ACS (21%). While the false negative rate in the CPS drops to 38%, it remains substantially higher than in FoodAPS and the other surveys, when restricting the sample to current administrative recipients.

These lower error rates among current recipients point toward recall errors, but could also be driven by differences in the time period and state under study, or the salience of receipt between current and former recipients, or other differences. The administrative variables in FoodAPS provide us with the number of days since the last payment and the number of days since the last transaction. This information enables us to examine the role that time elapsed since these events has on responses. Contrary to prior studies where the elapsed time depends on a respondent's SNAP receipt history, here the variation is due to the timing of the interview relative to the last SNAP payment or use. This variation, though only over the month before interview, is likely exogenous, adding to the credibility of the evidence.

Figure 3 plots how the reporting rate among administrative recipients varies with these two measures of time elapsed.1313 The figure excludes those with imputed administrative dates of last receipt or use. Including these observations does not change the figure much, though the decline in the reporting rate with use is slightly less pronounced.
Both the dashed line that is based on days since last administrative receipt and the solid line that is based on time since last use remain fairly stable at a reporting rate slightly above 80% for the first 20 days. The reporting rate declines rapidly with the number of days since last use after that. On the other hand, reporting rates do not seem to decline in the number of days passed since the last payment, even after 20 days. Figure 3 confirms prior results showing that the probability of reporting an event is declining in the length of time since the event occurred (e.g., Sudman and Bradburn 1973; Groves et al. 2009; Celhay et al. 2018b). We find that the probability of reporting receipt declines in time since last EBT card use rather than time since last SNAP payment, which suggests that recall of recent SNAP receipt is driven by recent SNAP use, rather than the receipt of payments.

Table 2 reports estimates from multivariate (single‐equation) Probit models of the probability of survey errors.1414 The multivariate analyses use a sample of 2256 observations throughout, because one observation is missing a value of one of the covariates. The results are not affected by dealing with the missing data in other ways.
The estimates in columns 1 and 2 are the determinants of the probability that an administrative recipient household fails to report SNAP in the survey, that is, is a false negative. In addition to conditioning on time since use and receipt, the model includes demographic and economic characteristics. The results confirm that it is time since last SNAP use rather than last payment that predicts failure to report SNAP receipt, holding other household characteristics constant. The number of days since the last ALERT transaction is significant and increases the probability of a false negative by 0.56 percentage points per day. The effect of the number of days since the last payment is insignificant and small at 0.14 percentage points per day.1515 The estimates are slightly larger if the imputed observations are excluded, so the results do not depend on the recoding of the administrative dates of receipt after the initial interview.

That time since last SNAP use rather than last payment drives the recall results may be due to people being more aware of card use than of payment receipt. The difficulty recalling a specific event (a SNAP payment) may not depend on the time since that event, but on the time since the last time the respondent was reminded of this event. This result suggests that the fact that FoodAPS attempted to interview the main food shopper may have contributed to the lower false negative rates. The main food shopper is more likely to possess and hence use the EBT card, which according to our analyses reduces recall problems. Our finding that card usage rather than payment receipt matters also points to another potential mechanism behind the common finding that larger households are more likely to fail to report program receipt: The larger the household, the less likely it is that a given respondent recently used the EBT card and hence the more likely that the household fails to report.

Demographic Characteristics

We would like to examine if the predictors of survey error differ between general economic household surveys and a specialized survey like FoodAPS, by comparing the determinants of FoodAPS errors to those in the literature. In addition to the determinants of false negative reports, Table 2 also reports estimates of a Probit model for the determinants of false positive reports, the probability that nonrecipients according to the administrative data report SNAP receipt. Meyer et al. (2018) and Celhay et al. (2018a) examine the role of household characteristics using more precise estimates from larger linked samples.

Overall, we find similar characteristics predict survey error as in the general economic surveys Meyer et al. (2018) and Celhay et al. (2018a) examine. Households with higher incomes and more education are more likely to be false negatives, but less likely to be false positives. Thus, in line with prior studies, we find these households report less receipt, rather than uniformly report less well. Conversely, those who are unemployed, or report receipt of other programs commit fewer false negative and more false positive errors. They report receipt more frequently. With reductions in false negative rates by 13–15%, reporting other programs strongly predicts correct reporting by SNAP recipients. As in Celhay et al. (2018a), we also find Hispanic respondents commit more false positives and more false negatives, that is, errors in reporting SNAP are more frequent regardless of true receipt status. Contrary to Celhay et al. (2018a), who find this result to be even more pronounced among black respondents, we do not find systematic differences in the error rates of black respondents and the overall population.

Contrary to these prior studies, we only find weak effects of household composition and no differences in reporting by gender, age, or disability status. These last results may be due to less precise estimates from our smaller sample, rather than differences in the nature of the survey error. The signs and magnitudes of most coefficients, even if insignificant, are well aligned with prior studies. Consequently, our results indicate that survey error varies systematically with demographic characteristics, but this relationship is similar in previously validated general economic surveys and a specialized survey like FoodAPS.

Interviewers

Interviewers play a key role in the survey process, so their impact on error rates has received considerable attention in the literature on survey design.1616 Bruckmeier et al. (2015) study the role of interviewers in transfer receipt and provide references.
Interviewers may affect response accuracy directly. They may decrease accuracy by not properly conducting interviews or conversely improve accuracy by following up on responses that seem likely to be erroneous. Some interviewers may probe more in states where SNAP is known under a different name. They may also have more subtle effects on answers by suggesting certain answers or creating stigma. FoodAPS interviewers were likely aware of the importance of SNAP for the survey and had information on the response to the screener question on SNAP receipt. The degree to which interviewers made use of this information and emphasized the importance of SNAP may have varied between interviewers, so studying interviewer heterogeneity in FoodAPS is particularly interesting.

The FoodAPS data include interviewer identifiers, which enables us to examine how error rates vary between interviewers. We first compute false positive and false negative rates by interviewer. The 2257 (initial) interviews in our sample were conducted by 92 interviewers, but many interviewers only conducted a few interviews. To exclude very noisy estimates of the error rates, we only examine the false negative rates of the 50 interviewers who conducted 7 or more interviews with administrative recipients and the false positive rates of the 69 interviewers who conducted 7 or more interviews with administrative nonrecipients. The excluded interviewers who conducted few interviews have a much higher average false negative rate—16 percentage points higher than the average rate among those with 7 or more interviews.1717 There are no false positives among the interviews with nonrecipients conducted by those with few interviews, but we cannot reject that this error rate is the same as the average false positive rate.
This difference shows that error rates vary between interviewers. Still, we cannot distinguish between several potential reasons for the higher error rates among those with few interviews, which may be due to interviewer experience or interviewer retention and assignment policies. Turning to interviewers with 7 or more interviews, we first test whether error rates are equal across interviewers. We reject that false negative rates are equal across interviewers with a p‐value below 0.0001. We do not reject that false positive rates are equal across interviewers (p‐value of 0.12). That we do not find any effects on false positives may indicate that false positive rates are not greatly influenced by interviewers, but may also be due to estimation noise from a much lower false positive rate.

Differences in unconditional error rates may be due to interviewer characteristics and behavior but may also arise from systematic differences in the respondents to which interviewers are assigned. We document that error rates vary systematically with respondent characteristics above. FoodAPS interviewers were assigned to households based on geography, so the demographic composition of interviewers' assigned areas likely causes differences in error rates between interviewers. To mitigate this problem, we analyze error rates controlling for demographics by adding interviewer dummies to the multivariate Probit models in Table 2. We estimate the conditional effect of interviewer i on the false positive and false negative rate as the marginal effect of the dummy for interviewer i from these two models.1818 Using Probit marginal effects to calculate these error rates means that only the differences in error rates between interviewers can be estimated. Using a linear probability model instead of a Probit does not substantively alter our conclusions in this section.
To mitigate the incidental parameter problem, we include only one dummy for all interviews conducted by interviewers with less than seven interviews.

Controlling for demographics, the false negative rate for interviewers with few interviews remains substantially higher than among those with more interviews: The Probit marginal effects suggest that it is 8 percentage points higher than the average.1919 The results in the remainder of this section are based on models that exclude the interviews conducted by those with few interviews, but including them does not affect the results.
We now only weakly reject that interviewers with many interviews are homogeneous in their conditional false negative rates with a p‐value of 0.08. This weak result combined with the large difference between those with few interviews suggests that overall interviewers differ systematically in their probability of producing a false negative response, even when assigned to households with the same demographic characteristics.

This result raises the question whether the differences in error rates between interviewers are large enough to make a substantive difference. Adding the interviewer dummies to an error Probit model with the covariates only substantially improves prediction accuracy.2020 For false negative rates, McFadden's pseudo‐ R² increases from 0.19 to 0.34, an 81% increase. For false positives, it increases from 0.30 to 0.50, a 67% increase.
It does not lead to meaningful changes in the estimated coefficients on the covariates of interest. Examining the magnitude of the interviewer effects is complicated, because the estimated interviewer effects confound true differences between interviewers and estimation noise. For false negatives, we find an interquartile range of the estimated interviewer effects large enough to leave scope for true interviewer effects of substantive importance.2121 Excluding interviewers with no false negatives for comparability, the interquartile range is 18 percentage points for the estimated unconditional interviewer effects and 19 percentage points for the estimated conditional effects. Including interviewers with no false negatives yields an unconditional interquartile range of 17 percentage points and a conditional interquartile range of 12 percentage points.
On the other hand, comparing the dispersion of the estimated effects to the dispersion one would expect based on estimation error alone suggests that this difference could be estimation noise.2222 The variance of the estimated conditional interviewer effects is smaller than the average of the variance of the estimation error in all models.

Overall, our results provide evidence that interviewers differ systematically in their error rates, even when conditioning on covariates. This finding suggests that interviewers are heterogeneous and that the differences are not entirely due to interviewer assignment, but our small sample does not provide strong evidence on the nature and magnitude of interviewer effects. We find interviewers with few interviews to have substantial error rates, but the results are neither precise enough to assess differences in the magnitude of the other interviewer effects, nor to establish how much of the variation between interviewers is due to interviewer heterogeneity and how much is due to differences in assignment.