I was listening to conversation between Patrick McKenzie and Dave Guarino the other day and stumbled upon a calculation flaw I can relate to.

Patrick McKenzie: Another thing you’ve mentioned to me before is the way formal accounting standards work in these systems. For example, if society establishes a set of rules and we accidentally overpay someone—let’s say $200 more than they were eligible for—we take a $200 hit to our score. If we underpay someone by $50 of the $200 they were eligible for, we take a $50 hit to our score.

But if society establishes a set of rules, and someone is eligible but never completes the application process, we take no hit—because we don’t measure that number.

Exactly. This falls under an area of federal policy called improper payments, and it’s one of my two pet issues. Like you just said, if someone applies and is denied inappropriately—or, even worse, can’t complete the application because it’s so full of friction—then really, we should view that as a $200 underpayment if they were eligible for $200 but got nothing. That should be considered just as bad as someone receiving $200 when they were eligible for zero.

But under current improper payments methodology, which applies across all government benefits programs, we don’t measure that. This comes from federal statutes and regulations implemented by OMB. The way improper payments are defined excludes non-payments, which makes sense in one way. If you start with a universe of improper payments, you’re naturally excluding those who didn’t receive payments at all. But that subtle difference has huge consequences—we don’t count the people who didn’t get through the system.

Patrick McKenzie: This is a wonky "legibility of the system to itself" kind of topic. I think a lot of people or advocates would hear this and say, "Oh, this is just the United States being stingy to poor people, film at 11." But it’s not simply that. If, due to a math error or similar, someone is paid $20 of the $200 they’re eligible for, that $180 difference would be surfaced in reports, counted as a mark against the program, and tracked over time across the beneficiary population.

But the zeros—the people who don’t get through the application process at all—don’t get counted. This isn’t because someone comes into the office with the goal of dropping as many people off the welfare rolls as possible. It’s simply because we started with the mindset that improper payments are the problem to solve. So, we look at the universe of payments and zoom in on which of those are improper, rather than focusing on the bigger picture of access.

Dave Guarino: Exactly. When you start from the flawed foundation that improper payments are the primary concern, it leads to absurd conclusions. Take it to the extreme: if a benefits program made zero improper payments because zero payments were made, you’d quickly realize something is seriously wrong. A benefits program is meant to distribute benefits.

And you're right—there’s no one sitting in a room, smoking a cigar, and gleefully excluding 100% of underpayments from the improper payments methodology. But structurally, we just don’t account for it. This ties into the Seeing Like a State aspect of it. One of the reasons I've heard for this is that it’s a bit harder, bureaucratically and administratively, to determine what someone was eligible for if they didn’t complete the application process. Plus, it’s more difficult to get in contact with those individuals afterward.

Don’t forget to count those unaccounted for. This is a fairly quick cross-check to any metric put forth. In my world, it manifested as an average, which necessitates a data point, and therefore ignores the scenarios in which no data was received. However, it was those that we were after in the first place.