Every validation vendor claims 99% accuracy. None will show you the test.

A number nobody can check

Read the homepages of the phone and email validation vendors. You will find 99.97% accuracy, 98% deliverability, 15% more accurate than competitors. What you will not find, on any of them, is the dataset those numbers were measured against, the method, or the date.

That is not an accusation of dishonesty. It is a description of a market norm, and the norm is stable for a reason: when nobody publishes a test, publishing one puts you at a disadvantage. Your honest 94% sits next to somebody else’s unaudited 99.97% and looks worse. Everybody loses except the vendor with the boldest marketing.

The buyer loses most. A validation verdict is a decision input - send or do not send, dial or do not dial, accept this signup or hold it - and the cost of a wrong verdict lands on them, not on the vendor who published the number.

What an honest benchmark has to do

Four things, none of them difficult, all of them absent from the claims on offer.

Publish the cases. A percentage with no test behind it is a slogan. If the cases are public, a reader who disagrees can point at a specific one and say why, which is the only kind of disagreement that improves anything.

Make them adversarial. Any provider scores well on well-formed inputs. The interesting question is what happens to a number that is structurally perfect and permanently unallocated, or to a mail server that answers 250 for every address you ask about. An easy set measures nothing and flatters everyone.

Separate what you measured from what you simulated. If the network tier ran against a stub because no upstream was configured, the resulting figure is a statement about your stub. Folding it into one headline number is the same failure as inventing the number, and much easier to do by accident.

Lead with the false positive rate. Overall accuracy hides the error that costs money. Saying an address is valid when it is not produces a hard bounce, a wasted dial, an accepted fraudulent signup. Saying you are unsure costs a little friction. One figure treats those as equivalent, and they are not.

The dataset

Twenty-three labelled cases, each chosen because it breaks a category of validator rather than because it is hard in the abstract. Published in full under CC0, which is to say it is not ours in any meaningful sense: take it, extend it, run it against us.

A sample of what is in it and why:

Our results, including the part that was not a measurement

On the structural cases - decidable offline from numbering plans, DNS and regulator allocations - we score 100%, and so would anybody who bothered to encode the allocations. There is nothing clever about it. The cases exist because most validators do not.

The false positive rate is zero: no case where we said valid and it was not.

The inconclusive rate is 4.3%, and that number is deliberately not zero. When a mail server accepts every address you ask about, no probe distinguishes a live mailbox from a dead one, and the honest verdict is that we do not know. Driving it to zero would mean resolving coin flips into confident answers, which is the behaviour that produces the false positives this benchmark exists to catch.

Until three days ago the network cases ran against a simulator, and the harness said so on every run. They now run against a live carrier register and a real mailbox probe. That is the difference between a claim and a measurement, and it is worth stating that we were on the wrong side of it very recently.

npm run bench

  Structural accuracy   100.0%   <- reproducible offline, on any machine
  Network accuracy      100.0%
  False positive rate     0.0%   <- the one that costs money
  Inconclusive rate       4.3%   <- unbilled by design

Three things the dataset found wrong with us

Publishing a test you actually run has a cost, which is that it keeps finding things.

Our list of catch-all domains named Yahoo, AOL, iCloud and Me. Probing all four with random local parts returned a clean rejection every time: they are not accept-all, and had not been for some time. We were answering "we cannot tell" for addresses a probe settles in 200 milliseconds. The list is now empty. What remains is structural - a throwaway inbox provider accepts everything because that is the product, not because of a configuration that might change.

Our carrier lookups were being run on landlines. Home registers hold mobile subscribers; asking one about a London 020 number returns whatever entity owns the numbering block, shaped like a subscriber record. It came back as "Jersey Telecom, reachable" and was about to contribute a strong positive signal to the verdict, at full price.

Our IP simulator invented attributions. For an ordinary address it reported a specific cloud provider, a risk score, and proxy and VPN both false. Four claims, none established, presented as findings - the exact behaviour this argument is against, shipping inside the product making it.

An invitation

The dataset is at /benchmark/dataset.json and the harness ships with the source. Run it against us. Run it against whoever you currently pay. Publish what you find, including if we come out badly - especially then.

If you think a case is wrong, say which and why. One already carries its own correction: the Yahoo case expected "unknown" for months on an assumption that had quietly stopped being true. It is kept rather than deleted, because a case that once encoded a stale belief is worth more as a reminder than as a gap.

And if you are a vendor with a better dataset, publish it and we will run against yours. The point was never that ours is the right test. It is that the category currently has no test at all, and any published one beats a number with nothing behind it.