Case study

CrossCheck

A public company reports each quarter three times: on the earnings call, in the investor deck and in its SEC XBRL filing. CrossCheck turns all three into source-linked claims and flags where they disagree.

351
injected errors of known size and type, run through both checkers
2.6%
false alarms on real consistent pairs: 1 of 39 for each checker
50.0%
of ×1000 unit errors caught by the Claude judge; the rules catch 100.0%
CrossCheck live demo: a consistency snapshot of revenue claims for PPC, DHR, GE and ADTN, each marked match, discrepant or unscored.
The live demo, with the real claims from Postgres.

The problem

The same quarterly numbers appear in three independent places: spoken on the call, laid out in the deck, and filed as structured facts. When two of them differ, the cause can be a restatement, a rounding choice, a unit slip or a stock split, and they rarely get checked against each other line by line.

I built CrossCheck because checking by hand that an earnings call, an investor deck and an XBRL filing say the same thing is slow and error-prone, and I wanted a system that does it and shows its evidence.

It runs on 5 real companies and their 2020 quarterly filings: Pilgrim's Pride, Danaher, General Electric, Travelers and Adtran.

How it works

Audio
faster-whisper transcribes the call, pyannote.audio separates the speakers and Claude pulls out the spoken figures.
Deck
Figures are read from the investor deck PDF by word position.
XBRL
Typed claims come straight from the SEC companyfacts JSON, no model involved.
Storage
Claims and their evidence spans live in PostgreSQL, with filing text in pgvector.
Rules
Pairs within 0.5% are consistent, beyond it discrepant; unknown units abstain.
Claude judge
Claude Sonnet 5 judges the same pairs and returns a verdict and rationale.

Results

Both checkers were run on 351 injected errors of known size and type, made by altering 41 real consistent pairs, and separately on 101 real hard pairs. The judge catches small gaps the rules ignore, and misses half the unit errors the rules catch.

Recall on injected value errors, by the size of the error (n = 38–39 per point). The rules flag any gap above their 0.5% tolerance, so their step is by construction. Orange whiskers are the judge's 95% confidence intervals.
Recall by injected error size
Error sizeRulesClaude judgeJudge 95% CI
0.3%0.0%51.3%36.2%–66.1%
0.49%0.0%65.8%49.9%–78.8%
0.51%100.0%65.8%49.9%–78.8%
1%100.0%76.9%61.7%–87.4%
2%100.0%84.2%69.6%–92.6%
5%100.0%97.4%86.5%–99.5%
10%100.0%100.0%90.8%–100.0%
Previous quarter n = 27
Rules100.0%
Claude judge100.0%
Year to date n = 18
Rules100.0%
Claude judge100.0%
Unit ×1000 n = 38
Rules100.0%
Claude judge50.0%
Recall on injected errors of each type. Same pairs for both checkers.
False alarms
2.6%
1 of 39 real consistent pairs, for each checker
Judge stability
99.3%
agreement over 5 repeats on injected pairs; 100.0% on real ones
Evaluation cost
$2.89
every judge call the evaluation used

What it gets wrong

  • Unit slips get past the judge. It caught 19 of 38 ×1000 errors and treated the rest as typos, in its own words below. The rules caught 38 of 38, so the two checkers cover each other.
  • Stock splits are an open gap. Neither checker knows about them: 18 of 18 split cases were false alarms.
  • The samples are small. 38–39 cases per error size, and the causes and blind labels were reviewed by one person, me. Most intervals span tens of points.
  • Reported, not patched. Changing the prompt after seeing these results would have invalidated the calls already made, so it was left as it was.

“…after converting units they match exactly, and a unit conversion alone is not evidence of discrepancy.”

The Claude judge, letting a ×1000 unit error through

What I learned

A rule and a model fail in different places. The rules are exact and blind below their tolerance; the judge sees the small gaps they ignore, then waves a ×1000 unit slip through as a typo. Neither is the product on its own: the pair is.

I would not have found the judge's blind spot by reading its answers. It only showed up when I injected errors of a known size and type and counted what came back.

Writing a weakness down is cheaper than hiding it. I left the prompt alone, reported the gap and pointed at the checker that covers it.

Let's talk

Remote preferred. Based in Spain.